[This entry was updated on May 2, 2007 with additional information about limitations of this technique when used with the DOS shell, as noted in the comments. --Oliver]
It’s sometimes handy to create an application that can run either with a GUI or, if started from the command line, a command line UI. For example, you might want an installer to work this way so that it can support both GUI-driven and “silent” (command-line) installations. Windows provides a simple but not necessarily obvious way to create these dual-mode applications.
Most Windows applications run on the windows subsystem. These applications can try to write to the console, but they aren’t attached to one by default. Using this method your GUI will work just fine but any command line output will be discarded.
An application can be run on a different subsystem, of which “console” is one choice, by setting the /subsystem flag of the linker. When these applications are run they are attached to the console, and they can also open new windows. The catch is that Windows will make sure there is always a console present. If the program is run by, say, double-clicking it in Explorer then Windows will first create a new black-and-white console window, attach the application to it, and then launch the application. The application can close the console window, but not before it’s been seen by the user.
Here’s an approach that does work: Create a windows subsystem application and call AttachConsole(). If run from the Explorer there’s no console to attach to, but that’s ok–you weren’t going to use it, anyway. If your application is run from the command line then this call will attach your application to that same console–which is exactly what you want.
This technique works when running the application from bash and similar shells, from Ant scripts, and so on. However, it does not work with the DOS shell because the DOS shell will not wait for any GUI (Windows subsytem) application. You can work around this in the DOS shell using the /wait option to the start command, like so:
c:> start /wait dualmode.exe