The secret lies in file path format. For example, Cygwin states the following:
Invalid filenames
Filenames invalid under Win32 are not necessarily invalid under Cygwin. There are a few rules which apply to Windows filenames. Most notably, DOS device names like AUX, COM1, LPT1 or PRN (to name a few) cannot be used as filename or extension in a native Win32 application. So filenames like prn.txt or foo.aux are invalid filenames for native Win32 applications.
This restriction doesn't apply to Cygwin applications. Cygwin can create and access files with such names just fine. Just don't try to use these files with native Win32 applications.
Looking at Cygwin source code and tracing calls to
mkdir AUX, we can see Nt* Windows API calls using \??\ in the beginning of the path:
1482 333244 [main] mkdir 7128 normalize_posix_path: src AUX
505 333749 [main] mkdir 7128 cwdstuff::get: posix /cygdrive/c/Users/billy/Downloads/___test___
944 334693 [main] mkdir 7128 cwdstuff::get: (/cygdrive/c/Users/billy/Downloads/___test___) = cwdstuff::get (0x600000010, 32768, 1, 0), errno 0
984 335677 [main] mkdir 7128 normalize_posix_path: /cygdrive/c/Users/billy/Downloads/___test___/AUX = normalize_posix_path (AUX)
983 336660 [main] mkdir 7128 mount_info::conv_to_win32_path: conv_to_win32_path (/cygdrive/c/Users/billy/Downloads/___test___/AUX)
1025 337685 [main] mkdir 7128 mount_info::cygdrive_win32_path: src '/cygdrive/c/Users/billy/Downloads/___test___/AUX', dst 'C:\Users\billy\Downloads\___test___\AUX'
985 338670 [main] mkdir 7128 set_flags: flags: binary (0x2)
926 339596 [main] mkdir 7128 mount_info::conv_to_win32_path: src_path /cygdrive/c/Users/billy/Downloads/___test___/AUX, dst C:\Users\billy\Downloads\___test___\AUX, flags 0x4022, rc 0
1081 340677 [main] mkdir 7128 symlink_info::check: 0xC0000034 = NtCreateFile (\??\C:\Users\billy\Downloads\___test___\AUX)
954 341631 [main] mkdir 7128 symlink_info::check: 0xC0000034 = NtQueryInformationFile (\??\C:\Users\billy\Downloads\___test___\AUX)
1046 342677 [main] mkdir 7128 symlink_info::check: 0xC0000034 = NtCreateFile (\??\C:\Users\billy\Downloads\___test___\AUX.lnk)
924 343601 [main] mkdir 7128 symlink_info::check: 0xC0000034 = NtQueryInformationFile (\??\C:\Users\billy\Downloads\___test___\AUX.lnk)
1001 344602 [main] mkdir 7128 symlink_info::check: 0 = symlink.check(C:\Users\billy\Downloads\___test___\AUX, 0xFFFFB680) (0x4022)
If you open CMD, and do the following:
mkdir \\.\c:\Users\<USER_ID>\path\AUX
Windows will happily create the AUX folder. You can also do the following to create CON.txt using Notepad:
notepad.exe \\.\c:\Users\billy\Downloads\__test__\CON.txt
Notepad will first state that it cannot find the file \\.\c:\Users\billy\Downloads\__test__\CON.txt and asks whether it should create it. If you click Yes, it will create file CON.txt. Note that neither TC, CMD, or explorer will see the file. If you repeat the same command "notepad.exe \\.\c:\Users\billy\Downloads\__test__\CON.txt", notepad will open the file successfully. I'm not sure if these files are created as alternate data streams but nevertheless, path formatting is necessary to work with special filenames.
Also, take a look at "\newlib-cygwin\winsup\cygwin\path.cc" in the Cygwin source.