The project's pre-pre-alpha will be out very soon (in a week or less) and will already give a nice taste of its design and speed. The primary focus thus far has been major challenges (toolchain, process creation and initialization, signals, glue layer between posix system call layer and libc, etc.). This means that while you will be able to test functions such as fork(), execve(), mmap(), or fopen() with a utf-8 path, some of the easy/easier-to-implement system calls will "surprisingly" still be missing.
For one or the other reason, fork() has become in many discussion of posix on windows something of a fetish. The interface surely has its place, and figuring out how to efficiently implement it took a huge amount of effort, yet the vast majority of applications do not truly need it. Matter of the fact is that even on linux, where fork(2) is natively supported, the sequence fork+execve is more costly than clone+execve (where clone's flags are CLONE_VM && !CLONE_THREAD). For additional reference, see for instance the implementation of posix_spawn in musl libc (http://git.musl-libc.org/cgit/musl/tree/src/process/posix_sp...).
If high performance of the posix layer were not possible the project would have not existed. Among the factors that make high performance possible are 1) direct use of kernel interfaces (aka the Native API, where most of the runtime layer is written as a user-space driver), 2) utf-8 as the primary supported multibyte encoding as a foundational concept rather than an afterthought, and 3) tls implementation that matches in speed the native tls facility.
The entire fork/exec model is bonkers. The most egregious offsense is that it leads to things like Linux's OOM killer. If you ever find yourself writing something called an "OOM killer" you need to stop and seriously reconsider your choices up until that point. Even on the older Unix systems when the fork/exec model was first introduced, it was bloody stupid because there were much less sophisticated memory managers at the time and the performance cost of fork/exec was massive.
No, OOM killer is completely orthogonal to this and is a consequence of not doing correct commit accounting. With strict commit accounting turned on (vm.overcommit_memory=2) fork will correctly fail when there is not sufficient physical backing. But you're correct that fork+exec is a bad model.
The OOM killer exists because of memory overcommitment, which exists because of fork/exec. The justification for overcommitment is that a big-ass process might fork just to do an exec immediately afterwards. If that is the case, then it would be lame to error out the fork because there's not enough room for a second copy of it. But if it doesn't actually exec, then suddenly there's not actually as much memory as it thinks it has as the forked process starts modifying things. This justified overcommitment in the first place and then it snowballed from there.
No, fork is only one path that can lead to overcommit. Allocation of new memory as COW references to a zero page, and COW writable MAP_PRIVATE mappings of files (such as the writable LOAD segments of any executable or library file) also lead to overcommit unless you do proper commit accounting. Any system that does not need to do detailed commit accounting to avoid overcommit is basically wasting the fact that it has virtual memory/MMU.
I never meant to suggest that fork is the only path that leads to overcommitment, but that fork/exec generally insists on overcommitment for reasonable use.
Good luck. Cygwin has explored this space thoroughly. If there were a better option that preserved compatibility, the project would use it. Keep in mind that Cygwin already uses the native API extensively. Read the code.
As for fork: believe me, I'd be happy if, as as side effect of this work, more applications used posix_spawn. But Cygwin can also support an efficient posix_spawn implementation: https://github.com/dcolascione/cygspawn
A lot of the complexity actually comes from mapping POSIX filesystem semantics to Unix ones. stat(2) in Cygwin is ungodly expensive for this reason. Your layer won't be able to avoid this work without providing fewer features.
There are still plenty of applications that use fork semantically, to keep the same process image in the child, which benefit somewhat from a fast fork. But most places where fork affects performance now are things that are already pessimized by using fork+exec instead of posix_spawn: the shell, make, cgi, etc. (GCC still uses vfork, but GNU make recently switched from vfork to fork because of vfork-related bugs.) Regardless of how fast or slow fork is on midipix (but I expect it to be fairly fast, much faster than cygwin), they would benefit a lot more from just switching to using posix_spawn.
To clarify, the numbers in my tweets are measured on Linux, comparing musl libc's posix_spawn (with CLONE_VM) to plain fork+exec (which should be independent of libc). I just posted the test program on our mailing list: http://www.openwall.com/lists/musl/2015/06/04/1
A quick update regarding toolchain work and release time: we have now taught binutils and gcc to do the right thing for the target with respect to weak symbols and GOT entries. That has been quite a ride, but gladly dynamic linking is now working exactly as desired (that is, without depending on import/export annotation in the libc headers). Focus has now shifted back to the runtime layer, and we are working in full speed towards the pre-pre-alpha release.
Which aspects of midipix, if any, will be under the GPL? If you plan to release any of the runtime components under the GPL, will you also offer a commercial license?
The overall approach is to license cross-platform tools under the MIT license, and the Windows-specific runtime components under GPLv2 and GPLv3 that could be supplemented with a commercial license.
With FILE_SHARE_DELETE there shouldn't be any particular problem, but I certainly want to test this with the posix flags you had in mind. Do you happen to have a minimal example that you deem problematic on Windows?
If you delete the backing file of (or perhaps even close its handle -- I don't remember) a memory-mapped file, you get VERY bizarre behavior in Windows. I don't remember what it was, but I remember I did it and the behavior I got was quite nonsensical.
This project sounds too good to be true. I would be quite astonished if they can make fork() work seamlessly without an amazingly high performance cost. In my experience the only way to do such a thing is to use NtCreateProcess(), but that function itself seems impossible for anyone besides Microsoft to use correctly. (I have tried numerous times and failed, and so have many others.)
Parent initializes a space in the Cygwin process table for child. Parent creates child suspended using Win32 CreateProcess call, giving the same path it was invoked with itself. Parent calls setjmp to save its own context and then sets a pointer to this in the Cygwin shared memory area (shared among all Cygwin tasks). Parent fills in the child's .data and .bss subsections by copying from its own address space into the suspended child's address space. Parent then starts the child. Parent waits on mutex for child to get to safe point. Child starts and discovers if has been forked and then longjumps using the saved jump buffer. Child sets mutex parent is waiting on and then blocks on another mutex waiting for parent to fill in its stack and heap. Parent notices child is in safe area, copies stack and heap from itself into child, releases the mutex the child is waiting on and returns from the fork call. Child wakes from blocking on mutex, recreates any mmapped areas passed to it via shared area and then returns from fork itself.
And where is the copy-on-write happening here? If I recall correctly, the problem was that attempting to reproduce fork()'s copy-on-write behavior on Windows resulted in a massive performance hit. (Which, IIRC, contributes to the slowness of Cygwin.)
EDIT: Oh, I'd skipped the part above "here's how it works"... that just says exactly the same thing I said above.
It is possibly to fork on Windows by using ZwCreateProcess directly, however the win32 subsystem and every single thirdparty library you are using does not expect this. If you are only using the NT native api then it should work just fine, but you are prevented from interacting with the win32 environment.
> if you are only using the NT native API then it should work just fine
I believe I had trouble executing any instructions in the new process at all. If you can make it work I'd like to see your code, otherwise I'm skeptical.
AFAIK this is how fork() is implemented in SUA, but I haven't tried it myself. You could load the SUA subsystem dll into IDA Pro and see how they actually do it.
I believe the POSIX subsystem has some kind of support from the native API that prevents you from writing your own random subsystem and expecting it to work; I don't remember what it was though (it's been a few years).
There's some special handling when loading the image where it does different things depending on the value of the subsystem field, but I don't know whether there's any special handling in the kernel that can't be duplicated by using the native api. But the NT kernel does very little when it comes to initializing new processes, most of the initialization is done in user space by ntdll, so it seems unlikely.
> had problems with getting it to work with the win32 subsystem
Keep in mind that the whole _point_ of Cygwin is to interact with the Windows world. If you want a POSIX sandbox, you can use a VM with fewer headaches and better performance.
Keep in mind that in addition to the build steps, it's common to break out to sed, grep, and shell to get basic string operations done due to extremely limited capability of make itself. On cygwin, this is slow for large projects.
I have few "real" performance issues with Cygwin as well. However, when I perform the same tasks in gnu/linux there is a noticeable speed increase. So what I mean to say is that Cygwin's performance isn't necessarily bad, it's just not as good as the real thing.
I use Cygwin as my terminal environment for everything that works. Any fs operation involving lots of files (e.g. find) is slow. Copying files is slow. CIFS in particular seems much slower via //server/share in Cygwin than native \\server\share.
One way to really see the latter is to try to create a git clone using the --reference option against a repo on a mapped drive; msysgit using the drive letter is OK, but cygwin git against the same repo via /cygdrive/mapped is very slow.
Edit: by 'everything that works' I mean everything that interacts with the terminal in a standard way. There are some command line utilities which output to the terminal some way other than stdout, making them very difficult to use in cygwin.
This is part of the reason why I don't think compatibility layers like this are worth bothering with if you're after performance - different OSs do things in different ways, and while it's possible to make one work like another, it's sub-optimal because that wasn't the use-case the OS was designed for.
It's really a high-level design decision: Applications originally designed for *nix will use fork(), while those for Windows will use something else - maybe threads, maybe CreateProcess().
Not only that, like Mac OS X, Windows is moving into another application model WinRT with containers. Just like Carbon, only selected Win32 APIs will survive in the long run.
Which leaves the question how much POSIX can be implemented on top of WinRT.
Similarly not all POSIX calls are allowed inside Mac OS X App Sandbox.
And if we restrain ourselves to POSIX there is little more than command line applications, TCP/IP headless servers and Motif GUIs.
> Not only that, like Mac OS X, Windows is moving into another application model WinRT with containers. Just like Carbon, only selected Win32 APIs will survive in the long run.
Microsoft would still let you run 16-bit DOS apps on Windows if Intel hadn't dropped compatibility from their 64-bit processors. Both of us will be dead and buried before Win32 goes away.
I work for Microsoft. As you can imagine, this is a problem we have devoted a considerable amount of energy to thinking about. A lot of this discussion has revolved around comparisons to previous solutions (particularly Cygwin and Interix/SUA), but I think it's worth backing up and thinking about what the fundamental limitations of implementing something approaching POSIX compliance in userspace. I will try to tell the story more or less as it is told internally by the people who built these things from scratch.
To be transparent, I am sort of worried that the authors have told us in this thread that they think the number of applications that "really need" `fork` is small, and that the discussions about POSIX subsystems on Windows are overindexed on discussions about `fork` without justification.
The truth of the matter is that the semantics of `fork` infect every API that creates process state. Every library, every syscall that creates process state will have a clear answer for what happens when a process that is using that bit of process state calls `fork`. This includes file and socket management, IPC stuff, threads, signals, and so on. To make `fork` work properly on Windows, you absolutely need to replicate what UNIX does here, or you will break a lot of apps, because make no mistake, a lot of apps depend on these semantics to work correctly. `fork` is not just a function that occasionally gets called and sometimes needs to be fast for people who are calling it. It is core to the semantics of the POSIX API, and if you don't treat it as such you are in for a bad time later.
Perhaps more worrying, though, is that there is a serious impedence mismatch between the UNIX and Windows models of asynchrony. Particularly in the case of sockets and signals, this difference is immense, and I am extremely skeptical we will find a good (or even close to production-worthy) solution in usermode. Maybe these good folks have found something I have missed; to me the approach just seems doomed.
And of course, this is all just the start of the problem. It is a much longer trudge to solve the "real" problem, which is immense. The original Interix POSIX subsystem in NT 3.5 (I'm told) had a `fork` that was just barely enough to pass the 1003.1-1990 validation suite, and fell over quickly if you pushed much harder. But they didn't have to mess with the `fork` implementation to turn that 67kloc into the more robust and conformant SFU 3.x; it was mostly a long tail of things extrernal to fork that just needed to be ironed out.
On top of that, to really have `fork` interop, you would likely need to redefine every Win32 API so it did the "right thing" under UNIXy things like signal delivery and fork, and that is a truly, terrifyingly tall order.
These are some good observations. Note that on modern POSIX, calling anything but async-signal-safe functions after forking in a multi-threaded process results in undefined behavior. I think it's totally reasonable to consider any programs using the WinAPI (not just POSIX functions provided through midipix) as being formally multi-threaded, and to consider the WinAPI non-AS-safe. I'm not 100% sure what midipix is doing in this regard, but my advice on the project (I'm the primary author/maintainer of musl libc, which midipix is using) has been not to worry about making arbitrary functions work after fork, but only supporting the things that POSIX requires to work. My view in general is that fork should be phased out, but posix_spawn is not sufficiently powerful yet to replace all uses of fork+exec -- it can't do advanced uid changes, setsid, resource limits, etc., and posix_spawn can't be used effectively from an async signal context because the API (attributes and file actions) inherently involves allocation.
There are two main scenarios for fork(), and the lack of distinction between them is what I suspect led to some of the articulated worries.
The first scenario, in the spirit of traditional unix, consists of the sequence fork+execve. This scenraio is supported in midipix, yet requires that the (optional) subsystem performs execve on behalf of the forkee. As noted earlier, if all the child does is call execve, then application authors should consider switching to posix_spawn not only due to its better portability, but also for performance reasons.
The second, and arguably more interesting scenario, takes place when the child brnaches away from the parent process, performs (in parallel) a designated portion of a larger task, and then exits. This second scenario has been thoroughly tested and works as expected with respect to IPC, I/O, signal delivery, and nested forking.
In either of the above scenarios, there is no expectation that the child could interact with csrss, create or operate on GDI handles, or otherwise use the WIN32 API directly. Then again, it is expected that a process that calls fork() will not have any of the GUI libraries loaded at the time the call is made. Any limitation here, if any, has little to no relevance to existing applications that depend of fork, but only to what I fondly call the "fetishization of fork" in discussions about posix-to-windows portability. Applications that truly depend on fork were written with neither WIN32 nor GDI in mind, and therefore will not break due to such windows-specific interfaces not being available (as an aside, an X11 application calling fork() from within its event loop is an equally bad design regardless of whether the operating system supports that or not).
Looking forward to further cooperation! I hope this addresses at least some of the questions you have raised.
The biggest problem with Cygwin is that programs linked with Cygwin inherit global state from from a Cygwin installation on the system they're running on. If you want to produce a Windows program that just runs on any system you install it on using Cygwin, it will work right for most users, but if a power user who has Cygwin installed and has their own custom mounts, options (like different binary/text mode settings), etc. tries to run your program, it might break spectacularly. This makes Cygwin a really poor choice for making binaries you want to distribute as standalone programs.
Aside from that, Cygwin tries to hard to be a complete Unix environment on Windows, whereas midipix just gives you enough to use interfaces that were standardized in POSIX as a reasonable, uniform API for all operating systems to provide. Some functions go beyond that, but you don't have to use them. And even some things that are mandatory in POSIX are optional in midipix; as I understand it, you can choose at build time whether you want the overhead of being able to support tty devices (and the associated semantics like job control, signals from the controlling tty, etc.).
The thing that wont work is if you try to mix and match a dll from one installation with binaries from another, but I think you can agree that that situation is fair. You just need your paths setup correctly.
Yes, this is the really big deal that everyone focused on fork and mmap semantics and other details is overlooking. Having midipix as the means of producing Windows versions of cross-platform software like Firefox should make it possible to remove a lot of ugly #ifdeffery and/or whole "portability layers" that are avoiding the standard functions like fopen because of a lack of Unicode support on Windows.
Actually no... the application makes all calls using utf-8, and is expected to provide it in a well-formed manner so that the system call layer could convert it to utf-16. In the reverse route, where utf-16 is read by the system call layer and then converted to utf-8 (getdents(2) and friends), it is expected that file names be in well-formed utf-16. For a file-system volume to have ill-formed utf-16 name entries would make for an interesting case... have not encountered that yet, but will certainly look into that.
This is very interesting, I could certainly use this for porting some of my projects to Windows. But what the website is missing is a "getting started" doc that explains how to set up a toolchain for cross (or native) compiling.
Cygwin is just a DLL for API translation with an associated package manager (Though many people think its more heavyweight than that and consequently dont like it).
Cygwin is just a DLL, and it works reasonably well, but it does come with some nontrivial baggage:
- It's GPL licensed, or you buy a license from RedHat; either might be too onerous for the people who develop midipix
- You can only have one cygwin1.dll in memory at a given time; If you have two cygwin using programs, they must both use exactly the same version of the cygwin1.dll; Which means that you can't just distribute a self-contained cygwin program and expect it to work.
- It's a bit clunky at the edges, with the mounts (it looks like you have a /usr directory on the root, and you can see it with "ls" or cygwin "dir" but not cmd "dir", for example; user integration is a bit clunky).
I'm not sure midipix will be better - some problems are inherent. However, cygwin was designed around Win95/NT4 deficiencies some 20 years ago. It has evolved very gracefully, but it's possible a modern version without all that legacy will work better.
Like you say, some issue are inherent in the problem-space.
but... multiple versions of Cygwin can coexist these days. It did used to be an issue, granted. tho these days quite a few programs distribute their own cygwin dll and tools.
I agree, licensing may be an issue for some people.
Thanks. It's good to know that the multiple version issue has been addressed - although from the FAQ it sounds like there are still a few (unlikely) corner cases one needs to keep in mind.
I'm sympathetic to people who want to develop portable GUI apps across Linux/Windows/OS X. If you're developing something like Inkscape or LibreOffice, most of your potential user base is on Windows and they are NOT going to install a Linux VM just to run your app.
But yeah, there are a lot of people on HN who constantly post about how their work issued them a cat and they have all these tools they install on their cat to make it act more like a dog but it makes a really terrible dog anyway and so cats must be crap. Uh, it's a cat. If you need a dog, go get a dog. Or run a dog on Hyper-V if you need a dog and a cat at the same time. My favorite is when people talk about how their favorite scheme they use to get their dog to catch mice doesn't work for their cat, therefore it's evidence that cats are terrible at catching mice.
>> I'm sympathetic to people who want to develop portable GUI apps across Linux/Windows/OS X.
I am too, but I wonder whether it is even a relevant goal anymore. It's one of those things that has always been a topic of conversation, and has never really happened in a way that a mass of end users has adopted. Everyone who ever ran a java desktop app on Windows back in the day probably has some theory of why, but I think it was mostly because there was never really a strong need on the user side, despite the idea being so attractive to devs.
Now... I don't even know what the world of work is going to look like in ten years... Android? Windows? Linux? Mac? Tablet? Phone? Laptop? All of the above and more, I assume, and the whole interface portability thing seems to be heading down the exact same road that it headed down on the desktop, i.e. either write native or accept some lesser solution.
I disagree there. Just virtualizing means that you now have another machine that you have to audit, keep up to date, have disaster recovery procedures for, etc. Plus, if your potentially trivial app needs to run on multiple boxes, that just amplifies these costs.
Things relying on cuda/opencl generally don't virtualize well, especially not with windows hosts. With Xen you could do GPU passthrough things, but even that isn't easy to setup.
A related project, flinux - https://github.com/wishstudio/flinux - allows you to run unmodified linux binaries on Windows. It is in the process of implementing all the linux syscalls...
I hope passing file descriptors over Unix sockets will work contrary to Cygwin. I really want to use ControlMaster in OpenSSH. Btw. if anyone knows a decent workaround I'll be super thankful as well.
Interix is a lot different because it requires installing a system component, which requires administrator privileges. Midipix produces applications that (at least as I understand it) run on basically any NT-based Windows with no special privileges.
Posix was an API standard, implementable anywhere. The very first POSIX implementation I assisted with, and it ran on CTOS which had a message-passing kernel and built-in networking. Nothing like Unix or Windows.
No word about debugging. How are they going to port, say, gdb? Are they going to implement ptrace, although it doesn't conform to any standard? If so, it also raises the question of WHICH flavor of ptrace?
Also, POSIX has a number of optional features, like realtime and queued signals. Which features exactly will be supported?
The general design allows for building the toolchain in its entirety (libc, libcgcc, libstdc++) independent of the runtime components. The idea behind this is that changes to the toolchain will fairly soon become rare [but see below], whereas changes to the runtime will be very frequent at least in the first couple of years.
Building the cross-compiler requires a native compiler and a shell environemnt that is capable of building gcc. The build process is trivial, and thus far has been tested on several Linux flavors (just asked that this also gets tested on BSD and OSX). Building gcc in an msys/cygwin environemnt is always a bit more tricky, so we have not spent much time trying that. Instead, we are working hard to become self-hosted as soon as possible.
+ building the cross-compiler:
git clone git://midipix.org/cbb/cbb-gcc &&
cd cbb-gcc &&
./cbb-midipix-cross-gcc.sh
yep, that's that. This will use $HOME/temp as a temporary folder, and install the toolchain to $HOME/midipix. Make sure you add $HOME/midipix/bin to your path in order to use the toolchain. As with all other gcc builds, you need to have the usual dependencies on the build system (gmp,mpfr,mpc,libelf,texinfo) and a working shell environment.
+ pre-pre-alpha, radical changes: some major changes to the toolchain are underway. If you find it hard to believe that building the cross-toolchain is that easy please go ahead and run the above commands from a nearby shell, but please also rebuild it in a day or two (look for a commit message mentioning "automatic creation of GOT entries" :))
+ as a kind reminder, cross-compiling is just for building applications; testing them can only be done with the runtime library, which is not out yet.
Almost forgot to mention: the current cross-compiler is based on gcc-4.6.4 since that is the last modern gcc which does not depend on C++, and is therefore easier and faster to build and run. Porting subsequent gcc versions, and likewise clang and cparser, is a high priority, and will follow the initial release of the runtime components.
That's probably the NTFS MFT rather than the actual runtime. Git plops lots of small files on the disk which end up in the MFT. The MFT is really slow. We have the same trouble with SVN.
You can tune this a bit with fsutil and turn off last access time, 8dot3 filenames and change mftzone size and it'll be significantly faster.
ReFS is a little better in this respect. I suspect this was a hang-back from early NT versions which were tuned to larger binary blobs as a normal file statistic rather than lots of small text files.
Very interesting. Do you have a pointer to some deeper info about the required NTFS tuning? e.g. specific MFT size recommendations, and possibly other changes?
If you create a product that runs on client's hardware, you want to be able to run consistently in as many platforms as the majority of your potential customer's. You are leaving money on the table if you don't.
If a subset of those platforms is Linux + (Commercial) Unixes + Windows, the Windows part is going to take twice as long and cost you thrice as much to get in line with the other 2... unless you have some clever solution like the one proposed here.
For one or the other reason, fork() has become in many discussion of posix on windows something of a fetish. The interface surely has its place, and figuring out how to efficiently implement it took a huge amount of effort, yet the vast majority of applications do not truly need it. Matter of the fact is that even on linux, where fork(2) is natively supported, the sequence fork+execve is more costly than clone+execve (where clone's flags are CLONE_VM && !CLONE_THREAD). For additional reference, see for instance the implementation of posix_spawn in musl libc (http://git.musl-libc.org/cgit/musl/tree/src/process/posix_sp...).
If high performance of the posix layer were not possible the project would have not existed. Among the factors that make high performance possible are 1) direct use of kernel interfaces (aka the Native API, where most of the runtime layer is written as a user-space driver), 2) utf-8 as the primary supported multibyte encoding as a foundational concept rather than an afterthought, and 3) tls implementation that matches in speed the native tls facility.