But Linux doesn't have the concept of a "subsystem", that's pretty much a Windows NT exclusive (inherited from its core predecessor OS/2). User mode like WINE is the correct approach on Linux.
It's pretty wild that Microsoft actually managed to reactivate the subsystem concept for (first generation) WSL after it had been dormant as an 1:1 boundary for so many iterations of Windows and even wilder how quickly (and elegantly) they switched to a completely different abstraction for WSL2. This switch was mostly motivated by cross-environment filesystem performance afaik, so it might be perfectly reasonable to take the other approach (an actual NT level subsystem) for the android use case: the guest apps won't do much file I/O outside of the subsystem's dedicated playground because of how Android is discouraging free-form filesystem use.
So I see the historical NT subsystem story restated all the time and that makes little sense. WSL1 used little (if any?) of the historical NT subsystem concepts and WSL2 even less.
SFU and its successor were, but that was a completely different architecture, never intended for binary compat. The short lived and of limited scope OS/2 subsys was for binary compat, but given the origin and NT and the similarities of some techs it is not surprising. MS-DOS support was way more a low level and VM thing and the bulk of it not that much different from the Dosemu approach.
The classical NT subsys was more of a userspace thing than a kernel space one. And Linux is already way more userspace neutral than the NT kernel is, anyway. You could do a Win compat userspace on top of the Linux syscall if you wanted. Of course it is heavily inspired by Posix, but NT syscall are heavily mapped to Win32.
> The short lived and of limited scope OS/2 subsys was for binary compat, but given the origin and NT and the similarities of some techs it is not surprising
They could have used a more classic NT environment subsystem architecture for WSL1, it would have mostly worked. The primary reason they didn't was performance, and to a lesser extent compatibility.
Unix systems create lots of short-lived processes, because processes on Unix are relatively cheap. By contrast, NT processes are much heavier weight. This is part of why the POSIX subsystem / Interix / SFU / SUA and Cygwin too always have had performance problems. Introducing a new lightweight process concept, the picoprocess, helped get over the performance issue. But these new lightweight processes can't be allowed to make arbitrary NT syscalls, which rules out the classic NT environment subsystem implementation pattern - have DLLs which make NT syscalls instead of e.g. OS/2 syscalls.
If they'd followed the classical Windows NT environment model, they would have enhanced NT to be able to load ELF executables and then shipped their own libc.so which made NT syscalls. That would have worked fine for most apps which don't make Linux syscalls directly, but would have had worse performance than WSL1 had. It also would have meant more work for them, since they'd be emulating not just the syscall layer but the libc layer too. (They could have sped things up by reusing the GNU libc code; to do that, they would have had to have open sourced most of the code of their subsystem, which would have probably been a positive overall, but I'm not sure how feasible that would have been given internal Microsoft politics around open source.)
To support the minority of apps which make Linux syscalls directly, bypassing libc, they needed something in the kernel to intercept the SYSENTER/SYSCALL/INT 0x80. They could have redirected it back to user mode for handling in their libc.so. That would have worked, it just would have been slower than WSL1 is. (And WSL1 already has enough performance issues, that performance was one of the main motivators for switching to the new architecture in WSL2.)
There are tons of programs that do not go through "the" libc under Linux, everybody and their dogs seem to reimplement their own locks recently for example. Plus the libc actually makes a way greater surface to cover given it gives you access to most of the syscall plus other functions.
Plus you still want all the fine semantics of Linux (mmap, cow or at least cow-like, etc.) and for tons of calls the libc is a trivial layer anyway, so you are not gaining much. The layout of processes also. At this point you are back to a big part of picoprocesses again, because you don't want the classic NT layout, nor cb to userspaces, etc.
Maybe would have been more of a practical idea for a BSD, but even then the remark about the surface of the libc API vs syscalls apply.
But Linux has a stable API, so it does not make much sense to do Linux compat without re-implementing it and given you can do just that, well do just that.
And you rightly note at the end that syscall support is needed, but at this point the whole architecture would have made absolutely no sense: if you are going to implement syscall eventually, just implement them in the first place, and not more code.
Also I doubt the only issues of WSL1 were performance. I doubt they would have been able to follow the upstream quickly enough, at least not without a pretty big dedicated team, and it seems difficult to justify. The WSL2 approach leverages techs that are used for many other things.
> There are tons of programs that do not go through "the" libc under Linux
I agree with you there. This is a big problem with using the classic environment subsystem model.
> Plus you still want all the fine semantics of Linux (mmap, cow or at least cow-like, etc.) and for tons of calls the libc is a trivial layer anyway, so you are not gaining much.
One potential advantage to the environment subsystem model – it might have pushed them to enhance the NT API to be closer in feature parity to the Linux API, which could then have benefited Win32.
> I doubt they would have been able to follow the upstream quickly enough, at least not without a pretty big dedicated team, and it seems difficult to justify.
Well, one relevant difference is that Linux is much more of a moving target than POSIX or OS/2 are. POSIX is a formal standard, and changes in formal standards are slow, they take years.
Back in the 1990s, OS/2 was a moving target, with IBM coming out with new versions with new features (2.x, 3.x, 4.x). But, Microsoft had decided they only wanted to support OS/2 1.x, and to ignore all new features in OS/2 2.x and higher, which made Microsoft's task much simpler. And even back when IBM was actively developing OS/2, they couldn't attain anywhere near the pace of change which is happening with Linux. Linux has far more people working on it than IBM ever had working on OS/2; and IBM, certainly back then (I don't know whether it is still true today), was notorious for slow, cumbersome and overly bureaucratic development practices, which was another limit on OS/2's velocity.
> But Linux doesn't have the concept of a "subsystem", that's pretty much a Windows NT exclusive (inherited from its core predecessor OS/2)
Mainstream OS/2 doesn't have "subsystems" in the sense that Windows NT does, so it is mistaken to suggest that Windows NT got the idea from OS/2.
Something like the NT "subsystem" concept is in the abortive IBM Workplace OS which became OS/2 PPC edition. Workplace OS is based on the Mach microkernel and designed to run both Unix and OS/2 as "personalities". But this was never released for x86, and the PPC port was only ever released in beta form and abandoned before GA.
I think Windows NT more likely got the idea from Mach than mainline OS/2. Mach calls them "personalities" not "subsystems" (although some people will use the word "personality" when talking about NT). If you look at the internals of macOS/iOS/Darwin/XNU, the Mach personality concept survives in highly vestigial form – XNU has its core Mach APIs (Mach personality) and BSD APIs (BSD personality). In principle Apple could add some third personality (e.g. Windows personality, Linux personality, Android personality, etc) but they probably never will. (CMU Mach was always intended to support multiple OS APIs, but the CMU Mach developers only ever implemented a BSD Unix personality, and Workplace OS's OS/2 personality was one of the few cases of anyone ever implementing a non-Unix personality on top of Mach; the BSD personality of XNU is a descendant of CMU Mach's original BSD personality, but updated with a lot of newer code from FreeBSD and also of NeXT/Apple's own devising)
NT drew a lot of influence from VMS, and VMS also has "subsystems", although VMS' subsystems have little in common with NT's other than the name. Other operating systems also have subsystems – for example, the Job Entry Subsystem (JES) is a major component of IBM's mainframe operating system z/OS (formerly known as MVS), but z/OS subsystems and NT subsystems have little in common other than the name as well. "Subsystem" is popular as a generic term for an OS component.
Also, as other respondents have pointed out, WSL1/WSL2 are not subsystems in the classic Windows NT sense.
I don't agree with you on that. (I am not the person who downvoted you, however.)
Both z/OS and Windows NT use the word "subsystem" to refer to OS components, but the commonality is purely at the terminological and most abstract conceptual level, I don't see any commonality at the level of technical details, and I see no evidence of any concrete technical influence from z/OS on Windows NT.
"Subsystem" is a highly generic term used to refer to an OS subcomponent across many operating systems – z/OS, OS/400, OpenVMS, Windows, Linux, among others. The meaning of the term varies widely, and what counts as a subsystem isn't always precisely defined. In z/OS, "subsystem" is a quite precisely defined term – it is something which is installed by calling the "IEFSSI REQUEST=ADD" macro, and which then is displayed in the output of the "DISPLAY SSI" console command. By contrast, in Windows NT, the term "subsystem" is never clearly defined – for example, is Client/Server Run-Time Subsystem (CSRSS) a subsystem or part of the Win32 subsystem (a subsystem of a subsystem)? Is the Session Manager Subsystem an integral subsystem or a part of the integral subsystem? I'm not sure there is a clear answer to these questions, whereas the question of what counts as a subsystem on z/OS is quite clear – is it visible in the "DISPLAY SSI" output?
(Slightly complicating the issue is that z/OS has both subsystems and functional subsystems, which are different things despite the similar name – subsystems implement the Subsystem Interface (SSI), functional subsystems implement the Functional Subsystem Interface (FSI).)
