For a detailed account, there's always _Show Stopper!_ by G. Pascal Zachary which is a shot at writing a _Soul of the new Machine_ style book about Microsoft during the creation of NT. If that sort of story entertains you, I recommend it.
Can someone tell me what is so great about SotNM? (It even won a Pulitzer.) I read a bunch of computer history books last year, and these were all more interesting IMO:
- John Gernter, The Idea Factory
- T.R. Reid, The Chip
- Stephen Levy, Hackers
- Katie Hafner and Matthew Lyon, Where Wizards Stay Up Late
- Michael Hiltzik, Dealers of Lightning
- Susan Lammers, Programmers at Work
(Actually I read Levy & Lammers a long time ago, but they're both better than SotNM too.)
Those were full of themes like patents and broad use of technology, interaction between government and private enterprise, monopolies, private research institutes and the need for profit, challenges commercializing your discoveries, the culture around early computer use, etc. They had profiles of famous computer pioneers. They told the history of tech I use every day.
SotNM didn't have anyone I recognized and was about a machine I'd never heard of before. Its biggest theme was how overworked the engineers were (also present in those other books, but not as dominant), at the cost of their health and marriages, with little-to-no reward. It was monotonous and depressing.
So what did other people appreciate about it? With books I don't expect to always "see" everything there on my own, so maybe someone can help me learn what I'm missing.
I actually like SoTM the best out of the list you've posted. More so than the others it's a study of the everyday people of technology, not just the heroes that we've all heard of. Dealers of Lightning, Hackers, and Where Wizards Stay Up Late all do a fine job of telling human stories of technology you use, but SoTM is mainly about the day-to-day life in an industry, the triumphs and failures on a much smaller, more human scale. A similar book could have been written about companies I've worked at, and I'm sure everyone here can say the same. It's less of a history and more of a study of people.
I enjoy this genera though, and I'll add Gertner and Lammers to the list. You may want to try Exploding the Phone by Phil Lapsley (about phone phreaking) and The New New Thing by Michael Lweis.
Thanks! I'll rethink SotNM some in light of that. I did enjoy the parts about "signing up" and the software/hardware interaction (although I wished for more technical details about that.) I didn't think it was bad btw, just not as rich as the others. Also after I wrote that comment I thought, "Monotony isn't always bad. The Iliad and Symphony of Sorrowful Songs are monotonous too. Maybe the form is expressing the long slog of the project." :-) Thanks also for the new recommendations!
SoaNM is an extended portrait of a single product development process/team under severe competitive pressure. The specific technology being developed isn’t the point (though it has to be explained for context) — it’s an attempt at capturing some archetypal characters and narrative of how technology is developed. It’s sort of like Silicon Valley, except instead of satirizing the outrageous nonsense that takes place in the Valley today, it takes seriously the mostly-serious work of development.
When I worked at Apple in the 90s, if someone asked “what is your work like?” I would give them that book.
Just wanted to say I really appreciated this answer and it does let me see the book in a new light. It's interesting how many of the others I listed are not in that kind of competitive environment!
I found SoaNM interesting because I worked on Data General minis, both the 16-bit ones and the 32-bit ones. You might find it interesting to go back and look at the assumptions Data General worked on in light of what happened within a few years. The design team originally wanted something VAX-like, e.g. with an instruction set designed to make assembly coding easy--RISC made that unfashionable. I also found it interesting that the need to preserve the original instruction set within the new meant that the 32-bit minis were stuck with four general-purpose registers and no register-relative byte addressing.
As far as I know, all that was left of DG was bought up by EMC for the sake of DG's Clariion SANs.
I'm actually in your camp :). Reading SotNM wasn't so much more interesting for me and I'd exactly same thought why this book is so much talked about. May be its clever title? The thing is that I've seen this story played out few times over during my career and I invariably leave feeling "ok, so what's new? I could have written that book too.". May be its my lesser ability to appreciate art and leaning too much on consuming just facts. On the other hand I found Showstoppers amazing, gripping and very informative. The scale is very different and that kind of landscape isn't experienced by everyone frequently.
Showstoppers is a really good book, not just about NT and it's history, but in my experience it gives a good overview of how such a large scale project looks from the point of view of everyone involved, including the CEO and people working on some specific modules.
Both those books are excellent. I first read Soul of a New Machine back in the 80's when I was a spotty young apprentice Data General engineer learning how to repair Nova's and Eclipse S-series machines.
I remember reading it about 10 years ago and found it quite fascinating. It goes into a level of personal detail on people involved, not solely technical but also tells what happened on a personal and emotion level. Pretty insightful in understanding how NT was built.
There two books that people rarely talk about so these are hidden gems:
* Hard Drive: Bill Gates and the Making of the Microsoft Empire
* Barbarians Led by Bill Gates
You really have to get past the title of second book. It's not really about bashing Microsoft or Gates but account of thirteen-year Microsoft veteran Marlin Eller that you perhaps never heard of. That makes this book all the more interesting and give real insight on how things were at lower levels in early days.
If you're interested in the design and implemention of the NT kernel, check out David Solomon's Inside Windows NT. The book's technical level is roughly at that of an undergraduate class on operating systems, going a little deeper than this article.
"Microsoft's internal project name for the new OS was OS/2 NT, because Microsoft's intention was for the new OS to succeed OS/2 yet retain the OS/2 API as its primary interface."
Right... There's another reason why it was originally called OS/2 NT: IBM had hired Microsoft to co-develop OS/2 with them before Windows NT development had begun (as early as 1985). The reason for a resemblence to OS/2 with the name and API compatibility would seem to be so that Microsoft could develop the NT product as "OS/2" which could be charged back to IBM. So, effectively, IBM was unknowingly paying for a chunk of NT development. By the time IBM realized they were getting worked over it was too late, they had already been taken for a ride by Microsoft. They lost in two ways: they lost the the desktop market to Windows which shipped pre-installed on nearly all consumer PC's, and paying for dev work on OS/2 1.3 which really was going into NT product). So IBM took OS/2 development back in-house in 1990 and licked their wounds, but it was too late by then. Except for the embedded market, OS/2 had lost.
Well, it helped that a Microsoft employee discovered a way to make virtualization work with their ongoing Windows 3.x design instead of having to go with OS/2.
It has been several years, but I think the story is described here "Unauthorized Windows 95", I no longer have the book.
> a Microsoft employee discovered a way to make virtualization work with their ongoing Windows 3.x design instead of having to go with OS/2.
That's sort of right. Larry Osterman goes into more detail here, but the gist is that Microsoft figured out how to run (mostly) unchanged Win16 apps in protected mode. This was released to market in 1990 and immediately turned into the best solution to get at more than 640K RAM.... which had been a latent and mostly unresolved issue for over five years. (So lots of pent-up demand.)
To some extent, _every_ Windows program on Win95 was a 16-bit program. (There was a lot of the 32-bit API that was built by thunking down to 16- bit equivalents.... one notable example was that Windows95 notepad was still limited to 32K files, etc.)
At least on the IBM side, they were careful about IP rights.
Summer of 1995, I had an internship (as a tester) on IBM's OS/2 LAN Server Enterprise team in Austin. This was well after the IBM/Microsoft 'divorce', but IBM's LAN Server product still had large volumes of code written by Microsoft and still displayed a Microsoft copyright message alongside the IBM copyright.
At that time, one of several other initiatives at IBM was a longer term project to remove all the Microsoft code from LAN Server. To accomplish this, they divided their developers into 'Clean' and 'Dirty' groups, where cleanliness was defined as not having ever seen the Microsoft code. It was then only the 'Clean' developers that could contribute new replacement code.
(The 'Enterprise' project was also interesting in and of itself. To help make the product more scalable, IBM was replacing all of the default directory/etc. services with a new version of DCE they were also developing for OS/2. 22 years later, it all seems pretty archaic, but it was fun at the time to be a bit involved.)
Also Mark Russinovich was the author of many life saving SysInternals tools and utilities[0]. He was also the author of the original Windows Internals books.
A persistent rumor is that Microsoft bought Winternals partly because Russinovich knew the guts of Windows better than a lot of their own engineers did.
According to the Wikipedia article, it looks like more of an embrace and extinguish move, at least regarding the utilities sources and Linux development.
"However, since the Microsoft acquisition, none of the utilities currently available are accompanied by source code, and the Linux versions are no longer maintained or available.
Some of the coding tricks used were based on the Windows Native API (NTAPI), which was (and still is) mostly undocumented by Microsoft. Using these coding examples - with source - would enable developers to create extraordinary programs that performed operations that would otherwise have been impossible using a standard API. Examples include hiding Registry information, intercepting or hooking APIs to monitor file operations by the OS, as well as Registry operations."
Hmm...random person on the internet adds citation free conspiracy theory with the description of the edit as:
"Describing the non-availability of source code since the Microsoft aquisition. Under 'Source Code and Technology' header (my experience as a developer)" [My emphasis in italics]
I'd say the grandparent comment is correct when he says that "Russinovich knew the guts of Windows better than a lot of their own engineers did".
Russinovich was on the team that re-architected Windows from the ground up for the modern era.
>A few weeks ago in Los Angeles, Microsoft technical fellow Mark Russinovich -- absolutely the world's leading authority on Windows performance and architecture -- took time to explain to developers attending PDC 2009 in Los Angeles exactly what [MinWin] is. In summary, it's a way to graft onto Windows some semblance of the architectural layering it should have had, if its architects in the 1980s had any foresight into how Windows would be used thirty years later. It enables current and future Microsoft developers to evolve new configurations of the operating system, without having to rewrite core services or worry about breaking dependencies between those services and upper-level APIs.
I know about that one where the source was taken down. My favorite is Skype however where they used to use SYSENTER directly and read your BIOS using NTVDM: https://www.pagetable.com/?p=27
My high school had a VAX in the '90s that served a whole lab of terminals and was available over dial-up. A lot of my early programming was on that machine. Later I bought a VAX off ebay, joined DEC's VMS hobbyist program and provided free accounts to people at HobbesNet (https://web.archive.org/web/20050125003015/http://hobbesnet....). I kinda regret getting rid of those machines now. It looks like the Deathrow cluster is gone now too (https://deathrow.vistech.net/410.html).
I bought a couple of VAXStations on eBay in 2004 with the idea to preserve the "VMS experience". I got one of them up and running with the then current VMS version, and connected to the Internet. They're still sitting in my office, not powered on in 14 years.
I have two VS 3100s that I keep intending to install VMS on. I used VMS in my first year of college, and there's been nothing quite like the versioning file system to come along since.
Just a note for those unfamiliar with this hardware, the VaxStation 3100 38 (the model I own) was produced in 1989. My configuration has 24MB of memory (Max was 32MB). The 32-bit CISC processor runs at 16Mhz. I don't know how it compares to contemporary workstations, like those from Sun or SGI, but compared to PCs at the time it was much more powerful. These workstations must have cost tens of thousands when new. I bought mine for $10 each at least 15 years ago, sans hard drive.
What I find interesting is that PC-DOS (MS-DOS) was also a mostly clean room copy of CP/M.
The guy who wrote it before Microsoft acquired it literally went through a CP/M programming manual and recreated the system calls for x86, changing a name here or there.
Neat parallel to how NT doesn't copy any VMS code, but still essentially copies VMS in several aspects.
There was a nice analysis of this done by Bob Zeidman where he compared the released CP/M 2.0 and DOS 2.0 source code. His basic conclusion was that source and basic commands were not copied, but that a large number of the system calls were copied.
And the new MS-DOS 2.0 APIs were based on Unix, sometimes down to the name (for example INT 21h/AH=42h is named LSEEK, and the value you put in AL matches exactly the "whence" argument of the Unix syscall lseek)
"trn" on Lobste.rs has been posting the company's slide decks about their progress. A recent one, "Proof Points to x86 Boot," shows they're getting close:
See "History of the Port" in bottom right for older posts. You might want to do it bottom-up going from oldest to newest. Also, a few things trn posted indicated they were doing non-x86 work like Alpha releases, too. Example below:
I remember messing around with that ages ago. I wonder if any of the contributors ended up working with VMS Software on the port. It looks, from their site, like they are getting close.
My first job was working on RSX11M (predecessor to VMS) on a PDP-11. One thing that I remember is that the virtual address space of a process was 64K (i.e. 16 bit address) but the machine had 256K of RAM. I never used VMS as by that time I'd moved from minicomputers to microprocessors.
Around 1996/97 I heard two very similar rumors, from two people who did not know each other. Both rumors had it that the NT source at that time was at least in some part the source of "Micah", the OS that Dave Cutler and crew had been working on for the Prism architecture. The NT source still had DEC copyrights and mentions of Micah in it. That's why DEC had such a weird deal with Microsoft, and why there was an Alpha AXP port of NT long after all other architectures went by the wayside.
As far as credibility of this rumor, if you have a copy of "Inside the Windows NT file system" (https://www.amazon.com/Inside-Windows-NT-File-System/dp/1556...) by Helen Custer, you'll see a more than passing resemblance of NTFS to ODS-2, the on-disk layout of VMS filesystem. G. Pascal Zachary has a big chapter on the development of NTFS, if I recall "Showstopper!" correctly, but he doesn't mention this... coincidence at all, giving full credit for NTFS to a couple of guys on the NT team.
Novelty perhaps? A quick and dirty Algolia search suggests this story is the most popular one ever with "VMS" in the title. On the personal side, Dad wrote VMS scripts for work.
They didn't copy directly, they just re-implemented from memory. I worked for Digital right out of college at the "DEC West" site off 520. A number of my co-workers REALLY hated Microsoft. Swore up & down they had colleges who had seen NT source code when they started working together with Microsoft on "Wolfpack" server clustering that they were using the exact same terms & constants (e.g. structs for system values were identical). Dave Cutler & his crowd were not popular. That $60M - $100M was payoff money to avoid a very public & very nasty copyright suit. And back then, $60M - $100M was considered real money.
Did they really? I thought MS just implemented their own version of the tech and was accused of violating patents. Did they actually steal IP?
Reading the Wikipedia article, it's not really clear what happened. Looks like MS review Stac code during licensing talks, but there's no mention of accusations of code theft. Then there's discussion of patent violation lawsuits and payoffs going both directions.
That's a little disingenuous. The "recipe ideas" were from the same minds that came up with them originally. This comes down to who really owns an idea, the entity that paid for it or the mind that thought it.
I wonder how relevant the NT kernel is nowadays. My impression is the kernel as a foundation is that it is a thing of beauty. Unfortunately, the windows structures atop it are eternally collapsing and rotted in parts.
With the introduction of the windows subsystem for linux, the whole edifice is coming down, threatening to bury the NT kernel and render it obsolete.
It seems Microsoft has capitulated and competition has been reduced to "'Doing Linux' better than linux". Which is to say, "We surrender! There is no plan. Start migrating away from our legacy tech; here is a path."
What's the point in continuing development on and for Windows if it no longer offers (and demands) technological distinctiveness? There is no point.
It seems just the opposite to me. The Windows Subsystem for Linux seems a testament to the NT Kernel and its deliberate approach from nearly three decades ago. Designed originally to support Win32, POSIX, and OS/2 frontends, for a long time it only ever really needed to exercise one of those things. Now it has the best POSIX layer it has ever had, focusing on byte-for-byte compatibility with binaries targeting the Linux kernel. That's incredible power, and the idea that the NT Kernel is doing it almost like running any other application is fascinating. If the NT Kernel is "Doing Linux better than Linux", that's a fascinating statement, because the NT Kernel is also running Win32, UWP, and all of its own chores at the same time. What's the Linux kernel doing? ;P
Sure, Win32 is a beast now past retirement, and Microsoft has been trying to help that retirement happen sooner rather than later with UWP, but again, that seems a testament to the underlying NT kernel, because the NT kernel doesn't care, it's still chugging along doing its thing and doing it well. It's also running on more hardware combinations and more processor architectures than most other kernels could dream of touching.
Sure, there will always be legacy Win32 apps. I work with too many VB6 applications to be more optimistic about that.
UWP is closer to being a distinct subsystem than "running atop Win32". Though it isn't an official subsystem, it has some of the hallmarks of one (and that was one of the complaints about Windows 8). They've been successfully blurring the lines between UWP and Win32 since Windows 8, but it's also not entirely accurate to state that UWP "runs atop Win32". (The old .NET Framework does, yes, but I didn't mention the .NET Framework, I mentioned UWP.)
If anything, with increasingly more of the Windows Shell moving to UWP, it may even start to be increasingly the case that it is Win32 running atop UWP.
COM itself is quite small, sure there are a lot of components that implement COM and use Win32 but I don't think it would take much to port COM itself to another platform.
COM itself as a format/standard is Win32 agnostic other than being built on some similar data structures to Win32.
Also, WinRT isn't COM. It's very like COM, "son of COM and .NET metadata" and all that, but WinRT and all the WinRT components don't use Win32 and never have, and even have a quite clean break with all of the Win32 data structures COM has embedded in the past.
(Though, I suppose some of that starts to get into the semantic weeds of whether or not you consider DirectX a part of "Win32" or not, as for instance XAML is built on DirectX and much of DirectX is provided as WinRT components, sort of. There are certainly a lot of Win32-adjacent things that live in all worlds like DirectX, but whether or not they can be considered Win32 is an interesting philosophical debate.)
NT pretty much only runs on x86 and ARM (yes, yes, 32- and 64-bit variants of both). yawn
Almost ever other OS out there allows for more. Linux, (Free|Open|Net)BSD, illumos, AROS... hell, even ReactOS has had ports to architectures Windows has never dreamed of :)
The Xbox 360 has a Windows NT-kernel and IIRC it runs with a PowerPC-based CPU. Older NT-kernels were available for PowerPC, MIPS and Alpha. Your point is still valid, though.
> If the NT Kernel is "Doing Linux better than Linux", that's a fascinating statement, because the NT Kernel is also running Win32, UWP, and all of its own chores at the same time. What's the Linux kernel doing? ;P
Microsoft, the former juggernaut, would say "Linux, who?", and then proceed to throw their sales and marketing weight around.
> a testament to the underlying NT kernel, because the NT kernel doesn't care, it's still chugging along doing its thing and doing it well.
It must be advantageous for Microsoft, at least in the short-term, to open themselves up and decrease friction between themselves and the Linux community, especially on the server-side where Linux won, than to innovate and compete on Microsoft-centric features and functions - to create something new that's more compelling.
Also, Microsoft risks seeding their own destruction when they create compelling new technologies like XMLHttpRequest that, once appropriated by their competition, reduces the compelling nature of any other technologies Microsoft created for platform lock-in. I can understand innovation fatigue setting in for management when they have enough institutional XMLHttpRequest stories to go along with the endless failed software project investments and evaporating markets.
I assign meaning to the decisions and directions that individuals and companies take. Microsoft paved a gilded path for application development that points AWAY from the Windows platform. Yes, there is novelty and a definite technical achievement there that one can appreciate. But I choose to focus on the motives rather than the technical achievement, and I am disturbed by what I perceive is Microsoft's capitulation in the consumer software/platform arena.
If Microsoft is soon to be the SAAS and Cloud company and not the Windows/Office company, then the software world is worse-off because competition is good.
> Microsoft paved a gilded path for application development that points AWAY from the Windows platform. Yes, there is novelty and a definite technical achievement there that one can appreciate. But I choose to focus on the motives rather than the technical achievement, and I am disturbed by what I perceive is Microsoft's capitulation in the consumer software/platform arena.
Most paths support two-directional travel. The ancient adage that "all roads lead to Rome" seems to apply. Rome built a lot of roads to travel away from Rome, but nearly everyone realized that all of those roads also lead back to Rome. That does seem to be something of an intent in Microsoft's strategy here: if Windows is great at reaching users where they are, then Windows is great at bringing users back.
WSL makes it extremely easy to build on/with/for Linux, without leaving Windows. It worked for Macintosh to bring BSD users to Macintosh. It seems to working for Microsoft that WSL has given an option for previously Mac-or-Die web developers to give Windows a second chance. There have been plenty of headlines here on HN of previously Mac-only users delighted by new Surface devices. Obviously anecdotal evidence, but signs do seem to point that new roads are leading to Rome.
I've been using WSL since it was first released on the Insider Fast Ring. It's been great. Sure, its had it its problems, but the team is really responsive to issues filed on github. Ive had all of my WSL issues fixed and usually in a new build in 2-3 weeks after filing.
WSL has also been a savior in that I no longer need a Linux VM at home. Which is "great" because virtualization support (at least for me) has been broken for months. I can't even boot my desktop with Intel virtualization enabled in the BIOS.
It just happens that Linux kind of killed the commercial UNIX offerings, with Aix and HP-UX the last ones standing.
Everyone else just happens to want to use POSIX like tools, so GNU/Windows is no different than running a Linux VM on top of an IBM z/OS or IBM i mainframe.
Microsoft is cleverly going after the users that Apple doesn't care any more, the ones that weren't interested in Apple platform to begin with, they just want a pretty UNIX to run their POSIX command line tools.
At the API and system service levels the reason for all that goofy cruft is that businesses still use it, a lot, and are willing to pay fairly big bucks if and only if they get to keep it.
Adding Linux features is just more cruft added to the kludge. Sells more units, particularly in Azure, not particularly part of some sort of strategic direction then, say, the Distributed Transaction Coordinator.
The UI and OS-level shovelware stuff on the other hand is a nightmare that might kill the not-quite-golden-anymore goose.
Still, it is more organized as desktop stack than any UNIX based alternative, with exception of macOS, which doesn't care much about their UNIX underpinnings, even during NeXT days.
