"He has changed the way engineering teams are structured, largely eliminating testers to speed software releases,... "
What could possibly go wrong?
Ok.. the sentence continues :
"and adding data scientists and designers to the engineering teams to ensure all features are informed by rigorous testing and good design principles. "
I don't quite understand, how can data scientists help to catch bugs? Automated testing?
This is probably why so many of the Windows Updates this year broke critical things, especially for enterprise. I feel Nadella is taking a "cheaper, faster" approach which is fine and dandy in the otherwise non-critical mobile space (mobile users are used to being treated like crap and having buggy releases) but in enterprise there's simply a higher standard of quality.
I'm not sure what this means for MS. At our shop we now have to overly-test all patches and then wait a minimum of 30 days to see what happens. This has saved our bacon several times under Nadella's MS when before we only had to do pretty casual testing of patches. This also means a lot of shops are no longer rushing to get security patches in because of his negligence and corner cutting. That means a less secure internet for all.
I really wish someone would step in and show Nadella that MS shouldnt just be an Apple-lite and that catering to enterprise should be MS's main goal, especially considering that's where almost all its revenue comes from. I'm sure playing Steve Jobs-lite is fun for Nadella, but its about time he grew up and started running with the big dogs. Frankly, I'm sick of the focus on mobile and other popular technology taking away from the core competencies that made some of these companies great. The recent actions and changes on OSX are disheartening as well. Or how the steam behind the Chromebook is more or less dead as Android eats the world.
Buying and updating an MS product shouldn't be a gamble. They're supposed to be the conservative and stable big brother to the industry. Ironically, its only now under new leadership that we have so much more incentive to move to FOSS products. Its the same amount of headaches, except with FOSS I have no licensing worries. Under Gates and Balmer, the commercial MS products were less headaches and delivered a fair amount of value. Now just installing an Exchange rollup is asking for a complete reimaging. We don't even bother with service packs anymore. Might as well roll out a new image with that cooked in. The fail rate on those is unacceptably high.
edit: why the downvotes? this echoes the sentiment at places like stackexchange and /r/sysadmin. testing at MS has gone to hell and everyone knows it. burying my comment does nothing to stop that reality. Its well known Nadella personally redid the QA and testing parts of MS and those outcomes have only gotten worse.
>Buying and updating an MS product shouldn't be a gamble. They're supposed to be the conservative and stable big brother to the industry
You can tell who's young around here. In the distant past you could have put IBM or Novel in that same blank 20 years ago. It wasn't till around the release of server 2003 that anybody thought that.
Next, every product is a gamble these days. If you're connected to the net, you're at risk. Before exploits may have taken weeks or months to fully circulate, now 0-days are very common and other vendors release the exploit information they have on your products in 90 days or less.
>Now just installing an Exchange rollup is asking for a complete reimaging
Again, I'm not sure when Exchange didn't explode in the past either.
My personal experience is that MS has been solid since Windows 2000. I have limited experience with the NT4 products but I remember those being just fine as well (i maintained two NT4 servers for my employer and used to host images for fark photoshop contests on one- shh).
Exchange has been SUPER solid. Its only 2010/2013 that we're seeing so many issues with what should be a trivial update process.
I think there's a real drop in quality lately. Heck, in the XP days we never bothered to test security updates. They all came in same day via WSUS. Nowadays? No way.
NT was even more solid before then, at least after the first 3.1 version which I don't think I did more than kick the tires of. The drop in quality really started after SP1 of NT 3.51, the 3rd major version.
Didn't Exchange have problems in the early part of this century, or earlier, with its database being easy to bork and slow to restore?
Your comment harmonizes with the word on the 'street' from in-person conversations from MS employees & people who know them. Testing as a role evidently has been disintegrated and QA turned into SDEs in general. I believe the goal is to force quality onto the developers responsibility instead of making it QA's responsibility.
We'll see how that plays out; it sounds like the first iteration has been brutal.
Disclaimer: MS employee/don't speak for the company etc; Yes, SDET has been (at least in my division and those I talk with) dissolved and integrated into SDE.
I raise an eye at the parent-parent post not because of the impact this will have (certainly some of the concerns stated here echo strongly with me) but that it's been a relatively recent change and I'd be skeptical that you can attribute any recent failures to this specifically, things just don't move that fast.
As to seeing how it pans out, it's certainly up in the air. As a former SDET there's a big wall to climb of making testability performance security etc. part of the first line of thinking alongside design and implementation.
I went through this in 2006 or so - SDETs were turned into SDEs overnight in my part of MSN. Some survived, I didn't - one thing it did was completely discount the skills of a tester. The other SDEs were not expected to be up-to-skill on testing, but we were stacked against all the other SDEs.
Those are good points worth reading. What you describe is very much the feared outcome of removing SDETs before SDEs are ready to deliver tested and solid products in the new model. I have personally not suffered any troubles, but your experience does not surprise me one bit.
I think the problem for a business like MS is that, like you said, most of their revenue comes from enterprise. They already dominate that market. It's not enough to tread water - they have to grow! How can they really grow in a market which is definitely not chomping at the bit for change and are slow to adopt new products (even windows 7)? Where else can they grow? Apple is making so much goddamn money right now it's crazy - MS has to be thinking of ways to get a piece of that if they want to grow.
I guess for example that the http://support.microsoft.com/kb/3004394 Win7 fiasco comes from the update being only tested on machines with test roots installed, right? And Win8.0/Server2012 seems to be treated as second class with more than one bug affecting only that one, which reminds me of the odd support lifecycle where the server edition was treated as an older release but the client edition get treated as a older service pack.
It's a confused sentence, but I think it means he removed "testing" as a separate group of people, and made "testing" something inherent to the design process.
In other words, Microsoft is doing what Google and other companies do: have individual teams responsible for everything for their project(s): design, development, devops, testing. One team has responsibilities for a vertical slice of a system: build it, deploy it, be responsible for it.
Agree totally, poor writing. It seems that Microsoft, under Mr Nadella's stewardship, are moving towards a more iterative design approach, which seems to be working well.
They are not eliminating testing. This was really bad wording. When I interviewed in Redmond with Microsoft, I was told that Microsoft used to hire SDE's (software development engineers) and SDET's (software development engineers in testing) as two distinct positions. Now, if you are hired as a "software engineer", you are responsible for writing software as well as writing tests. This was done to decrease friction between what used to be the "software development team" and the "QA testing team", and hopefully lead to faster software releases.
A developer can test their own code, but they will only test it through the paths that they are aware of and thus coded for. An independent tester needs to be there to explore different avenues and try things that a developer wouldn't have thought of. Developers do think very differently to users.
But if you're working on software that actually matters, you're supposed to have some friction between the people building it and the people testing it. This is otherwise known by terms such as "independent testing" and "lack of bias".
Moving fast and breaking stuff is OK if you're building a web service that doesn't really matter. If Facebook falls over, the productivity at a significant number of businesses probably goes up.
On the other hand, if yesterday's OS security patch stops everyone's PC from booting this morning, or if a new version of your e-mail server has a glitch that means half your company can't log in to read their mail or even that important messages from customers are actually being lost, that probably comes with a cost measured in large numbers of dollars per minute at a medium-sized business, or if you prefer, in careers per hour within the IT group.
there is a separate, smaller "quality" team with a separate reporting hierarchy that is supposed to address this - that's where the reference to data scientists, etc. in the article comes from.
basically the idea is to split the SDET role into two - SE testing (closer to the code than SDETs used to be) and "quality" (further from the code than SDETs used to be). SDET testing, though not directly driven by the code structure, tended to be at a pretty low level driven by a detailed functional spec.
Software engineers should be able to test their own code. The faster they get feedback on their code and its impact to users, the better they understand the various cases and can strive for better quality. Having a separate QA team has a fairly long feedback cycle.
Ideally you'd get both engineers writing tests for their code as well QA teams for products.
Software engineers should be able to test their own code.
Indeed, but there is also a good reason that professional editors have new writing proofread or even technically reviewed by people other than the original author.
Maybe there's some subtlety that I'm not picking up here about Microsoft's previous set-up and/or the model they've been moving to under Nadella's leadership. We seem to have some past and present Microsoft developers around these parts, so perhaps their comments will clarify this. But as reported, this looks to me like a retrograde step, an attempt to be more agile and fast-moving rather than something likely to improve the quality of the finished product.
What could possibly go wrong?
Ok.. the sentence continues :
"and adding data scientists and designers to the engineering teams to ensure all features are informed by rigorous testing and good design principles. "
I don't quite understand, how can data scientists help to catch bugs? Automated testing?