This story made me think. Until now I had the impression that mediocrity at large corporations is inevitable no matter how many smart people they employ, simply due to the size of the corporation. For every talented, competent employee they have 10 average or below-average employees. Even at Google, even (and especially) at Microsoft. Competent people cannot choose who to work with so whatever extra they do is lost in the mediocrity that surrounds them.
After reading this article I have realized that even at large corporations if you make isolated and largely idependent small teams of highly competent people that can select their members themselves, you can create "startup" bubbles inside the organization that can deliver outstanding results. Maybe this is the way to go when growing large, from a medium-sized enterprise to a large corporation?
It's a mistake to think that corporate mediocrity follows from employee mediocrity. On the contrary, a significant amount of bad software is made to work by the efforts of a large number of rather smart people.
A more nuanced view might focus on how corporate culture, a short term focus on sales due to stock market pressures, and the inevitable lack of accountability that seems to shield the management chain from harm provided they continue to take no risks around innovation combine to create some weird effects. Look in particular at how some big software providers make acquisitions, which are then polished and re-branded as a new version solution and then shoved into a customer pipeline.
These customer pipelines are the real magic - no matter what junk gets shoved down them, nine times out of ten, junk gets paid for. Nobody on either side cares. Takes too much personal risk in the big corporate world to point out that the emperor wears no clothes.
Growing small teams of highly competent people seems to naturally lead to duplicated effort and turf wars as well. Don't think that a team of magic snowflakes will outshine all the "drips" hanging on . . . I've spent too much time in big corporate IT to not realize that most people walking the halls in these places are damn smart. They're just hamstrung by the corporate structure. But - this is why small startups have an advantage. They're not nailed down by convention.
One problem is when external forces bring in incompetent management. Being part of such a team may be a great experience, but you are always at the mercy of higher ups. Your one real point of leverage is to walk out the door. If the people managing you don't respect what you bring to the table, you lose even this. And if they don't understand it, which is what often happens with "interchangeable" management, they are unlikely to respect it. Then, reflect that there is the potential for this disconnect at each level of management. Start counting the levels that exist in your organization.
If you are smart, you will develop an informal network of contacts/relationships throughout your organization. And keep your resume up to date. (I think that smart people develop such a network not only or even primarily out of a calculated decision to manage their career, but because they find interesting what other people know and do and struggle with and solve, and helping them.)
Perhaps this is one reason social skills (of a certain sort) tend to predominate in large organizations. Given the situation, they become the most essential tools to a career.
Also, regarding social equanimity, a large part of corporate life is "not rocking the boat". At lower ranks, this can imply a strong uniformity of treatment. Your increase will be between 0% and 4%, regardless, because those are the absolute limits that have been chosen. If someone were to receive 10%, there would be a lot of jealousy and unrest. The only way to larger increases is promotion. Promotion changes one's duties. Ergo, certain duties will never be compensated beyond "industry standards". The social structure caps their compensation and so, minus the individual who 'self-sacrifices', their effectiveness.
And this is, in part, where your "average or below-average" coworkers come from. Management won't sweeten the pot, even if, as the posted story implies, going after the top of the talent pool produces far greater returns than the additional investment.
I guess this comment is kind of thrown together. Large social structures tend to enforce norms and drift towards a lowest common denominator. Exceptions occur, but over time, they are fighting the tide.
I've had some good managers and some bad ones. The difference for me has been whether they add-to or reduce my set of concerns.
A bad one will make you aware of every microissue their boss has to worry about, and makes you code in some terrible things in order to please them.
A good one acts as a firewall between their boss and their subordinates. Bad ideas from above are pushed back (as much as they can), lots of independent (and usually fairly random) requirements are combined into something logical, and my resulting problem set is something reasonable to create, test, and maintain.
A boss with programming experience will be able to negotiate a deliverable that's internally consistent enough to make for a good product. They do some of the up-front thinking to reduce the problem down before it's even given to me.
A nontechnical boss just sees a spreadsheet of checkboxes that we have to complete. They're like a simple IO controller: send requests, wait for completion.
I would go as far as to say there really are only two things a lower- to mid-level manager should focus on. First--and this is by far the most important thing--she should make sure her subordinates have the best possible environment to do their jobs. The second responsibility is hiring the right people. With these two ingredients in place, whatever she does on top of this would actually harm the performance of her team.
What usually surprises me about large organizations is how little power the managers have over these two points. A manager in charge of internal applications development may negotiate the functional specs with the business, but have no influence whatsoever over the hardware that's supplied to the developers.
This had little to do with competence! The team was successful because it grouped together some people who actually enjoyed their job. Individually, they were only slightly better at it than their peers; as a group of "like-minded individuals" (quoted from the article), they spent much more energy collaborating on ways to improve their software testing process.
"Like-minded" is only tangentially related to intelligence or competence or any of the other traits that arrogant people like to ascribe to themselves. It's really about people that have similar priorities and styles and work well together.
If you can sufficiently insulate a group from the corporation in general, sure, you can do better than corporate mediocrity. For a while. But that's harder and harder to do. Particularly for decent stretches of time.
Consider: it would be in the best interest of any project manager to steer their project away from the Black Team. They aren't incentivized to ship good software, but to ship on schedule. And should avoidance be impossible, they absolutely will point corporate HR at any and every aspect of that strong team, to break-up, slow-down or demoralize them.
Similarly, with higher-up management. They'll need certain projects to be treated with kid gloves. So they'll scheme and conspire to either defeat the team and ruin its effectiveness or defeat the process, so that the team's effectiveness is rendered moot.
It's the perverted reward structure in large corporations that causes most of their problems. There's no lack of people trying to fix those problems. But any solution (including this one) will ultimately be defeated so long as that reward structure remains.
Big businesses often avoid that by giving separate profit & loss responsibility to each division. That way, each individual group competes on the marketplace, and incompetent groups can't hide by breaking up competent ones.
It's an open question how to adapt this to "infrastructure" groups that are shared by the whole organization, which may explain why there're so many horror stories about corporate IT and HR. I guess technically QA is also an infrastructure group, so it's surprising the Black Team survived for so long, but I guess the improvements they made were visible enough that they had senior management's protection.
I think the Black Team's longevity had more to do with the corporate reality of its time. Corporations in the 60s and 70s still valued investment that provided benefits over the long term. So shipping quality was still important.
Corporations today are worried only about the next quarter, or what things will look like maybe twelve to eighteen months down the line. But certainly no further than when the options vest and management exits.
And that's why divisions don't help. Because defeating the process to ship crap today has a better effect on this and next quarter's P&L, versus increased spending testing/refining the project for no real increase in revenue in the short term.
Down the line, when product quality would become known and helping the product sell better than it would otherwise, those responsible for making the decisions will likely be gone.
(They'll definitely have planned on being gone by then.)
it would be in the best interest of any project manager to steer their project away from the Black Team
So shipping quality was still important.
I'm largely in agreement. This line from the start of the article stood out:
Customers do not pay for defective products.
Now, you really don't want to be the manager who has customers refusing to pay due to bugs. If the Black Team can spot any problems well in advance of the deadline, all the better.
However, if customers are happy to pay for stuff that doesn't work (eg, where the people who make buying decisions don't talk to the people who will be made to use the software - quite common is my impression) then sure, stay a hundred miles away from the Black Team.
I think the important part is to allow these groups to be self-selecting. In other words, the smartest people should gravitate towards each other if they are allowed to. For a really interesting take on how to do this take a look at: http://en.wikipedia.org/wiki/WL_Gore_and_Associates#Gore_Cul...
I couldn't find a good source for how many employees they have, but its in the thousands, so they have been able to make this work on a large scale.
They talk about them in (I think) Peopleware. It's part of a segment talking about how fostering a unique culture for your team can increase motivation. They mention that this culture persisted in the Black Team after all its original members had moved on. I'm not sure how long it continued on after that.
If you haven't read Peopleware you really should. It's one of the best books on managing people working in IT around, still. If you like Spolsky, DeMarco and Lister were talking about this stuff years ago.
It's been in and out of print on and off for years, I buy a few copies whenever it's back at the regular price. I've given away 6 copies now to various people.
Edit: I tried searching for anything related to IBM and their Black Team. It seems Google saturates my search request with everything linking back to this article.
Having worked in similar environments in the early 90's on desktop software, there are some serious downsides to this approach.
Basically, it can lead to having a rift between developers and QA, and in a big corporation, it can become a bureaucratic mess and a 'us' vs. 'them' attitude between groups. I've been in long heated meetings between QA and Dev arguing over individual bugs, not so much on behalf of the customer as opposed to face-saving exercise.
From the article: "And the things they did to software went beyond all bounds of rational use testing and were more akin to software torture." In a perfect world with unlimited time, it's great to find bugs through contortions but usually there is some costs associated with it either with QA not focused on other areas that matter, or having a developer focused on confirming that bug as opposed to working on something else.
Sometimes it's great in a bigger organization to have a unique culture so I agree with the jist of the article and to other HN comments. For instance, I worked at a company on a smaller Macintosh team that fostered a healthy competition with the much bigger Windows team which worked well. I'm just not so sure it works well when people are essentially working on the same project where interfacing with each other is a requirement.
I think a crucial reason this worked is that there was a measurable objective, the value of which was universally understandable. Find software bugs. This is how they measured and selected the initial team, and this is how the team itself could prove its effectiveness in a management-compatible, objective manner and therefore defend its existence. Finding other areas where this could work is an interesting excercise.
I agree that measurability is important, but there are plenty of measurable objectives that don't inspire this sort of espirit de corp exhibitted here. I would love to know more about the leadership of this group, and if the leadership of the group fostered this sort of group identification. Is there a leadership lesson here, or did this emerge organically?
You could make the same arguments in every industry that exists, somehow software is the only one that gets away with it. Even our counterparts in hardware are held to a higher standard.
Djikstra and Hoare are the only ones who tried to advance a higher standard and yet beyond naming an award for one and singing praises of the other, the field has continued to accept lesser standards.
Desktop computers have tens of millions of lines of active software, and for most users they only exhibit a bug every week or so. By any measure, that level of reliability far exceeds any other industry. Chipmakers achieve 0.999999 reliability, which is impressive, but if software had only that level of reliability it would crash every millisecond.
Setting aside all other factors (complexity, progress made, resources available, etc), I would argue that the culture of bug acceptance has lead to more bugs in software than would have existed had it been less tolerant.
I'm trying to puzzle out the logic behind chipmakers' products being less reliable than the software that runs on them. Does the weakest link in the chain not break first?
They're different units. When chipmakers claim six-sigma (about 0.999999 reliability) they mean that when you order a million chips from them, on average 1 will be bad.
It's not apples-to-apples, but if you look at the number of lines of code executed before a noticeable bug it's in the trillions.
You could make the same arguments in every industry that exists
You could, but the argument would be false there. Building a car, constructing a house, sewing a shirt, writing a book, etc. are all not significantly more complex than they were decades ago. Software is orders of magnitude more complex than decades ago.
You would be incorrect - having worked in the industry for a short while, I can attest that modern automobiles are at least an order of magnitude more complex than something from, say, the 60s.
They still operate on the same mechanical principles, but we exploit these in vastly more complex and efficient ways. If you want to make this argument, then you also have to accept that computers are "just" current running across silicon.
No, the reason why we get away with this is because replacement is easy. You get a car right the first time because a large recall can cost in the hundreds of millions, and be logistically impossible to execute. For us it's a simple "Hey! We has update! Yes/No?" dialog box.
"modern automobiles are at least an order of magnitude more complex than something from, say, the 60s". I could believe a car today is even two or three orders of magnitude more complex than a car of the sixties (100 times or a 1000 times more complex).
But that's compared to Moore's law that a computer becomes an order of magnitude more complex roughly each eighteen months for twenty or more orders of magnitude increase from the sixties.
Also, modern cars do have lots of bugs in sense of suboptimal behaviors. They just cannot fail utterly without people being pissed.
We're not talking about silicon here - we're talking about software. There's no doubt that the complexity of silicon has increased by leaps and bounds in the decades since the 60s - but what about the code we write to drive them?
The complexity of code - not the compiled binary - the text you punch into the machine and what it semantically represents, has not really gotten that much more complicated over the years. We've introduced several new paradigms since the COBOL mainframes: object orientation, functional, to name a couple. It'd be hard to argue, though, that it follows Moore's Law. Not even close.
> They just cannot fail utterly without people being pissed.
This is an important point: people who expect flawless behaviour from software because other fields of engineering demonstrate it, IMHO, are misguided. Aircraft engineers work incredibly slowly because the consequences of fucking up is perhaps thousands of deaths, and billions in liabilities. Car engineers are the same on a lesser scale. There's no need to expect flawless, 100% perfect function when you don't need flawless, 100% perfect function.
We could spend 20 years developing the perfect toaster that will never, ever burn your toast. Or we can spend 2 months on something that will get it right 97% of the time, and just move on with our lives.
In addition, it's not as if those other activities actually exhibit many fewer mistakes than software does. Cars get recalled all the time for small things that slipped through the design and testing phases. Your average book has a number of typos on first printing. The difference is that cars are usually designed with enough redundant features that they can deal with minor flaws and book typos inconsequential enough that people are willing to accept them without any coaxing.
Software is different not just because it is more complex, but because computers are much more finicky than roads and people. A logic flaw in software can cost millions; an equivalent flaw in a book will likely never be noticed.
Depends on the job. I've seen some very unfortunate PC deployments (most in the 90s) that axed perfectly good 3270-based systems in favor of clunky, crash-prone, button-and-wizard GUIs that were so bad, employees had to go back to paper-based workflows to get their jobs done.
Terminal-based systems often have a very steep learning curve, but I've seen many cases where they were better designed and met business needs better than COTS replacements.
Granted, those terminal-based systems were probably phenomenally expensive when they were put in (mostly in the 70s or early 80s, that I saw), and I suppose it's possible that after 15-20 years we have a biased sample that only represents the best of the breed, but they got the job done.
If by "green screen" you mean 3270 terminal, then ... yes, but it would be pretty hard. The networking hardware for a 3270 is pretty expensive. (I was going to say "getting pretty expensive," but in reality SNA has always been expensive; it's just that other networking hardware has gotten dirt cheap by comparison.) And, of course, you need some sort of actual backend computer to attach the terminal to.
But sure, if you had a terminal, a spare IBM system (the iSeries is probably the lowest end) set up to support SNA, and then a networking controller to attach the 3270 to, you could probably get Lynx working.
Alternately, if you just want the effect, you could just pick up an old green/black monochrome display and hook it up (via a physical adapter) to any VGA card. Buggy implementations excepted, all VGA cards should be backwards-compatible to MGA.
Add a nice IBM Model M keyboard and you'd be all set. I suspect someone has probably done this as a case mod before (if not ... I might); stick a modern mobo in an old IBM PC case with the original monitor and keyboard.
As long as the problems are matched to the capabilities of the language, I don't actually care about what the language and capabilities are.
I judge how good things are by the human element. That hasn't changed. The software industry is as inclined as always to engage in death marches, shortchange quality, seek imaginary silver bullets, and devalue experience. In short it isn't fundamentally different at all.
maybe I'm just feeling negative today, but this sounds like some bullshit to me. it sounds like the kind of thing that would be appealing to managers: having "not exceptionally intelligent or talented" employees who become completely obsessed with doing their jobs. great, you can pay them average salaries and they will be so loyal to your giant company. the story and the events themselves are fine, but I feel like the reason this story (and some of these details seem pretty apocryphal to me, like programmers crying because of bug reports) is so popular is that management types are taking the wrong lesson from it: that it would be desirable if employees were unhealthily obsessed with doing their jobs. then they try and manipulate their own people into doing the same thing.
"great, you can pay them average salaries and they will be so loyal to your giant company"
The point is that they become loyal to each other, the company reaping a reward is just a side benefit of people working towards a common goal.
I was on a similar 'elite' skunkworks dev team in a company of ~ 35,000 that had executive support, and the results were insanely good. You can't manipulate people to do this; You can however give them the freedom to make it happen.
I've had the opportunity to work on much smaller 'elite' teams before (7-10 people), and if I didn't have to eat and pay rent I'd do it for free.
The amount of productivity, support, and "you grok me"-ness is so extreme that it's really a wonder to behold. As a regular dev now I really do look back on those days and wish I can do it all over again.
I really, really hope that the team that's in charge of Apple's app review process doesn't start emulating these guys, with legal things instead of technical things. Though it seems like they kind of already have.
After reading this article I have realized that even at large corporations if you make isolated and largely idependent small teams of highly competent people that can select their members themselves, you can create "startup" bubbles inside the organization that can deliver outstanding results. Maybe this is the way to go when growing large, from a medium-sized enterprise to a large corporation?