His thoughts on internal tools and support are very insightful. It's why I am typically sad when a company outsources its support.
When a company does its own support with internal personnel, it has a vested interest in making the customers happy and also serving them as efficiently as possible.
When a company outsources its technical support, there is no real incentive for the outsource partner to let the company know how the processes and tools can be improved and automated. In fact, quite the opposite. If doubling the business means double the support costs, that works out great for the outsourcing partner.
Perhaps it's still possible to outsource the support work, but still care deeply about the details of how that work is done, and how the process can be improved/automated with systems. I haven't seen it work that way, however.
I spent a few years at Apple writing internal tools.
IMHO, this is the way to have an effect on a company completely disproportionate to any other activity I know of. The tools I wrote at Apple have enabled projects which, AFAIK, are completely unheard-of at any other tech company.
That said, don't expect a payoff proportionate to the effect of the tool.
From a company owner's perspective, however, excellent tools can provide an advantage that is difficult for competitors to match. That's worth an awful lot.
Yup, I've done this as well at current and past jobs. I'll usually just invent a new project, it gets the boss's attention and they like it, so I get to work on that rather than the normal boring stuff.
I like much of the advice, but the style and presentation of the message is, like so many things I've seen associated with Facebook, lacking any sense of humility. There's no "we did this because it worked for us and we're sharing in hopes that it may help others." Instead it's "Other people think this. They are wrong." Or "A commonly held belief is X. It is false."
This hubris has certainly been a powerful ally to Facebook's founder, but like so many other powerful people, companies, governments and organizations that came before them, I can't help but think it will ultimately lead to demise (unless tempered).
While hubris can be a problem, I'm very much a fan of stating your opinion clear and bold. One might be wrong, but astute readers should be able to judge ideas for themselves. Speaking clearly will also make it much easier for people to see if you're wrong.
I think giving clear statements of your opinion and being proven wrong is much better than obscuring your point with "maybe" and "personally" and "IMHO". That's depending on the context of course, as you don't want to unnecessarily offend people (which is bad in general, but also counterproductive). But in a general analysis like this, not attacking particular people's shortcomings? Go for it.
A reminder to "think for yourself" can be helpful, and does not necessarily imply a lack of clarity or boldness. Sometimes a story is told so well, that you get emotionally wrapped up and forget to take a step back and make a fair-minded analysis. Not everyone is an astute reader.
You also included some conditional statements with, "depending on the context" and "But in a general analysis". I see no problem with acknowledging that things aren't always black & white.
I have to single this out: must we all prepare our rhetoric for the remedial education of those who were never made self-sufficient in the processing of external knowledge into internal beliefs?
So the problem of "hiring the best" is solved by making hiring your top priority. Likewise the problems of development are solved by making tools your top priority. Presumably along with everything else that is a top priority, like making something insanely great.
I'm sorry, I just sensitive to how many times I've seen "top priority" in somebody's management presentation, as if it solves something.
That said, I mostly agree with the gist of what he's saying here.
I know what you mean, but you may be surprised to know how many people feel like their organization doesn't have any top priorities, so overkill on the hyperbole isn't necessarily a bad thing. One of the issues in large organizations is trying to engage staff strongly enough so they really do feel a personal responsibility for the success of the company. It's hard, and very few large corporations do it well.
I agree this is a real problem. I guess I've come from organizations where everything is declared a top priority, therefore nothing is. And nothing can be pushed aside for a true priority.
In particular, it is damaging for a leader to declare a top priority and then go on the same as usual, not making extra efforts to invest in that priority, because of course the reality is that so many things are important, and they lack the will to focus at the expense of something else.
If something is that important, you'll direct significant resources to the goal and I imagine the author of the article is sincere in doing that. Many organizations however are resource-constrained and must make tough choices. It's just mismanagement to declare lots of top priorities - heck, you don't have to say anything, just show us how you are allocating the money and people, and adjusting your expectations of everybody's schedules and deadlines.
With all due respect to Facebook and many great engineers at Facebook, if you look at quality of Facebook API ( documentation, bugs, reliability, compatibility), I don't think they followed "hiring the best" in the division working on Facebook API.
I disagree. The fact is that Facebook does not optimize for documentation, reliability or compatibility. The reason is because their goal is to be as agile as possible, and iterate extremely quickly. It's true that this makes life terrible for developers, but that's not a problem because Facebook does not have a shortage of developers. Over the past 3.5 years the Facebook platform has evolved so quickly that a lot of bugs simply disappeared by deprecation; if they had spent more time on solid architecture they would have left a lot on the table business-wise.
Also, I'm not sure if you used the Facebook API when it was new, but I remember in 2007 being distinctly blown away by how functional it was and how far ahead of the curve they were on web APIs. You can argue that "the best" engineers wouldn't churn out something so ephemeral, but that's subjective.
"You will begin to get the (objectively) best candidates"
If "it's [everyone's] job to say no-hire" when they're "not sure" about a candidate, I'd like to know what is done to avoid systemic bias in hiring decisions.
Anyone here from FB able to comment? How diverse is the work force, especially compared to applicants?
Hi, I'm the author. Someone alerted me that this showed up on HN, so I figured I should come by to clarify.
I believe my statement might be clear. What I mean by that is "it's [everyone's] job to say no-hire," as opposed to anyone feeling like "well, I'm not sure about this candidate, but I'll let someone else reject them because I don't want to be perceived as mean." One problem with hiring is that everyone must be willing to uphold the standard - if you have too many people who are never willing to be the one to say no-hire, the standard will fall - everyone has to be willing to take the responsibility for saying no.
That said, systemic bias is an orthogonal (and real) issue. Being "not sure" should be applied to "does this candidate measure up" not "is this candidate a lot like me?"
Saying "no hire" can be hard for some people, but becomes easier if your team adopts the attitude that "maybe means no". It's probably better for the team to miss out hiring some promising "maybe" candidates than later trying to fire a bad one.
Good to know. Does Facebook collect data about how they're doing internally to avoid these systemic biases? Is this tracked?
Although it appears orthogonal, if everyone starts making hiring decisions without any checks and balances that an HR person would put in, I'd assume more bias to creep in. At the very least, I'd want the "standard" to be clearly defined.
One of the things that strikes me about what I've heard about hiring practices is that they all seem primarily focussed on keeping the wrong people out rather than on finding the right people.
Does anyone consider the possibility that there might be really capable people who don't fit the model that N developers have of what a good candidate should be ?
Here is a personal anecdote on the cost of not hiring a right person.
I have a very strong background in low level (embedded as well as device driver) software development. I have a long and visible track record in U-Boot development, including being a sub-repository "lieutenant".
Canonical advertised for a "Ubuntu Mobile Developer - ARM" developer for which I felt (and still feel) I was a very strong candidate. I submitted my resume and got no response. I did some searching and found the contact within Canonical who was responsible for the position and got an immediate response when I contacted him.
I did a phone interview with a Canonical employee, not the person hiring but another who was a technical expert. I thought the interview went well overall, but he seemed disturbed that I had not actively used Bazaar even though I had used other DVCSes, including evaluating several of them for U-Boot - I ultimately recommended git for U-Boot. I've also used Mercurial extensively in my job, so I would say I have more than passing familiarity with DVCSes.
Anyway, I didn't "make the cut" with Canonical. I've since gotten a new job and am no longer in the market, but Canonical is now advertising for three software engineers for that position http://webapps.ubuntu.com/employment/canonical_SE%20PS%20HB1....
The kicker? Canonical did not have "familiarly with uBoot" [sic] as a desirable qualification for candidates before I applied. They added it immediately after I entered their interview process. In other words, Canonical did not even know they were looking for me before I applied, discovered that at least one of my qualifications were important because I applied, and yet did not hire me. A year and a half later, not only have they not filled the position, they now have three copies of the unfilled position.
I once worked for a company that hired the way Facebook does and it is a recipe for disaster in the long term. Two things brought an end to the practice but it was too little too late:
1.) Employees were agreeing to hire people who weren't as skilled as they are because they were afraid of newer and younger competition. As each generation of new hires were brought in they were successively worse than the previous. The most senior employees saw the degradation of overall talent level and eventually left for greener pastures. The company slowly ended up becoming composed of people who were barely qualified for their positions and several years after I had left they were bought out for pennies.
2.) They were sued for discrimination, it was a ridiculous accusation, but the company had to settle. The reason was that without consciously trying we had all hired people who were of similar backgrounds to ourselves in many regards. Apparently "One of our team members wasn't sure about her" doesn't cut it in court.
Sure there are. The standard rationale is that when hiring, the cost of a false negative is far lower than the cost of a false positive.
I guess it's also true that the cost of a false negative is unknown and not felt. It would be a really extreme case to be forced to say, "Hey, remember that woman we didn't hire two years ago? She went off and founded Company Y that's now eating our lunch." With a false positive, you feel the pain of cleaning up their messes until after they're gone.
Finally, sad to say, the majority of job applicants are not competent to do serious engineering work. A negative bias is right most of the time.
Finally, sad to say, the majority of job applicants are not competent to do serious engineering work.
I've often heard this claim. I have no data for denying it.
However I have a suspicion that when people make this claim the observation that it is actually based on is that "the majority of job applicants aren't very good at answering programmer interview questions.". They are hence making an implicit assumption that not being good at interview questions implies not being competent to do serious engineering work. I think this assumption is highly suspect, because of the extreme differences in stress levels, time scales and general context between interviews and actual work situations.
Strongly encourage applicants to contribute to any open source project. Reviewing this code will tell you more than any stupid programming question. This takes less time and can be done asychronously. An interview can be used to determine if the qualified applicant will fit in your culture.
And if they haven't contributed any open source code, then whack them with stupid programming questions.
The processes section intrigues. As the "CTO" in a 4-person startup, I'm constantly juggling between getting $hit done and documenting what the heck I did to get $hit done. Seems to be a fine balance.
He's wrong. You should not focus on tools, focus on people and ideas. The people will then make the tools (and throw them away when they cause too much friction) as needed.
But Facebook is still young, they're still learning. Unfortunately they don't seem to be learning from the past, which means they get to repeat everyone else's mistakes.
I think you're both in agreement. I believe the point about tools was not to enshrine any particular tool, but rather, to create a culture where internal tools are viewed as being the most important technical contribution.
This is backwards from many technology companies where internal tooling is often given a lower priority and the work assigned to less skilled engineers. This is often done with noble intent, like to try to create the best value possible for customers via improving customer facing features. But I think that proves a short term strategy rather than focusing on enabling your organization as a whole. You'd be surprised how many resources become free for feature work when your tools and infrastructure are so good that you don't wast effort on what could be automated or prevented.
This comment is correct. My intention is not to prioritize tools over thinking about anything else. If you read the essay linked from it, I said "your most talented engineers should be working on your tools." So when I say "highest priority," I mean development priority, i.e. you shouldn't have your best people working on features and your second-tier people doing tool work, like many organizations do.
You might like to short-circuit a lot of this learning.
There was a fighter pilot who studied this for 40 years named John Boyd. He architected the f15, 16, 18, A-10 and the first iraq invasion. His academic work short circuits what all the dotcoms (including facebook) are slowly learning through trial and error.
If you want a starting place Robert corams book on boyd is excellent, while Frans Osinga's thesis Science, strategy and war is essential reading for the hard subject matter.
Sorry I responded too quickly.
People,
Ideas,
Hardware (technology)
In that order are what you should focus. Near 100% of managements time should be spent on people and ideas. Get that right and the people will take care of the technology.
One tool to implement that is a mission order (No civilian counterpart), which is where you define an intent to be achieved and operating parameters to be met but not how that intent is to be carried out. parameters are mutually agreed on - and the subordinate has the right to refuse. Usually you leave most of the definition of the intent up to the subordinate. Like "build a database useful for storing graph data" or "Find a way to improve ads customer acquisition".
EFAS culture - An implementation of blitzkrieg for business gleaned from interviews done in the 70's implements this nicely.
No, he isn't. Tools need to be a high priority, or else they end up being crap. Amazon, Disney, Northrup Grumman, SAIC... all of the big companies I've worked for had the same problem: their tools were terrible.
At most of them it just meant that HR ended up wasting time dealing with crappy tools, which is no big loss, since HR is a usually a waste of oxygen anyway. At Disney and Amazon, it meant that the people producing consumer-facing content wasted a lot of time fighting the tools that were supposed to enable their jobs, and a lot of developers had to waste a lot of time supporting the tools that weren't doing their job.
For example, just two and a half years ago, one of the buyers at Amazon showed me how she set up the relationships so that when you look at an iPod, you get a collection of compatible accessories. She had to create relationships for EVERY permutation between each accessory and for each accessory to the iPod. By hand. In a spreadsheet (Excel, I think.) Then she uploaded it to the site, and every once in a while a relationship would mysteriously fail, because of another relationship managed in another tool that she had no access to conflicted with it.
There is actually a problem when organizations are focusing too much on tools.
In order to efficiently address a disruptive innovation, a corporations need to change their processes and since tools are connected to processes, these good tools make that change even harder.
Yes, it good to have good tools for developing the current product, but organization needs to be ready to eliminate internal tools without hesitations.
I agree -- becoming too attached to existing tools is a major drag industry-wide. Even if it's well-designed in the beginning, which from what I've seen is extremely rare, it tends to devolve into a mess given enough time.
To get maximal value out of the tools, everyone has to be willing to toss them when they start to becoming a big enough impediment. (Deciding on that isn't always easy, of course.)
Perhaps it's just word definitions you and the author are disagreeing on.
Making the organization more efficient is a big win, we can probably all agree. Making the people more efficient through improved systems is a big win, we can probably all agree. Letting the people who perform the process figure out how to make it better is a vitally important part of that, though, as you point out.
But making tools and systems improvement a priority doesn't necessarily mean that we're going to elevate the systems to the point that they become sacred cows even when they get in the way of getting things done.
It is a case of letting the best people determine when a new tool will speed up a process and having the environment where they can take the time to write the tool.
Too often the environment is hostile to what might be a risky investigation. I've spent a week investigating a tool to potentially save four weeks. I failed, as I hadn't sufficiently understood the complexity of the problem. But ultimately that failure led to a better understanding. I changed the way we carried out the task, saving us a week anyway.
My project manager at the time thought that week was a total waste of time. (Thankfully he was later "let go" when 360 degrees of people who had contact were saying the same things). If I do that task again, I will have a go at v2 of the tool, and probably get it right.
Too many people think tools are a waste of developer time. It appears that the best tools where I've worked have been created in developer's free time, and then used widely by the team, almost to the shock of the management. Only then do the developers get a little work time to maintain or even improve them.
But yes, you can go the other way. In the largest organisation I have worked in, there was the "infra" team. Their job was to provide the environments, from source control to db instances (Oracle ones), test environments (multi VM set-ups), etc. However their removal from the actual development team resulted in tools and processes which took about as long as for us to do these things with our own tools... the same ones they'd taken to improve.
A number of good responses about writing good tools. That's a no brainer.
It's scary to how many project managers I've had to explain why I have allocated 20% of my project planning to writing a new tool. They see it as wasted time because they don't think that we will be redoing the task again... when it's something sales try and sell with every project?!
The biggest thing that struck me were the strong statements about technical managers. I think the sentiment is correct, technically experienced managers are great. But it reads like you expect all technical managers to be up to date on their coding? Or just be competent at writing pseudo-code? Hopefully it is the latter!
When a company does its own support with internal personnel, it has a vested interest in making the customers happy and also serving them as efficiently as possible.
When a company outsources its technical support, there is no real incentive for the outsource partner to let the company know how the processes and tools can be improved and automated. In fact, quite the opposite. If doubling the business means double the support costs, that works out great for the outsourcing partner.
Perhaps it's still possible to outsource the support work, but still care deeply about the details of how that work is done, and how the process can be improved/automated with systems. I haven't seen it work that way, however.