I've become very cynical since the past year or so because there is a lot of noise/articles on the net, all knowing what's best for you or wrong with what you are doing. Yet without context or true authority to talk about it. My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.
Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case. Startup principles don't apply nor academia. Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars.
>My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.
The thing about critique is that you don't need to have made an equivalent project/effort/artwork/whatever to be able to critique it with a valid objection.
You just need to be able to spot an actual issue.
Sure, some might think they've spotted an issue when there is none. Or not understand that the issue exists for a reason (e.g. engineering tradeoff).
But those are orthogonal to the subject.
The key is that critique can come from any source, and it doesn't need someone to have done something "equivalent" or to have specific credentials to be valid.
The latter can at best serve as a rough filter to throw out some noise. But they will throw a lot of valid critique as well, which can come from everybody, including a layman.
> You just need to be able to spot an actual issue.
My problem with the article is that it's extremely hard for humans in general, and critics in particular, to accurately model the alternative. I think that's the heart of what the commenter is talking about.
Unless you've worked at Facebook's scale, it's really difficult to tell what happens. (I don't work anywhere near that scale, so maybe take my opinion with a grain of salt.) This critic is reading signals and attributing them to a failure on Facebook's part, but who knows if an alternate scenario (a) actually makes things better, or just results in different trade-offs; or (b) is even feasible for a company of Facebook's size to do. I have no idea. There are only a handful Facebook-sized enterprises that have ever existed.
My problem with the article is that it's extremely hard for humans in general, and critics in particular, to accurately model the alternative.
The lack of a visible solution does not make all criticism less valid. Criticism can be an expression of flaws without including a drubbing for not doing better. It's often made public as combination of the two, but the later can be disqualified without impeaching the validity of the former.
Even if the criticized thing is the best possible expression of the goal(s) given all the other constraints, it is still right to accept and express that limitations exist. -- Pretending they don't is just an ugly attempt to prevent the critic from refining their criticism to include the goals and self-imposed constants they view as causing the flaw.
It's nearly impossible to read this article without receiving the impression that the author is heavily implying that there is indeed a better way (vague insinuations of slowing down and focusing on quality).
Sure, but the problem I (and I presume others) have with the author not presenting an alternative or an even an acknowledgement he that doesn't necessarily know of one is that the message loses a bit of credibility. I think the author knows that code quality isn't actually the most important output/outcome for Facebook, but as far as I can tell, he does not relay that.
In short, he states that Facebook has a code quality problem, one which most of us would expect of an organization of that size, but is it really a problem? Would their bottom-line be better with higher-quality code that took longer (higher cost) to develop? Make that argument. Convince us why code quality is a problem and how improving it would make sense to all stakeholders (not just engineering). I think this message resonates with an audience of software developers (myself included), but it doesn't pay attention to all of the competing interests.
I think he looked at publically available data and drew his own conclusions in line with his expertise. All valid. Feynman never built a space shuttle (I believe) but was instrumental in identifying reasons for the Challenger disaster.
To OP's point, at what point do we consider the Chesterton's fence effect? Pointing out code quality problems is of course easy when the effects of those quality problems are obvious, but as OP mentioned, Facebook is working at a scale that isn't likely familiar to most critics. Sometimes, when things get big enough, you have to do things that nobody else is doing, because nobody else has the problems that you have.
If all you've ever built was houses, it's hard to give too much credence to your critique of a skyscraper, especially if you aren't privy to the underlying geographical features like bedrock depth, wind velocity at altitude, etc.
> If all you've ever built was houses, it's hard to give too much credence to your critique of a skyscraper, especially if you aren't privy to the underlying geographical features like bedrock depth, wind velocity at altitude, etc.
This analogy isn't apt. We're talking iOS app development versus Facebook's iOS app development. This is a very easy to compare thing. Yes, with Facebook's scale they likely run into some interesting issues than many will never but that doesn't make it into something vastly different. It's still an iOS app that may have more optimizations than other apps.
It's like comparing a regular house being built and a house that has to support every type of person in existence. They're still a house.
I find the skyscraper comparison actually quite apt. You build an app that is 10 features, while the main facebook app itself is 100s of features. When you build a building that is more than 20 stories high, you have to change everything about that building. No more wood construction, you have to use steel, you have to think about the bedrock, you need to put in elevators and so on.
I bet if facebook was able to ship an app that only came with the mostly used %20 and was able to download the 100kb of binary for each of the mini features inside their app on demand, it would be a far more scalable application. But they cant, because iOS stops them.
Not a skyscraper, then; rather, a house, but one which had to be built of brick-sized prefab components, where a constant revolving door of contractors and architects worked either by adding or replacing bricks one-at-a-time.
To put that another way: sometimes the problem isn't the problem. Sometimes the ungainliness of the tool or process used to solve the problem, is the problem. If you've got a 1000-ton hammer, you might (in theory) be able to hammer a nail, but you've got some other logistical challenges to solve first.
Chesterton's fence is a good reason to lament rather than forgive the ugliness in a large existing code-base. That's because it makes it hard to distinguish what is genuinely necessary from the genuinely bad ideas.
My experience with huge real-world code bases is that yes they do encode lots of important functinal knowledge about real-world requirements that a new programmer will not see. But it will also contain even more plain old stupidity.
The issue is that its much easier to critique and call out issues than solving them. Most of the time, the individuals involved are aware of the issues already - critique is only valuable once the criticizer has made an effort to truly understand the problem and what's already been done / being done.
Facebook is a really successful company, and they blame everybody but themselves for these problems.
The obvious conclusion is: Because they're so successful, they're right and XCode, Dalvik, Git et al are wrong, and their hacks are entirely justified.
The less obvious conclusion is: despite them being so successful, they have a code quality issue, and the tools are entirely justified in breaking under the load that's being put on them.
Arguably, the big problem here is that the obvious conclusion is wrong, and the less obvious conclusion is right. And, because the right conclusion is the less obvious one, there's plenty of value in advising the kind of onlooker that's likely to want to emulate facebook's success (read: the sort of startup that's precisely HN's target audience!) that they shouldn't accept FB's excuses at face value, and should think about avoiding these problems in the future.
> they have a code quality issue, and the tools are entirely justified in breaking under the load that's being put on them
That's not one conclusion, it's two.
Suppose for the sake of argument we grant the first part and say Facebook has a code quality issue and they should have less code. (I'm not familiar enough with Facebook to have an opinion on the matter, but I'm willing to postulate this for the sake of argument.)
That does not in any way imply the second conclusion! One of the most important differences between a good tool and a shoddy one is precisely that a shoddy tool can only withstand the load that should be put on it, whereas a good tool can go beyond that.
There's above-tolerance load, and then there's misapplication. A good tool might work beyond tolerance, but a great tool should "fail early" if it can tell it's clearly being misapplied.
You don't want space-shuttle O-rings that fray under high tension; you want space-shuttle O-rings that fray under low tension, because they're never supposed to be under any tension and so incorrect installation that puts tension on them should result in them shredding apart before they ever get out of the factory.
Right. So if Git detects a SHA checksum failure, it should complain immediately instead of silently trying to muddle along, and so it does. But I don't think that's what was going on in this case?
Sure, kind of. I agree: good tools should be able to handle being pushed beyond what counts as reasonable usage. But all tools break, at some point. Now we're splitting hairs: Is Facebook's use of the tools merely somewhat unreasonable, such that the tools should still hold? Or is it sufficiently past reasonability that they shouldn't have to hold any longer?
To answer that, the fact that they managed to break not one but several tools is highly informative. I'm willing to accept that some tools might not work at scale. I'm less willing to accept that what basically amounts to the whole stack for mobile development is _that_ weak, not without very solid evidence.
Near as I can tell, there haven't been reports of this sort of issue with MS Office for either Android or iOS, or from other applications that should, by any reasonable metric, be more complex than facebook.
> Is Facebook's use of the tools merely somewhat unreasonable, such that the tools should still hold? Or is it sufficiently past reasonability that they shouldn't have to hold any longer?
A fair question. The most reliable way to answer it is to see whether there are similar tools that hold up better under load. For example, are there rival version control systems that scale better than Git? Ditto for the other tools.
(Let's all watch out for the human tendency to interpret an affirmative answer as taking a side in e.g. a Git vs X competition. A more useful way to interpret it would be as a source of ideas on how to improve Git.)
Totally agree; it's a case of fallacy by results-oriented thinking. I don't actually know if that's recognized in any of the official lists of logical/rhetorical fallacies, but it ought to be. Well, I guess it's a variant of the 'Post hoc ergo propter hoc' fallacy, but seems like an important special case.
I'm no fan of the "knowing-all-that's-best" trend of articles either, but here's why I don't mind this:
Facebook popularized the idea of "move fast and break things." Move fast, by all means, and don't be paralyzed by fear of all change whatsoever, but I really didn't appreciate it when my classmates thought that by being willing to break things constantly in our group project they were showing us all their potential to be the next Mark Zuckerberg or Steve Jobs. Or when some of those classmates were hired as my team mates and brought that crap into a customer-facing service. I interviewed at Facebook and was scared by the cowboy mentality I sensed in every room and with every interviewer. I'd like the glorification of that to be balanced by, "hey here's some concrete data demonstrating they have some real issues that could clearly be addressed better."
I've seen that "move fast and break things" mantra used in projects, and yeah, some people think they're being the next Zuckerberg when doing so. But what I've (almost) never seen is someone who actually fixes the shit when it's broken.
Fine, go ahead and break stuff, but be willing and able to actually FIX it afterwards. That part never seems to get picked up.
Do you have some examples? I think the most valuable criticism is that which comes with some ideas for improvement, which it is difficult for people who don't know how the thing they're criticizing works to give. In my experience, the criticism of the non-experts that you're lauding, most often points out things that the experts already know about, but simply haven't been able to solve. In general, I think identifying problems is the easy part, and that good ideas for solutions is where the value is.
> My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.[...]Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case.
Sorry but, in my opinion, this is dangerous thinking. Saying you shouldn't say anything unless you've done equivalent / better creates the implication that anyone who hasn't worked on a Facebook scale mobile app doesn't know what they're talking about or can't provide something to the conversation that is at least informational or useful.
Do you complain about politics? If so then I would expect you to have at least held office in the past otherwise you should hold your tongue.
Have you complained about a truck driver on the road? Well, have you driven a truck? If not you should not complain.
This is an absurd leap of logic. There's many quality programmers who have never worked at a large company and many more terrible programmers who have worked at large companies.
Anyone can share their opinion and regardless of where they have worked it may or may not be valuable. If you are judging value based on the market CAP of their LinkedIn profile then you are probably missing out on most good information out there.
>> My question these days when I read such article, "What has the author done that is equivalent?"
Well I happen to (vaguely) know the author and have worked with him in the past and he has done a lot of great work, not all of it in the public domain though.
Comparing him with an organisation that employs 17k people is a little unfair but he comes out of it favourably IMHO!
Rephrase it to "What experience does the author have that indicates their experience would actually scale to Facebook levels?"
Is it bullet proof? Of course not. As an easy example, there are many that are more overweight than I am that know nutrition better than I do. That said, it is does seem somewhat logical to question health advice from clearly unhealthy people.
>As an easy example, there are many that are more overweight than I am that know nutrition better than I do. That said, it is does seem somewhat logical to question health advice from clearly unhealthy people.
Using your easy example with OP's logic, a 100-pound overweight person trying to lose 100 pounds by throwing up and taking dangerous diet pills should also discount the advice of a physician who've never lost 100 pounds themselves or successfully helped a patient lose at least 100 pounds.
Hmm... if that is what the OP intended, yes. Mayhap I was giving too much charity, but I would take it more of "Where is the evidence that you have successfully helped a patient lose 100 pounds?"
That is, again, it is not bullet proof. But, it is too often for someone to have experience that does not scale to the speed and size of facebook in this regard. Life is too easy to argue in the extremes of straw men. Wanting to know of experience that backs an argument is perfectly reasonable.
I agree to some extent, but think of it as more of this is how the sausage is made type of thing. As engineers we like to opine about perfect code, functional this, and monad that. It pains us to see that business success requires very little of even good code much less perfect. The reality is that the code rarely matters as long as it can be held together long enough for the next user or investor check to clear the bank.
This is a reality I've had a great deal of difficulty coming to terms with. I haven't internalized it fully, but I can't help but think not acknowledging this has harmed my career.
Until someone provided s counter-example the assumption is that yes, this is just what happens. We have entire industries of competing methodologies, some almost cult-like in their slavish devotion to "the one true process", and none have birthed any great successes. Why is that?
Nasa, budget is $19 billion. I can't find a public official record of their code size, but the IIS is approx 6 mil lines of code, and i don't think the other missions they did (all 33 unmanned space missions in total) are too far off, so lets give each an average of 5mil per mission. That's 165 million lines of code. And arguably, probably more difficult code than facebook's domain.
And all of the code for the NASA HR system, and budget management tools, and conference system, etc. The Facebook code base is not just the web pages you see and backend to serve them, but encompasses just about everything used to run the company. It is also built and maintained by a much smaller group of coders for a much more ambiguous target domain. (And no, I do not think that the NASA domain is in any way more difficult than Facebook's.) I guess I have to question the relevancy of your counter-example.
>And no, I do not think that the NASA domain is in any way more difficult than Facebook's.
You might not think it, but you'd be wrong. Facebook was for several first years just a set of PHP crud pages and heaps of servers to run them well.
Since then they added their own compiler and such, but the core of what they do is serving web pages at scale.
All known problems, solved by tens of teams all over the world, even at larger scales (Google for one).
NASA's problem on the other hand is quite unique, any errors can cost lives, and their tasks frequently include totally novel solutions for totally novel problems related to space travel, guidance, simulations, materials science, and so on.
"Facebook was for several first years just a set of PHP crud pages and heaps of servers to run them well."
Personally, I think we've all got a piece of the elephant here. It is true that there are no known methodologies that could build Facebook at the speed it has been built without producing a globally-incoherent design. Anything that could produce a globally-coherent design would have slowed them down so much that they wouldn't have been such big winners in the first place. (So "globally incoherent" isn't much of a criticism here, really.)
On the other hand, there are options other than "a big set of PHP CRUD pages and heaps of servers" available to us now, too, and I expect those options to continue to advance in usability. Even the various projects that improve on PHP would bring more benefit if you could use them from day one instead of a retrofit.
Facebook didn't win because of the software, but because of marketing. At its inception, it felt exclusive. You were special of you got an invite.
The other social networks were open season. Facebook made some good design choices with the simple looking UI, and didn't offer the customisations MySpace had. But that's a design choice, not software engineering choice.
No one wins because of the software (code quality). They win on marketing and that software (UIx) doing a needed task people are willing to "pay" for.
HN tends to be rather myopic in that programmers exist to program. Nope. Programmers exist to use a tool to solve a real world problem. The best way to do that is with the minimal amount of design and time required to accomplish the task at the desired level of reliability. Very few projects require anything resembling the "code quality" talked about on HN - in most cases I would say trying to enforce such principles generally result in worse outcomes and folks would have been better off spending 1/10th the money on a 16 year old off elance.com and simply fixed problems as they came up.
Typically the thing that is done the quickest wins, even if the implementation would make you cry. The fact is from a business standpoint - it truly rarely matters. Very little is as critical as people think.
I've noticed the industry becoming vastly disconnected with this fact recently.
Probably true, but it's also important that they did not lose because of the software. There are lot of great ideas that have gone in the drain because of sloppy software.
Many successful companies run the worst software you will ever seen. Yesterday there was a post about MUMPS which got me flashbacks of clients that showed me software that ran their entire 100m+ euro / year factory mostly written in MS Access / Excel on a shared drive (with the lovely locking Windows does!) mostly (this particular client had some CRM and an ERP from Dutch companies 'laying on the shelf' but 'never got around to it'). One of the biggest EU factories that creates the belts for conveyor belts runs on the worst PHP code as ERP I have ever seen. Many sites that are not Google or Facebook etc run on hacked together crap in whatever language and tech and do fine.
Twitter was crap at the start, most software in electronics was beyond unusable until recently (in some systems) etc.
You can of course lose because of sloppy software, but if your marketing is done well, I don't see why that would happen. There is an enormous overestimation how much people care about that kind of thing here on HN. From the first decade of the Windows versions to tons of Twitter outages after the launch to getting their computers hacked, passwords stolen, CCs stolen, privacy taken, slow as molasses systems, forced reboots, many crashes, bodged updates, forced updates, virus/malware scanners taking up 50%+ of your /still too expensive/ resources too much of the time, really bad iteration of the Facebook app at the moment (at least on Android), broken airplane booking forms, 404 support pages etc etc etc and yet no-one goes away because most people curse and move on to whatever. Unless your marketing is bad aka no-one uses it, it usually won't fail because of sloppy software.
If programmers at NASA make a serious mistake on say the space stations's code, people will die and billions of dollars of equipment will be lost. The same cannot be said for a social website which serves a bunch of content. NASA has way more on the line than Facebook.
Well, if someone at Facebook screws up and inadvertently reveals certain names or locations or the existence of particular groups within some countries people get arrested, tortured, and die. Last I checked NASA screw ups only kill a couple of people at a time. So you are right, they are not even close to comparable. See how easy it is to make false equivalences? I am familiar with the referenced article and if anyone in the software industry outside of few specialized areas tried to push a process like that they would need to start the endeavor by polishing their resume because they would soon be looking for a new job.
History is littered with counter-examples and great successes. Each generation of company is more efficient or solves more complicated problems than those before it. Facebook itself is an industry leader in open source innovation, which is why their product is so smooth. But not all companies specialize in hard tech, and all companies get old.
>I can't quite decide if you're being deliberately provocative or just misinformed.
Or maybe I genuinely disagree in your assessment that any of those technologies (apart from GraphQL, and that is still nowhere near major league) had any major "industry impact".
It's ad hominem; the author is being attacked to invalidate the argument, instead of attacking the argument itself. Which is never valid.
Appeal to authority usually takes the form of skipping making an actual argument for something and instead resting upon "X said so" and not explaining further. This is also not valid but is sometimes a shorthand for saying "X made argument Y," which can be valid if argument Y is valid (in which case the fact that X made the argument is purely incidental trivia).
Well, sure, they can. But don't you think someone who has been involved in a massive company would have more trenchant insights into massive companies?
You have right to criticize, but don't expect your criticism to be taken too seriously. There are two reasons for it: a.) you don't have frame of reference to compare whether facebooks result is better or worst then other companies b.) you don't know what tradeoff are necessary for making it work nor what it takes to achieve success.
Criticism of the government from someone who has experience running a large bureaucracy, negotiating, balancing groups with competing interests et cetera will rightly, other things equal, be taken more seriously than from someone whose experience of politics doesn't extend past ranting on Twitter.
Of course, other things may not be equal. If the latter person has sufficiently convincing arguments, can back them up with reference to sufficiently solid evidence, then these things may carry the point on their own merit.
Government criticism from people who have experience with running government or being in the politics tend to have much higher quality then criticism coming from sixteen years old high school class president.
So when it comes to the highest level of government, only those who have reached such heights may say a peep. Only criticism from former US presidents regarding Trump, please!
When random dude possibly with asperger gives advice/criticism on how to negotiate, reasonable government official will igore him - even if the criticism came in form of a blog. Likewise, succesfull activists ignore armchair advice from people on anonymous twitter. Not sure how is that controversial. Torwalds does not spend his time worrying about whether random bloggers agree with linux core style guide.
You did not demonstrated enough of knowledge/expeciance for me to take your criticism seriously is valid answer.
Criticism against governments by former governors should and is taken more seriously than criticism by some joe who's never had to wrangle a bureaucracy.
It's fallacious if you use it merely as "Authority X said so," no matter whether the authority is true or false. A fallacy happens when the conclusions do not follow from the premises of an argument. E.g. it doesn't follow from the fact that "Al Gore says that Global Warming is happening," that Global Warming is happening.
If, on the other hand, you say "Al Gore said Global Warming is happening, Al Gore's argument is premise A and B, therefore C" that's not fallacious because your argument doesn't actually rest on the identity of the authority, but the argument they made.
But all of this is irrelevant because it's not an appeal to authority. It's an ad hominem. The author is being attacked for not-being-an-authority.
It's a phrase that I've usually encountered in the past tense. Google Ngrams shows that in 1800 "held forth" was used about three times as frequently as "hold forth" [1], but this ratio decreased as both uses became less common.
Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars
There's something called waste. The issue is that we don't know how much better the situation can be but we can recognize that it's pretty shitty at the moment. If they're making billions of dollars and they're a big company and have all these code quality issues, what hope does a small company have when trying to make quality software? How many more billions could have been made with less churn and less maintenance costs?
That's a good point you make. But that's also why we have a formal system for making logical arguments. Critical essays which break things down into a premise, arguments, and a conclusion. This allows anyone to challenge anyone else without falling prey to the bias of deferring to authority.
Thus we can and should challenge writers and pundits, but probably from the perspective of first asking if they are making a logical argument or an emotional one instead of making it about their experience levels
I've become very cynical since the past year or so because there is a lot of noise/articles on the net, all knowing what's best for you or wrong with what you are doing. Yet without context or true authority to talk about it. My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.
Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case. Startup principles don't apply nor academia. Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars.