> [10] Make your project robust to re-orgs. A company management hierarchy is inherently fragile (a tree is a 1-connected graph, after all); socialize the project continuously with managers who might take over in the future. Do whatever it takes to make sure that manager churn does not result in unfair career outcomes for ICs.
> [17] For storage systems, bias heavily in the beginning towards consistency and durability rather than availability; these are harder to measure and harder to fix if broken. Because availability is easier to measure, there will be external pressure to prioritize it first; push back.
It's kind of crazy how well the author is able to move from the very top to the very bottom of the technical project management stack.
This is, to me, the definition of a senior software engineer. Awareness across every single part of the process of making software. Not expertise, necessarily, but enough knowledge to know what to be on the lookout for.
More of a team leader at least, non-IC. Some Senior ICs deliberately don't want any part in politics, just want a task and want to get it done (see for example advice on HN about avoiding meetings and politics at all costs).
One ill effect of not deciding to participate is that you'll be subjected to rules that which you play no part in establishing. I think it's about the balance to strike between being well and content with what's going around, and needing to exercise control to change the environment.
A friend described it to me once "a leaf node" in entire company structure - not a manager. I think it is a good description, also reason why saying "engineer" does not cover all ICs.
I really think this is a great list however I think one of the major factors why this project succeeded isn't listed but is mentioned in the intro.
> We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams
Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for project success in my experience, especially with complex projects like this. That small team can have a much tighter communication loop and all keep a single unified picture of what the end goal is. When the team grows this single unified picture is no longer feasible so it has to be broken down into sections, which is a good thing, but if the core isn't setup so that the sections are sufficiently decoupled things fall apart quickly.
That's a great point; I should definitely have included it in the list! I think new projects (especially ones that seek to innovate) have two phases: in the first phase, everyone's trying to kill it; in the second phase, everyone's trying to grow it. The kill-it phase is actually valuable for exactly the reason you outlined: it gives a small team the chance to create something highly coherent. (The grow-it phase is sometimes worse than the kill-it phase, but that's an entirely different story).
Reminds me of Kent Beck's 3x [0] (there are also video presentations online).
So from that perspective you're describing the first two "explore" and "expand". The third is also interesting "extract" where a project reduces risk over everything else.
Both are missing the fourth part of a lifecycle. If we take the 4x analogy further (which is a popular video game genre) then there would be a "extinguish" phase. In a 4x video game that is where you take over the whole play area (map etc.), essentially by bullying out or assimilating the other factions.
In the real world this is also kind of true (see Amazon as a popular example in tech, or price dumping strategies practiced by Pharma and other large industries). But it is generally frowned upon as it hurts the economy and erodes social contracts. Also for most projects and organizations it isn't feasible to do so. So the "extinguish" part could be understood as "gracefully fading out" from which new cycles can be grown.
I have been part of platform and product teams. So I think I sort of know what the author means by "kill-it" phase.
When new platform is being worked upon it's extremely difficult to get product teams to adopt. The PMs on the product teams view a new platform as a distraction because it seemingly doesn't help them launch new features. Worse it's taking the engineers who could be working on features to integrate with this new platform.
Typically during planning phase not too many oppose a new platform. However, when it comes to integrating (or adopting it) then everyone suddenly becomes conservative; no one wants to be the first mover, take the risk to be early adopter. It could also stem from product teams' prior experience of having burnt their fingers trying to integrate with a new platform.
I have seen quite a few platform teams get abandoned after everyone signed up. The platform team need to put in an extra effort to get product teams adopt.
Well said! This has happened to me, I was introducing a new system and in conversation with a a primary customer team during the planning and initial dev constantly. They provided good feedback, and were supportive. When they realised we would stop supporting the old system, the attitude changed immediately and they did everything they could to derail the project.
Have you tried to build something (lets say, a new kind of storage engine that is not SQL-based) and bring it up to other developers or people at your workplace (never mind business guys)? What is/was their response? Perhaps kill it with fire. If not due to valid technical reasons, financial reasons might seal the deal for it.
And I totally understand the impulse to kill it. Do not unleash it onto the world unless you have thought about your projects long term future (something php & js ignored to do at the time it seems). Thus you end up with tons of pet projects that are either sensibly kept private/ununleashed or because people fear the judgement of others (aka the kill it impulse).
I'd say the nature of the beast is that entrenchment/reinforcement of existing tools/patterns and mental models tend to stick around as they are safe, so a ton of innovation never occurs as someone might not bother to follow a certain path. It's the same with water flowing into existing grooves/cracks in rocks, instead of making new grooves, unless forced to.
The natural reaction to "let's build something in-house" is almost invariably "let's try an OTS solution, surely one must already exist". Most successful internal projects require either an organizational culture of "we're at the forefront of technology" (but then you get "surely one must already exist in some other division" instead), a nuclear icebreaker of a champion that is willing to put their neck on the line, or a series of (un)fortunate events that leave your project as the only way out.
I think this can be related to what Mel Conway stated in his famous "How Committees Invent?"
> Let us first examine the tendency to overpopulate a
design effort. It is a natural temptation of the initial designer-the one whose preliminary design concepts influence the organization of the design effort-to delegate
tasks when the apparent complexity of the system approaches his limits of comprehension. This is the turning
point in the course of the design. Either he struggles to
reduce the system to comprehensibility and wins, or else he
loses control of it. The outcome is almost predictable if
there is schedule pressure and a budget to be managed.
Lot of good, actionable learnings there. I liked this one in particular.
> [26] In a transparent codebase with quick review cycles, APIs will leak implementation details unless you gate-keep.
It's tempting to make quick fixes to APIs but please take it slow. Think through existing customers as well as how could new ones will use it.
Also, something I learned while building/maintaining public APIs was to always clearly document (along with examples) the context in which an API should (ideally) be used and how it combines with other APIs to enable common use cases.
This is a really excellent list. Some of it is specific to the kind of low level work this person was doing, but a lot of it isn’t. I would love to work on a team with the author of this blog post.
Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?
The author says
[40] Keep track of research in your space. Soon you’ll have a shorthand with your ICs that enables super-fast communication: “what if we try that thing from projectX? And combine it with the technique in projectY?”.
so by that learning it is not at all weird that they would refer to infra at an alternative company
> [16] Design as a team; implement as individuals.
This is pithy. I'm curious - does anyone here follow this philosophy? How do you actually design as a team? A meeting for every single design decision?
I'm used to the team finding general alignment on the problem in meetings, but then a single person coming up with the design, writing a doc, and circulating it for comments.
We wanted to ensure everyone on the team has explored the full design space, and everyone agreed that the chosen solution is the best we can think of, given the current information and constraints.
Once we have done that, everyone is aligned and it doesn't matter who to implement it; it will be implemented in the same way.
I think your example could be seen as designing as a team, its just one person who does the writing, but everyone who needed to be involved was involved.
On a smaller scale I've worked in teams where when someone picked up a story, for certain cases we would first do a rough outline of the implementation and then take that to one or two devs from the team, just to talk about how to tackle the problem. Not a meeting, just five minutes of "here's what I think would work". Its hard but good alignment before writing code really helps with finding the errors in a design.
Within a high level design there's often a lot of room for variations in the implementation such as code structure; though this freedom decreases significantly if you have a wide enough audience to need to consider Hyrum's Law.
Concretely, let's say I'm designing a term-rewriting system. The program will be inspecting an abstract syntax tree (AST) for certain patterns, and if it detects those patterns, rewriting them. (e.g. if we see the AST for "1 + 1" rewrite to "2"). I might write up a document on the architecture for the system, and a list of rewriting rules I need. How those rules are written can be left up to individuals on my team as long as it fits into the architecture, has the required functionality, and is of reasonable quality.
I understood this advice as "avoid doing parallelized design work", where "team" means a single coordinated actor rather than a bunch of individual streams. I don't think the emphasis here is collective design.
Wow! One hell of a good list. And well explained. You bet I book marked it. Heck even the 1-connected comment was worth the read. Because it's true. I'm dealing with that right now.
+5 and another reason I love HN. Now, if it you would have made it 43 items, a nice prime number, then I would be jealous to the point of angry (that's a riff on John Daily).
I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations? And oh by the way shipping this major product in under a year?
I think you are assuming all these things are independent of one another. I think deep engagement with customers directs your problem statement. Build the smallest possible thing to test the idea you are proposing. If you are trying to publish than its a requirement to be reading papers in general, you need to be able to argue why this idea you are testing is orthogonal to other ideas being proposed. As time goes on you just scan important conferences in your field and get better at knowing which papers you need to skim versus deeply understand.
Edit: Also intelligence is an over-rated metric, persistence is far more important when it comes to these hard problems.
Yeah I know what you mean, I'm in awe of the author's ability to create the perfect environment for his team to succeed. Part of the key, I think, is that he was dedicated to caring about every factor that might affect success (customers, project management, design, and the other categories from the list), which left his team free to execute on the technical aspects.
I also think we might be underestimating how much one could do with one year, a relatively well-specified problem, and a team of three super motivated and intelligent coders.
I think you assess your skills and what you bring, figure out the deficiencies, and then find people who can fill in other gaps(and hopefully have some overlapping skills as well). At the end of the day having half of the time and intellect is a great place to be, that is much better than having none of the time and intellect.
There is a critical element the post doesn't state: a successful small project has (has) very limited scope of customers and use patterns. That is where complexity in writing databases arises: limited scope databases are far easier.
[2] Be careful to have the right number of customers (in the beginning, just one) and the right customers (whose requirements allow you to build out key technology); and grow that number carefully.
Although the author says his advice is for people "building new infra at large companies", there's certainly a number of points that apply to a larger audience.
For example:
> Measurement is a means, not an end.
Reminds me of a lesson I learned early in my career. Bad things happen when you lose sight of your purpose and focus just on measurement.
> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations.
I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.
The first engineer on the Delos team here. This design decision gave us an edge: we delivered the first production cluster in 8 months by layering over existing systems ; and we smoothly migrated all underlying implementations in the following 2 years without our customer aware of it (0 downtime). This design decision is well captured by the OSDI’20 Virtual Consensus in Delos paper.
Very interesting. How did you deal with the issue where the old API had some feature X that wasn't supported in the new API but some customers depended on feature X?
I guess that a hard question for user facing API: you need to support both the old and the new features during the migration.
In our case, this design principle mostly applied to our internal APIs. Particularly, we designed our internal sub components (consensus protocol, particularly) in a way we can swap it without any downtime.
Right. So it seems you did a lift and shift over two years without changing the basic API surface. That’s ok, but stability of requirements throughout a rearchitecting project is a luxury and not the norm
We use the same principle on some of the frameworks I have worked on, it pushes you to make the API more explicit and keep the context localized. That makes codemods easier. This can sometimes lead to a more verbose / less magical API, but IMO at scale the tradeoff is worth it.
"[1] Keep your customers happy" - It's hard to actually have a team that respects this lesson. Even if one or two people on the team believe in this rule, others won't, and if they have equal say in the final product, it means the customers will suffer. Dogfooding is really the only way to fight this. Become your own most demanding customer.
"[4] But realize that customers may not express what they really need; don’t take requirements at face-value, instead spend the time to understand their use case in detail. Read their code." - Yep - Users Are Liars. They will lie right to your face, about requirements, deadlines, use cases, functionality, costs, etc, for no reason at all. Never believe anything a user says. And obviously, never tell them you think this.
"[9] If you get good and/or aligned managers, be as understanding, supportive, and accommodating as you can. If you don’t get such managers…" - Become the shadow manager; Team Lead, Lead Developer, Delegate Architect, etc. Ask the manager if you can get "hands-on experience" with various tasks like: running ceremonies, taking meeting notes, coordinating work, planning, resolving blockers. Reach out to other teams in private to resolve problems without involving any management, and be careful that the people you reach out to don't widen scope to their managers. Hide problems and bubble up accomplishments to the bad manager. You'll get none of the credit, but the manager won't have the opportunity to make everyone's lives bad.
"[10] Make your project robust to re-orgs." - Of these, the most resilient I've seen are the ones that promise the moon, and then deftly delay everyone's realization that it doesn't live up to the hype until they're all deeply bought in. Don't make the mistake of trying to get an executive to buy in, because they'll be gone in 3 years and everyone will ditch your project.
"[15] Design APIs with migration to new implementations as a first-class consideration" - Yes, yes, yes!! Literally all technology is replaced eventually, most of it in 3-5 years. Plan the exit strategy at the outset.
"[21] Have the right number of abstractions" - The correct number of abstractions is the minimum possible. Every abstraction should be a solution to a problem. You have to define the problem that the abstraction was meant to solve, and then ask yourself if there is a simpler way to solve that problem.
This is a good list, but I'm trying to understand the following in better detail:
> [12] Be conservative on APIs and liberal with implementations.
> [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail).
The word implementation is thrown out a lot. What does implementation mean in these quotes? The databases used (ie. dependency injection), the endpoints (ie. /user, /user/123), or something else?
I'm trying to visualize this in my head and am having trouble.
Co-creator of Delos here, Let me give a concrete example for [18].
We have a "StorageEngine" interface that wraps around RocksDB, i.e. "RocksDbStorageEngine" to provide key-value interfaces; however we are worried that someone would leak the implementation detail (RocksDB is so powerful) from the "StorageEngine" interface.
To solve the problem, we write another fully in memory storage engine implementation called "MemoryStorageEngine", which acts as a specification of the "StorageEngine" interface.
We run all storage engine tests against both implementations (rocksdb/memory), except for the durability part; we also configured the system to run over both storage engines.
By doing this:
- we can use the memory based one to enforce the behavior RocksDB based one by running unit tests against both implementations.
- no one can easily leak some implementation details of RocksDB from the StorageEngine interface; to do that, you need to first add those advanced feature into memory based StorageEngine as well!
This is just one example, and we have a list of such abstractions, such as MetadataStore, Logs etc. We created multiple implementation of each of these interfaces and ensure the system could run on any combination of those implementations.
Once people use your APIs you cannot trivially modify the bits being used or take things away.
A large API likely has complexity, security implications, messy transaction boundaries, etc... keep it small and simple, think of UX of an API, make it intuitive and capable.
That's the conservative bit... now the liberal bit.
Make clients, make one in Python and look at date parsing, make one in Go and consider struct re-use and how to deal with sets/arrays of things, make one in JavaScript and consider how easy it it to re-use parts of the code. If you have an endpoint that returns a list of things... is it supposed to be paginated over? How easy is that? Is it an infinite list? How easy is that? What would a client want to achieve? Is it possible?
When you produce prototype applications in a variety of languages the majority of the small inconsistencies and issues are super obvious.
And remember, once the API is out there it's hard to change it and you can't remove things easily... so discovering all of this early helps you get it more right than wrong the first time round.
To take Stripe's example, the API is as documented here[1]. Essentially a documented contract between a platform provider and their clients.
Implementation is things like, as you noted, DB used, programming language, and so on.
By "Be conservative on APIs" they mean be careful what you promise to your clients as communicated and documented through APIs. Once promised it's almost impossible to go back; you will lose your clients' trust and piss them off. In case of Stripe they are committing to support idempotent requests[2] which is sort of a big deal. And I don't expect them to drop that support in any of their future APIs because I imagine hundreds of millions of $$ business would be relying on that idempotency support.
This list is amazing for most software that i worked on. Dropping "database" from list name would only remove couple of items from the list. Most of the items are applicable much broader than just for database projects. Favourited for sure for future references.
The main thing i learned in my first production db work is that you pay for every bit of complexity you introduce. As a noob you are somehow much more afraid to not "model the data right" so you tend to create much to many tables much to many n to m relationships you make everything a string a uuid a relationship, never use bools etc.
But then you have to iterate and live with those systems and then you pay and every bit of complexity makes everything harder.
To give you some simple examples:
1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results.
2. You just made this thing a relationship now every time you have to join.
In the end, if you can keep it simple even if you don't model the data "perfectly". But of course sometimes the complexity is really necessary. To know when is being a good Backend / Db Engineer.
This is the whole point of the database: model the data exactly the way the business needs it, so you can have easy 1:1 conversations. Then, build some convenience views or other mapping layers on top of this data for use in reporting systems, etc.
Don't screw up the physical tables just because you don't like the knowledge that they contain in certain contexts. Facts are facts. If you don't like what you are seeing, change your query or filter the result set. If a customer record needs to be retained after deletion per biz requirements, then an IsDeleted fact is 100% the right path.
Both of your points enumerated above are trivially solved with views.
> 1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join.
Whether or not these were good decisions depends on the use cases of course, but papering over db implementation details like this with a view is pretty straightforward, no?
I’d interpret this to mean keeping you API specification basic to minimise backwards compatibility problems and reduce future refactoring difficulty. Don’t have special methods for unique cases, stick to restful principles where possible, keep data structures simple, etc.
In this kind of structure, what role does managers fill? It sounds like ICs can manage themselves, and the project manager manages the project... What's left for managers to do?
Manage bureaucracy around people: leave, hiring, firing, and promoting. They also generally need to be involved in things that may be more project management depending on the authority project mangers have in organizations: procuring new tools, updating senior management, escalation for cross team issues/communication, filter out noise from external forces, etc.
A "diff" in Phabricator is roughly equivalent to a PR, yes. A single point-in-time collection of, well, diffs against head/master that get code-reviewed and committed as a unit. If you were at Google, you'd call it a "CL"
Someone correct me if I'm wrong, but if my memory is correct, I was under the impression that a pull request was a term coined in the early Github days specific to their PR feature.
Yes, diff is a generic name, after a Unix tool with the same name. However, it hadn’t been the most popular name for the self contained unit of changes, that you would send to others to review or apply — the “patch” was. In Linux git workflow, you would use the “git format-patch” command to generate patch to send to others for review and to apply it to their branch.
However, even if you're using an email based workflow you're still making a request to somebody to pull your changes into their tree, and I'm reasonably sure 'pull request' as a term for that came from the linux workflow. GH more provided features to formalise a common concept (which, even where I've disagreed with them on the details of how they formalised it, has definitely been a net win overall).
author of the post here: apologies for the acronyms! I code-switched from academia/research to industry via FB a few years back and just picked up a bunch of terms that I thought were universal in industry. IC does mean Individual Contributor or "person that is not a manager" -- I'll add a definition on top!
I wonder if it says anything about Facebook's culture that I've never heard of the term Individual Contributor before. Everywhere else, it's Team Member or something along those lines indicating that you're not on your own.
I would add that, in context- it’s commonly used to differentiate job functions between managers and… well, ICs at companies where managers and ICs are peers or where ICs are potentially more senior than some managers in title and pay, sometimes by a lot.
So yeah, it’s a term of industry that often shows up in tech companies because of the usefulness of that distinction in those environments.
> Everywhere else, it's Team Member or something along those lines indicating that you're not on your own.
Individual Contributor means you don't manage other people, but it doesn't imply you are a team member. It helps to think of it as a role where at some point one is senior/skilled/specialized enough to contribute as a team of 1, working with one or more teams at a time.
I've only ever seen it to mean "individual contributor", but then this would make it odd/bad advice. There should only be 1 "customer". Sure, that customer may have a whole team that actually needs the product, and may provide them for clarification or what not, but to avoid unrecorded/incorrect handshake agreements, specs, or pet requirements that have nothing to do with the initial product ordered there should only be 1 POC for the product unless otherwise specified.
When multiple teams are using the thing, talking to at least one developer on each of those teams is going to surface things you won't find out anywhere else.
I agree that an individual user shouldn't be driving roadmap stuff directly, but getting their perspective, pet peeves, war stories etc. is still valuable advisory input to the overall process.
You should keep a record of what you're doing and why, but getting clear and accurate feedback from the real decision makers who have control over the technical success of your product is essential.
Despite all that, I still find it absolutely necessary to get in the same headspace with someone on the other "side". We can and should always float any outcomes to the primary points of contact on either side, but a POC is a primary point of contact, not the only point of contact.
Put differently, interfacing directly with customer ICs is good and necessary, so long as it's done with the go-ahead and oversight of each POC.
Where does the whole Individual Contributor terminology suddenly come from? I've suddenly seen it pop up many times over the last months and tbh the term doesn't make any sense. We just used to call them developers, engineers, or whatever.
I've been hearing the term in silicon valley for over a decade, it's not particularly new.
A lot of time in projects like these you end up getting shunted to tech leads, managers, etc, whose job it is to "take the coordination work" so that people writing code can "focus on implementation". The point of this advice is very specifically to bypass the coordinators and go directly to the people using your project in their code to make sure you understand their concerns, and don't play political telephone.
I've seen its popularity coincide with emphasis on non-managerial growth opportunities in large tech companies. Gitlab uses it for determining compensation, for example (the top IC is equivalent to a VP in their org).
IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service).
(Vouched for this flagged comment after seeing the other comment)
Thanks, I had never heard this term until I started working at my current employer. It seems like bay area lingo is assumed to be understood by everyone, even though it may not be used in the industry as widely as some folks assume.
In Australia outside of IT roles, IC means In-Command i.e. a team lead and 2IC as 2nd In-Command or the person who becomes team lead if the team lead is on leave or resigns.
> IC is a commonly used acronym for Individual Contributor
Its not though. Outside of hacker news I have literally never heard this term before, nor does it make any sense. If I'd start talking about "Individual Contributors" to my colleagues they would all be confused wth I'm talking about and wondering what's wrong with me. Who even coined this stupid term, and why do we pretend we need a new name for "engineers"?
It has been around in industry for at least twenty years. Facebook is a bit unusual in using it as an actual job title, but it's by no means new. I remember talking about how DEC had a healthy IC track separate from going into management, and DEC hasn't even existed as a separate company since 1998.
It's certainly regularly used but of course 'regularly' and 'common enough you can assume your audience already knows it' aren't necessarily the same thing in practice, usually at the most inconvenient possible moment.
Exactly. And this "IC" thing is a good example of something they call a negative externality. Negative externality in this sense is when you have to pay/are affected by something you don't want like polluted air.
The original author saved maybe couple of seconds to write "Individual Contributor" on the first occasion of the phrase, but made many people here spent many seconds trying to figure out (probably by searching the internet) on what that acronym means.
This is especially pronounced in some companies where the acronyms might be understandable just to a few members of some team, and when when this is communicated to the wider company, it can cause even a greater confusion (as you cannot search for it online) and waste of time for such a small "irrelevant" thing.
So in other words, it can be compressed a bit in: "why should I waste a little bit of my time, when I can waste everyone else's time". Or simply: it is disrespecting your audience.
The original article beside that is quite interesting though.
> [17] For storage systems, bias heavily in the beginning towards consistency and durability rather than availability; these are harder to measure and harder to fix if broken. Because availability is easier to measure, there will be external pressure to prioritize it first; push back.
It's kind of crazy how well the author is able to move from the very top to the very bottom of the technical project management stack.