I want to broaden my horizon regarding how things are solved in the real world. Other than some very high-profile companies (like Netflix, github) and companies that I've worked at, it's hard for me to find easily digestible (20-60 mins) examples of actual working architecture of differently sized companies from different business verticals.
At what level of scale might one expect to need what's going on in the "Edge Cluster", as opposed letting all the requests fly right into the app servers?
Keep in mind that the author of that blog doesn't actually talk to anyone at the company they are writing about, they just collect articles around the internet and public statements and piece it together from that.
For example one of the most popular article on that site (which is part of their book now) is the article on Netflix. A lot of that was cribbed directly from my talks, but they never reached out to me to even check it over, and as such missed a lot of nuance and detail, things I didn't cover in my talks.
Same thing for the article about reddit -- also cribbed a lot from my talks.
It's a fine overview, but light on specifics. I've reached out a few times and some things have been corrected after the fact, but I don't know if the other articles have been reviewed.
So my point is, be warned that the articles on that site are not primary sources but are derived from them.
Other times, they would directly talk to a single employee, but get skewed or misleading information based entirely just on that one employee's POV.
Their post about Tumblr's architecture [1] focused a lot about JVM-based services, HBase, etc which in reality was only ever used for a tiny subset of the backend. The huge section on "Cell Design for Dashboard Inbox" was especially ridiculous: the systems described there were literally a mix of complete vaporware and failed/canceled projects that never even got close to production.
As an early Tumblr engineer, I was really upset to read this nonsense. I spent several months of my life working very long hours to successfully scale the existing (PHP/MySQL) dashboard activity feed architecture in 2011-2012. It continued to be used as-is for many years after this interview, with lower latency and much lower cost than the proposed hbase/scala cell replacement.
And of course, engineering candidates being interviewed would always ask about this hbase cell architecture thing that they read about in High Scalability...
It isn't really a choice, it is just what happens due to incentives and changing needs and leaders and politics and technologies inside the company.
People assume that software architecture is like building architecture, in some ways it is, but NO ONE has ever showed up to a construction site that was half way done and said "Hey guys the steel framing we ordered has been delayed so please continue building the rest of the building by replacing anything that was original designed for steel beams with bamboo.
This is awesome (as well as the ball of mud "paper").
I think the building construction analogy might be similar to a home which has seen multiple remodels.
Now imagine a scenario where you have an absentee owner with a lot of money, a permanently staffed architect and a bunch of extremely able, slightly competitive contractors all on staff - each trying to prove their annual salary.
The original one story building would quickly become an ten story nightmare of a building.
Just another example of Silicon Valley being ahead of its time: The Winchester House is pretty much exactly this. It used to be 5 stories, too, until the 1906 quake gave it a haircut.
A better description is that of garden landscaping.
You have a reasonable idea of what you want to achieve, and it looks good on paper, but until you have actually walked through it, touched and felt it, you don't know whether it really does what you wanted.
And of course, it changes as it matures, and what was great at first can become overgrown and resource-hungry.
I like Casey Muratori's view of software architecture being more akin to urban planning than to building architecture. Taking that view seems to alleviate many of the apparent paradoxes.
> Other than some very high-profile companies (like Netflix, github) ... it's hard for me to find easily digestible (20-60 mins) examples
So I think this doesn't meet your requirements, but I like Tech Dummies Narendra L's YouTube videos [0].
He introduces big tech companies' systems in 30-60min videos and it's not difficult to understand.
so like.. not to be too cynical but how does he know his representation is correct or at least not misleading? a lot of youtuber content is just made up.
Yeah... It's dangerous to accept random YouTubers' content as fact mindlessly.
I'm not sure all he says are correct, but at least he uses the target companies' engineer blogs, external articles, and some open-sourced part of systems (and list them in the video's detail section).
His main targets are often big techs like Twitter, Uber, and Netflix, so I guess such documents are often available.
Narendra's content is awesome, but I think you're right to be skeptical. His content is more focused on how to answer system design interview questions about how the companies operate.
One thing I don't remember explicitly called out, is that most all architectures are grown. There are scarily few situations where starting with the complicated idea is a good idea.
> A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.
So that instead of focusing on the architecture as it-is, we should pay more attention to the evolving stories of software architectures because we can learn more from them.
InfoQ's QCon conference frequently has an "architectures you always wondered about", which frequently has good talks. You can find them here: https://www.infoq.com/architecture-design/.
I came across this a while back
It is a court document from one of the UK post office horizon IT system scandal.
It has a very detailed review of the system and its history dumbed down to the level a lawyer could (maybe) understand.
It stands out because it is quite hard to find examples of this level of detail about such a large scale distributed system which aren't internet / web tech companies.
Just browsed through it and was surprised at the great explanation of RDBMS' provided.
Excerpt:
21. The main use of a relational database is securely to store large volumes of structured information. The way it does so can be understood as having large numbers (tens, hundreds or even more) of different spreadsheets (which are called tables) and which are linked to one another. Two different tables in a database are linked to one another (in a 'relation') when they both have one or more columns with the same meaning and share values in those columns.
I think this is also a important lesson on how to write simple, direct and succinct documentation; no unnecessary fancy "methodologies/processes" required (i am now a fan of Dr. Worden :-).
Well worth reading in entirety.
PS: Any other overviews of "less sexy" software systems that you can share (eg. Banking, Insurance, Railway Systems etc.)? I am of the opinion that these are the real success stories of the Software Industry. They are battle tested and proven over time.
It's good to note that it depends largely on the company you're looking at.
I work for a very large organization (~£6bil in revenue, £700mil in profit last year) and we suffer from the "mud" problem - nothing about our technology stack is particularly special, it's just a hodgepodge of many different technologies that struggle to work together. That's not entirely fair - I work within a very unique solution inside of this firm, but I'm in a very unique position and I'm sad to say that it took a silly amount of hard work just to be able to not work on legacy applications.
That being said, the companies you mention (Netflix, Github) work completely differently - they were designed with tech in mind! They probably are much more lean in a technological sense, and don't suffer from enterprise architectural issues that large legacy firms do.
I suspect that this inability to move has singlehandedly killed more than one company, though I haven't studied the market to the point that I could really name any. The real kudos has to be given to large companies that existed before the internet and were able to move away from their slow-to-adapt, horribly inefficient legacy systems.
I often point out an anecdote from my early e-commerce days in the late 90s where a customer wanted in-store pickup like everyone else they saw on the web doing. But they just had purchased an already outdated (but IBM so nobody got fired) Point Of Sale system for 40 stores which did inventory management as a batch at the close of business each day over Frame Relay lines or even dialup. The concept of a VPN was alien to them.
Since they were selling some limited edition high priced items that were allocated to each store, there were often only 1 or 2 at a given location of what would be a popular item.
So you can imagine when I explained that the huge investment in the legacy system just a few years before was a big blocker for ‘pick up in store’.
I think we had to hardcode something that would remove an item for sale online if there were less than 2 left or some awful hack like that to reduce customer complaints.
The purpose of each episode is for anyone to walk away having a reasonable understanding of why and how a company built and deployed their app with XYZ technologies without needing to know anything up front. There's over 100 different companies / individuals who were on the show.
I tried to make it as efficient as possible to get these details. There's a lot more detail than a few bullet points but it doesn't get super lost in the woods with a million low level details that's specific to 1 company. It's basically an hour or 2 conversation for each episode where we cover everything from building to deploying their app, lessons learned, etc..
Like most things in software engineering, it's qualitative and empirical - but also has very strong potential to function as a supporting "first principles" theory for so many things.
I think this paper has a fantastic corollary in Peter Naur's "Programming as theory building" which triumphantly explores the implications of institutional knowledge in long term software maintenance.
https://pages.cs.wisc.edu/~remzi/Naur.pdf
Work at more companies! Lots of great resources here, however, from experience I would say take all public presentations about how things work inside a company with a big grain of salt. They always have a vested interest in advertising successes, and public presentations always focus on some filter of interestingness. You won’t see the important “real world” parts of what’s left out unless you’re part of the organization.
I was developing some architecture training recently and had this very same question. It’s not easy to find realistic architectures.
The best I found was the German contact tracing app — Corona Warn App. It was done by a group of consultancies in collaboration with the German govt, and went from inception to launch in around fifty days — largely if not totally open source.
It’s got full git history so you can see it evolve over time, along with the implementations (also on Github).
There’s a pretty fascinating short talk by one of the people who led the project on youtube too — more about the process side though: https://youtu.be/5y1sHSkPWRg?t=1770
In the real world [of software], things are solved by choosing the tech with the lowest barrier to entry, not reading any documentation, getting a minimum-not-quite-viable-proof-of-concept working in a development environment, then making that production, over-working a select few to keep it running, and a lot of crossed fingers and heads in sand. The only thing you'll learn from different verticals and sizes is how size and scope have no correlation to how things are built or whether they work well.
The interesting part is how larger scale makes things fail more often, and the response to increased failure can either be running around with your hair on fire for years, or a solid firefighting team, or actually teaching teams not to build products that catch on fire. The only way to get the last one is by focusing on people, not technology.
Thoughtworks keeps a "technology radar" that I have found very interesting. I won't post the "whys" but it's worth looking for upcoming components and tech that they are seeing used more in consulting. https://www.thoughtworks.com/radar/techniques
Wow, thanks for this link. At a glance, there are tons of great and interesting articles to view. I am saddened by the fact that there isn't more content on self-driving vehicles.
I'm looking for something similar for design interview practice purposes in my job hunt.
All the systems design resources I can find are aimed at L4/L5, where the focus is e.g. on how to implement a rate limiter on a single machine, or at best saying you can distribute it by putting the counters on a cache server.
I'm trying for L6 and can identify many of the issues with a L5 design (redundancy, sharding, global latency, hot spots, local batching), but it's hard not to miss the obvious, and to offer practical/realworld solutions, when my day job is embedded compilers and not large scale systems.
This is mostly a rant but I appreciate suggestions.
I guess searching YouTube is best for these things.
https://martinfowler.com/ but I'm not sure if he touches the real world sometimes, it all feels very academic rather than pragmatic.
I wonder if you'll find "good" outcomes though, it seems to most startups or companies bumble their way to an architecture that works for them. It might not be correct but it might be best way to build a company without architecting everything too much up front.
Read the engineering blogs of big companies like Google, Netflix, Dropbox, etc. and especially read papers they publish. Google has a book out now about its software engineering practices too--although it's not specifically on architecture you can glean a ton of info about how Google services work internally from its software processes: https://abseil.io/resources/swe-book
Try and search for SDKs of some large software. Usually those would be programs for creating content - audio, 3d modelling, 2d drawing, etc. Every major vendor has a plugin architecture that quite obviously leaks implementation details. So stuff like Adobe Photoshop, Autodesk 3dsMax, FL Studio. All these have public SDKs that you can download, explore and write plugins for. You can probably think of some more programs that support third party plugins.
I'm very interesting in the architecture of systems similar to Amazon SQS. Interestingly I couldn't find much discussion on such systems. I guess it's because SQS is such a typical iceberg system that has sophisticated designs to provide dead simple APIs: having a queue that supports competing consumers and simply scales infinitely (in the eyes of users) with users provisioning capacity is no joke.
+1... I haven't read through it in a while, but I read it frequently earlier in my career and have found that really valuable. In the cases where I've been asked about situations that I had no first-hand experience in interviews, it's been helpful to draw on knowledge from that reading. Being able to say, "well, Company X got to scale Y using technique Z" sounds more compelling than taking guesses.
This doesn’t really answer your question, but gleaning it from job descriptions is one way I do it.
If I’m curious how a company did something, searching for their job descriptions can turn up interesting stuff like what languages and frameworks they use, and often from there you can infer what their architecture might look like.
Etsy has a "Code as Craft" blog with lots of interesting reads. It's not been as active the last 2 years but has been re-launched with more posts the last few months.
https://techengineering.io/ aggregates the technical blog posts from various tech companies and they can be sorted and filtered based on reading time and the architecture you are interested in.
The book "System Design Interview" by Alex Su and Sahn Lam is a good place to get digestible examples. It walks you through step-by-step how you might solve various systems problems, introducing the pieces you need. Each problem fits your 20-60 min request perfectly.
AWS Summit is approaching. I usually find other teams, even nominal competitors, or hulking behemoths of industry, to be quite proud of what they've built, and generous with their battle tested knowledge. All you have to do is reach out and ask ;)
Be careful with things like this - it is ultimately an AWS marketing channel, so they’re not going to bring on people who say “we tried running everything on Lambda and it turned out to be a deployment nightmare”. The very best way to do this is find a tech meet-up around the sort of thing you do, and then go for the after event drinks. Get to know people, chat with them, and find out all the many ways architectures can shoot you in the foot.
I know it's not a 20-60 min example, but I find reading open source repos very informing:. I'm the cofounder of Budibase, and I like to jump on a call with new contributors and take them through the high-level arch and repo:
https://github.com/Budibase/budibase
http://aosabook.org/en/index.html
A good example is Scalable Web Architecture and Distributed Systems by Kate Matsudaira:
http://aosabook.org/en/distsys.html