Does anyone have any good resources for learning how to better architect systems? Other than, say, The Architecture of Open Source Applications: http://aosabook.org/en/index.html
Reading source code of prominent projects is a good way to get better, but it's hard to understand the architecture if you aren't aware of the design constraints/requirements that led it to be the way. Surely there must be a way to fast track this to some extent. Under the deliberate practice model by Anders Ericsson's (i.e. the 10,000 hour rule), simply reading and understanding code is suboptimal because of the lack of immediate feedback on your understanding. I think it'd be interesting to see tutorials/guides that showed how a system design has changed throughout its history as well as the reasons for those changes.
I beleive that there is one answer to this question which does 80% of the job, it is easy to express and yet in the same time impossible to grasp for so many programmers/managers which leads to so many awful codebase.
As the post says very well: start by thinking about what you would do if you had infinite resources. That's the ideal case. Find a real bottleneck (money, cpu, memory, ?) and rethink your architecture accordingly, find the next real bottleneck and so on.
This is gold. I remember a quote from a prominent game programmer, I think it's John Carmack but I'm not sure, that when thinking of an approach to a new game engine he would start on the basis of an imaginary computer with instant CPU calculation and infinite memory. I can see the parallels here.
In my experience, there is no better way than to build it. You learn quickly from your failures and become aware of subtle issues that no text-book or article will ever touch since they are so domain specific.
The best way to learn by doing, IMO, is by joining an organization that works on a large and complex code base. It won't be as exciting as writing new code, but adding to an existing complex system will force you to take into consideration things that you otherwise would not.
parent has some great points. You only will really learn by doing.
A couple of things you can do is start reading up on well designed systems and the gotcha stories. Personally, I track highscalability.com since I spend a lot of time working in distributed systems. They tend to have interesting insights and will link to tech talks and blogs that contain a wealth of information. Also go read up on some of the papers explaining the designs behind systems you use (e.g. Amazon dynamo) and why they made the choices they did. It'll give you some insight into how people measure tradeoffs and figuring out what's important.
Take a task of some inherent complexity but that can still be coded in 10-20 pages of code or less, create one solution, and then try out various different versions of it, or even complete rewrites, changing coding style, decomposition into modules, performance tradeoffs taken, data structures chosen and so forth, and then compare those with respect to length, code clarity, create benchmarks, etc.
Example projects: ray tracer, hash table (way more choices and trade-offs to be made than you would expect), toy grep, toy pseudo-pascal or lisp compiler, basic web crawler, ... It's hard, but you will learn a ton. Even putting substantial effort into improving a one page program, incubating and trying out ideas over a few days, can be beneficial if there is some "meat" in that program to chew on.
In general I believe software architecture is much more low level and less philosophical than a most aspiring architects believe. It's really about knowing the trade-offs of a decision before making it, and this rests on knowing a lot about data structures, algorithms, existing components (like databases, operating systems) etc. Those little programs in non-trivial problem areas often already show up a lot of those things.
It applies to all creative work, but I've had good experience with "forced alternatives." That is to say,
Forcing myself or my team to come up with three options for architecture even if the first one seems to work. The process often leads to great instead of good outcomes because in scenarios 2 and 3 you're free to moon-shoot as you have the safety net of a workable option #1.
One strategy I've found works for me is to keep a document explaining why you designed the system the way you did. What were the alternative possibilities? Why didn't you pursue them instead? Do so without referencing projects or implementation details, and derive your reasoning from first principles.
I revisit the document as I'm working on a project, so when I go to code a new feature, I make sure I can justify the need to create it with respect to the larger design of the system it lives in.
This reminds me of a tip Guy Steele gave in his talk on the Language Design panel of the Dynamic Languages Wizards Series in 2001 [1]. He said: It's important to establish design principles early on, that is in effect to do meta-design, because as you evolve the design, or as the language grows later, you want to remain true to an initial vision. This has to do with keeping the description small, and if you can resort to a meta-principle to help you resolve a particular design decision, it keeps the language more consistent. (17:09-17:29) [2]
This is a good book, though a few years old now:
Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design
(http://shop.oreilly.com/product/9780596517984.do)
Disclaimer: I'm the lead developer for one of the featured projects (though wasn't at the time).
It would be faster to be mentored by (/work with) someone who knows how to architect systems. The better your mentor, the better you will be. If one thinks he is already good enough at architecting systems, obviously he will not improve very quickly. There is always someone better.
Bram hints at this, but doesn't quite make it as explicit as I would: the thing that distinguishes great programmers is the ability to understand how the pieces interact. Anyone can understand how one piece of code works, even if it's fairly complex. A good programmer can understand how one very complex piece of code works. However, any modern codebase of any significance is impossible for anyone to understand as a single monolithic whole. The key ability is to create a clean mental model of each piece, even if the actual implementation or interface of that piece is nowhere near so clean, and put those models together to understand all the "between the cracks" stuff. That's the key bit of genius behind BitTorrent, for example. It's not that the individual pieces are so special, but that the emergent behavior of those pieces interacting with one another is very beneficial.
I find best way to learn is to read existing code. When I find interesting project, I download source code and read it from top to bottom. I may even fix a few bugs before hitting compiler.
You need good quality code to begin with. I have had a couple of jobs as a maintenance programmer for large projects, and the code on one was not something you could learn from. On the other hand I have been playing about with some Django third party modules, upgrading them to Python 3 for my project, and that is code I can learn from.
One of the jobs was for an investment bank. The job was crap. No one in my team of 5 lasted much more than a year. And sadly they did have the resources to hire more developers and address the problem that way. Brute force recruitment.
> My suggestion for learning software architecture is to practice. Obviously you can't practice it by doing hundreds of projects, because each one of them takes too long, but you can easily design a hundred architectures for problems which only exist on paper, and where you strive to just get the solution to work on paper.
But, how do you get feedback to tell you if you're making the right or wrong decisions?
Thanks for adding the 2004. I wondered why he was still bashing poor subversion and talking about codeville. Interesting thoughts otherwise. When I interview potential co-workers I ask what makes good code and it's interesting to hear the variety of answers. Definitely one of the best interview questions I have discovered. I think Mr. Cohen would pass my interview :-)
Changing things dramatically for the better always seems to involve "forgetting what you know", ditching everything, and starting from scratch, doesn't it?
In relation to the other post about the Core Infrastructure Initiative here, you could use that principle and realize that funding and improving OpenSSL, will still keep things more or less as "secure" as they've been until now. But to truly improve the security of the infrastructure of the Internet, we need to rethink things from scratch, and fund and adopt those, too (in browsers, operating systems, etc).
I'd also include knowing a lot about computers. Like how they work at a low level, how they represent numbers, low level networking protocol, how the operating system interfaces like interrupts, how processes and threads work. A computer science education really helps, but you'll also have to be able to solve problems and think about how theory and real physics world collide.
Reading source code of prominent projects is a good way to get better, but it's hard to understand the architecture if you aren't aware of the design constraints/requirements that led it to be the way. Surely there must be a way to fast track this to some extent. Under the deliberate practice model by Anders Ericsson's (i.e. the 10,000 hour rule), simply reading and understanding code is suboptimal because of the lack of immediate feedback on your understanding. I think it'd be interesting to see tutorials/guides that showed how a system design has changed throughout its history as well as the reasons for those changes.