Curiosity: Why did you start with flat files? It looks like hackernews was started in 2007, relational databases had been around for quite some time at that point, and were the standard way to store such forum data (see: every popular forum framework at the time)... the decision to store this sort of data (news/link forum with comments, 100% text) as flat files is very confusing to me.
My guess is that Arc - the lisp language running HN created by Paul Graham - was new, and coding and maintaining a database driver was out of question.
Today, perhaps the way to go would be to use some sort of json webservice interface to a database written in another language rather than writing a driver.
That would be my guess as well. It's one thing to decide you want a simple forum and have it coded within a couple of days. It's entirely another to spend months creating a stable database library and keep it upto date with all the latest changes.
I don't know why he did it, but some possible reasons:
If you already understand your OS well, filesystems have simple, known, and reliable performance characteristics. Databases involve a lot more code, and are harder to reason about.
If you're starting something, it pays to start simple. How many 2007 projects made it to today with this much traffic? A very tiny fraction.
If you're keeping a lot of data hot in RAM and working with it directly (which I hazily understand is HN's approach), then databases don't buy you much. Typical database usage is to use a database not just as a persistence engine, but a calculation engine, a locking engine, a cross-machine coordination engine, and other stuff as well. If all you need is persistence, then that isn't very hard to do yourself.
For things you intend to build and maintain yourself, "standard way" may not buy you anything. Graham already had toolbox he knew perfectly well. He didn't have a lot of incentive to learn somebody else's way.
