I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.
Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory.
That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.
I suspect if pg was not using Arc he would not be keeping everything in memory.
I don't want to speak for pg, but I highly doubt this. He chose the architecture he did for good reason. Looking things up in memory is much faster than looking them up in the database. The only issue is that the site is too big to fit entirely into memory now, so he needs to write something that cleanly manages the memory space. (BerkeleyDB calls this a pager; data is stored in pages that can be in memory or disk.)
SQL databases are a very poor fit for sites like news.yc. Things like threaded comments, for example, are hard to model efficiently. If you use an object database (or roll your own in-memory thing), though, it is very easy to model.
Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone to use Rails. Rails is not the only way to do things, and it's rarely the best way.
Actually, most of the websites I write don't use SQL.
I don't think your "poor fit" comment is true. You don't need to model the threaded nature of comments at all in your database. Since there's a relatively small number of comments per post, and most access to comments is probably show-me-all-comments-for-this-submission, you usually just need a single foreign key, each comment to its original grandparent submission.
In this specific context the relevant advantage of SQL over rolling your own is that there is less distinction between keeping data in memory and on disk. Otherwise, you can certainly do anything with memory + SQL that you can with memory + disk.
Sure, you can store it in a relational database, but you have to map your data to some structure that it doesn't have. That's what we call a hack.
If you use an object database, you can store your data exactly as it is represented in memory. That is much cleaner, IMO.
(Relational databases are like programming languages. Just like you can use any programming language for any task, and you can hack any data you want into the relational model. But that's not always the best way. Sometimes a key/value store, or an object store, or a document store is a better model. Using a better model means you need to write less code, which means your app will have fewer bugs.)
