It's running out of memory. The half million or so of HN comments and stories no longer all fit in memory at once. This is not in itself a problem. We only load stuff as needed, and only crawlers ever look at the older stuff. But it does mean that eventually (currently after a day or so) we run out of space and the server crashes and restarts.
I'm working on a less crude version of garbage collection.
It's a great scalability test then! Memory is cheap, so that should not be a really big issue then, 32G of ram in a server is nothing special these days. Unless of course you're already maxed out, then it is a different issue.
