Digg threads become too full and the model just didn't scale up. A thread with 10k up votes and 1000 comments is not inviting to new users. A stupid or insightful comment on such a thread will be completely ignored if it wasn't the first post.
In contrast Reddit has thousands of small communities that let new users be heard. Down voting and arguing over "bad" comments gives an impression of authority that make the user want to learn and impress the group. It is really all just a game.
It was added after about a year or so, and there was two real driving forces behind it. The first was that people started using 'tags' in their submission titles to do things like "Ask Reddit" and also that the main page started getting overrun with Ron Paul tributes because apparently he was the second coming of Christ and would be the panacea that the US needed.
And since most of us started visiting Reddit for the tech/programming/science content people saw that stuff as spam.
Still, the subreddit system is really the only reason I still frequent Reddit. The smaller you go, the higher quality the content becomes and the better the community is (in general).
IMO, 50 000 subscribers is about the tipping point for subreddit quality. Beyond that number things start to go down hill real fast. You also don't want too few subscribers, however, or there will be too little new content posted. At about 10 000 subscribers, the amount of posts should be high enough that there's always something to see.
In contrast Reddit has thousands of small communities that let new users be heard. Down voting and arguing over "bad" comments gives an impression of authority that make the user want to learn and impress the group. It is really all just a game.