Maybe it comes down to a difference of how people approach books? Do you think that everything in a book should ultimately have an effect on the story? Do you think Hugo's descriptions in Les Misérables contribute to the story, or that it should only ever be read in an abridged version?
Personally, I think those things (and Tom!) do contribute a lot to the story, so maybe that's why I liked Tom. There's more to writing than simply telling a story and making everything pertinent to that, and thus there's more to reading. Of course, I also like worldbuilding and would read D&D manuals simply for that, without ever playing sometimes, so another reason why I liked Tom, but I can't help but wonder if it's split into two camps because of how people approach reading and literature.
Couldn't the same thing be said of Tolkien's colorful description of the countryside that can go on for pages at a time?