The BGG rankings are a particular viewpoint into a general take on board games; the way I look at it - anything high ranked is going to be "good" even if I don't particular like the play style, theme, or execution. Many are "good but not for me".
The problem with the rankings is that some very playable games are ranked low. It's a meme on BGG but CandyLand is famously shit on: https://boardgamegeek.com/boardgame/5048/candy-land - but it's a perfectly acceptable "game" for very young children, and if you've ever dealt with said children you know that getting them to successfully complete a CandyLand session without going backwards on the track, randomly wandering off, taking multiple cards, etc is a serious feat.
It's not a great game but it's certainly still playable in the niche it occupies.
Yeah, I've heard of very few of the top ones, and they look like the super complex kind of games I don't play. I did play Pandemic Legacy and didn't like it. Then I cmd+F'd for Diplomacy, Catan, and chess on the top list, didn't see them, and said nah.
Catan isn't actually all that great (and I say that as someone who bought his set back in 1995). We have much better games these days.
Stripped of its cultural significance, chess isn't that great either for a general audience. It suffers from analysis paralysis. I would only recommend it to a niche audience.
I love Diplomacy; I played many games over email, in person and even participated in the German Diplomacy championship (face-to-face) back in the 2000s. Yet, it's also a very niche game, and I can see why some people would rate it low. I actually prefer the game over email, because it gives you more time to think, sending emails resembles the telegraphs of the early 20th century and I am less likely to burn out from a months long game with one move every few days than one very intense day of face-to-face play.
Btw, if you like Diplomacy, I recommend playing on the 1900 map variant. It's basically an improved version of the standard map.
Catan is merely ok from a serious gaming perspective, but for most people sitting around having fun, it's a go-to. There's value in it looking nice and being easy to understand, and I can see it staying popular 20 years from now. The top listed modern games have only ever come up in niche board game group settings for me, while even chess and Diplomacy have been much more popular. This is only my experience, but I'm probably not alone in thinking the top games are all very uncommon.
I've actually only ever played Diplomacy over Backstabbr.com with friends, so analogous to email/postal play (if you still count that as a board game). Completed several games now, and I've been wanting to do it in-person. That 1900 variant has more detailed notes than any other variant I've seen, and I'd like to give it a try!
I agree with your evaluation of the top listed games on BGG.
Catan can be fun, especially the first few dozen times you play it. But it wears thin even for a casual group. We have better casual games these days. And even back then, we did: I found that eg Bohnanza released about the same time as Catan holds up much better to repeated plays; even though it looks much less impressive at first. The big problem with Catan is that the initial settlement placement determines so much of the rest of the game, and there aren't that many meaningful decisions left in the rest of the game.
I did most of my in-person games of Diplomacy when I was working in Cambridge, Cambridgeshire. I was working for a startup spun off from the local university, and noticed that the university societies were actually (mostly) open to the general public. So I just joined the Diplomacy society.
That's very lucky of you. Diplomacy is my favorite game, and some of my friends like it too, but it can be hard to get 7 people committed to something that might last 2 months (or 4 hours if in-person). I'm sure I can find some club in California for this if I try, but I'm already in too many clubs.
In a 'frictionless vacuum', Diplomacy is one of my favourite games.
After a grueling day playing Diplomacy, I usually had second thoughts. (But that wouldn't keep me from doing it again a few months later.) But I did learn that I prefer play-by-email after all. The games are higher quality, and you can actually finish them.
The people who review board games are ... a special bunch. Those games are very popular and well ranked, but they're not considered "best of the best" by people who "seriously play games".
One of the best usages of BGG is to find a game you like, and use it to find games that are like that one, and see what you want.
I know how it is. Some board games are almost like strategy video games with the amount of detail and immersion. Usually those same board game groups play D&D or something as well, like my housemates back in college. I just never liked those. If I'm gonna play a serious game, it's something simple with depth in play style, like Diplomacy.
The problem with the rankings is that some very playable games are ranked low. It's a meme on BGG but CandyLand is famously shit on: https://boardgamegeek.com/boardgame/5048/candy-land - but it's a perfectly acceptable "game" for very young children, and if you've ever dealt with said children you know that getting them to successfully complete a CandyLand session without going backwards on the track, randomly wandering off, taking multiple cards, etc is a serious feat.
It's not a great game but it's certainly still playable in the niche it occupies.