This is definitely not the 'recency bias' I expected.
I've always noticed that when a show is released it gets WAY higher ratings in the first week or so of the release date. I've learned not to trust the ratings until at least a 2 weeks after a release.
It looks like this analysis focuses on the year a film was released. This seems like it would be a lot harder to determine because film quality changes (subjective) and the audience writing reviews is probably changing over time.
tend to have many early bad reviews by mainstream people who can't get past things that they find squicky and the lack of the usual "Hollywood Happy Ending" but that get better reviews from "true fans" later on who enjoy the fact that the directors tried to make something different even if it was flawed.
Interestingly, the opposite of what happened with Star Wars: The Last Jedi.
Casual fans and critics loved the subversion of genre tropes, story roles, character archetypes, and plot points.
"true fans" reviled that the direction align with their vision for the franchise, and it sounds like Episode 9 will be a return to the comfortable and familiar.
In that subthread, I posted a heat map data visualization I made of IMDB rating vs. release year, which is suspiciously similar to the one OP made: https://news.ycombinator.com/item?id=20099344
It's not plagiarism, but if the OP got the idea for checking for recency bias from the HN thread, they should explicitly credit that.
Hey - not OP but I wrote the article. There was a lot of discussion about recency bias in that thread, including the graph you shared and I decided to expand on the analysis. Note there are a few other graphs in my post. I did just draw this up quickly yesterday. But you're right, sorry - I should have cited your comment in my post. Adding it now.
Hi, there are what seem to me (not a statistician) to be a number of serious errors in the article. The most egregious is the first conclusion, which threatens to undermine the whole thing.
> There does not seem to be any strong connection between number of votes and a movie's IMDB rating.
Well, this just isn't true. I happen to know this because I've looked at this data before myself. The issue is that you're cutting off the graph at a maximum of 60k votes with no explanation or even pointing out that you're doing it! I'm sure this is just an honest mistake, but it cuts out basically every movie that is actually popular!
This slowly ate at me until it was enough to get me out of bed to redo this graph myself. I've uploaded my quick and dirty result in R here: https://i.imgur.com/TTuCFEL.png
As you can see from the graph, there's a direct and obvious link between the number of votes and the average rating. The reason I say this threatens to undermine the whole thing is that I also know from previous experience that more recent films tend to have a lot more ratings on average. This (naively) suggests to me that we should see some recency bias if only because more recent blockbusters get a much larger number of votes than others and the sort of people who vote on blockbusters have different standards for what makes a great movie. (Avengers: Endgame was briefly the #1 movie of all time on the top 250 list.)
I don't have time to plot all the different graphs you did, but perhaps you can recheck your results after fixing at least this issue, and get back to us?
Follow up: I redid one of the graphs to satisfy my own interest in the question. Instead of averaging over all the movies ever released like the article did, I averaged over all the votes. This was an attempt to answer the question of whether the average viewing experience of a film from year x is better or worse than that of a film from year y.
I found that there has been a noticeable decline in the average rating of over half a point (out of 10) since ~1930 or so. https://i.imgur.com/bY9vPvk.png
I think this should be explained in a combination of two ways:
* History acts as a filter. If I choose to watch a movie from 1947 it's probably because a lot of people over the years have said it's good.
* To some extent, older movies may really be better than more recent ones.
I still suspect that the blockbuster effect means that the movies people are mostly to watch are going to receive higher ratings than the average film overall. And this is mostly because people rating blockbusters are less critical than the film-viewing community as a whole. So while there might be no "recency bias" of the kind this article was looking for, there might be blockbuster bias where in the last several decades studios have figured out how to capitalize on a less critical / cynical audience. (That hockey stick graph of 8-10 ratings in the article is certainly suggestive.)
Actually -- one more follow up for the zero people who will read this -- there's a simpler explanation, which other people who do these kinds of statistics should probably note.
The average rating between 1920 and 1980 fluctuated at about 7.5. Then the number of movies exploded, and the variability in quality of them exploded even more than that. Movies that were better than the 1980 average had their ratings compressed (you can only do a little better than 7.5 on IMDB). Movies that were worse were much less compressed (you can easily do much worse than a 7.5). So the average gets dragged down since there are now both more movies getting 0-4s and movies getting 8-10s.
I realized this after plotting the median, not the mean, and observing that it stays remarkably constant, and possibly even shows a little recency bias for the last 2 years.
Part of the reason I make data vizzes is so people can expand on the ideas, so there's no issue there! (as long as proper credit is given to the original work)
Doesn't this analysis rest on the assumption that movies are equally good each year? Maybe I suffer from nostalgia (or survivorship bias) but I'm not sure this is a good assumption.
The problem with past movies is you only see the good ones. I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies - because while both suffer from horrible plots usually modern movies at least have interesting visuals.
However for certain categories the past was most definitely better. Dramas being the main one. Without a lot of fancy effects the plot was the only thing you had to carry the movie. They also don't beat things over your head and can embrace subtlety in a way modern movies do not.
I also really liked the 70s trend of movies that didn't really have a satisfying ending, you just get to experience the lives of people who don't really improve. 5 easy pieces or The Last Detail fit that category very well.
> I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies
That's because back then we didn't have the internet. No matter how bad a movie was, people would go see it for at least a week before the word got out how bad it was. Looking up movie times was a pain, you had to plan ahead and check the newspaper. If you went on a date Friday/Saturday night, the most likely thing to do was show up the movie theater after dinner and buy a ticket for the movie that started next, since you probably hadn't heard of any of them, or you would buy a ticket that you'd seen the most advertising for.
There were no ratings to check, no reviews to read, unless you happen to read them in the paper that morning.
Nowadays, they know they have to produce at least something minimally good, because word will get out very quickly that something is bad, and it's super easy for people to check rating when they arrive at the theater, or even before they arrive.
So you want to say that Shutter Island is a bad example of drama? It's all just old-farts rants, nothing more.
Everything is evolving.
In old movies actors play often too dumb, just few of them have good examples of how to play a role.
Some "legendary" movies from the past would fail in theaters nowadays without their legendary reputation.
We have more movies right now, more choices, so our demands are higher - otherwise we will not have enough time to watch them all. Also, we have much more choices of how to watch them - age of digital distribution.
Higher demands give higher quality, movies evolve both in visuals and in acting art.
I would suggest an higher standard deviation on quality, meaning that you could reach very top notch standards (your point) meanwhile mass producing total flat works as in music and i presume, all of the arts (my view).
Lol, compare modern music with trash we had in 80s, 90s. Even nowadays' pop trash sounds significantly better than old hits made on synthesizers. Everything evolves, as I said. Maybe you just forgot low-quality movies of previous years, but they did exist, tones of them.
There were no meaning to mass produce and diffuse music in those decades like there are today. I'm not saying 80s and 90s music was perfect (why pick those decades?), i say the more the demand the more the money the more the highs and the lows.
The "quality" you're referring to, i don't know: you're talking about what features of a sound...fidelity, loudness, dynamic?
Anyway, truly the 80s were the years of trash music...but trash as in metal though.
Good is relative. And you could easily rabbit hole on this. Do people have the same sense of quality? Have people changed their rating scale based on production/effects quality? Etc etc. Probably best to just take the subjective rating.
Incidentally, in case the creator sees this, this site is borderline unusable for me on mobile Firefox. Text gets cut off in strange places, I can't zoom out properly to see all of the animations. "Request desktop site" seems to fix some things but break others - at least I can zoom all the way out but the text becomes hard to read.
Situation might be the same on Chrome (didn't check), but regardless it could use some work since it's pretty broken as is.
Thanks! After playing with it a bit it's clear that part of the problem is that the charts are getting generated on the fly and that's very performance intensive. My brand new Android phone is just choking on the page, to the point I can actually feel the phone overheating. Perhaps static images would be an improvement for mobile devices, with the option to load the full data on request.
Of course, there are plenty of examples of that.
Like the assassination of the NK leader movie, which at first received 10 star rating. Now look at it.
Sane goes with all the new blockbuster movies, you should usually wait awhile for the real ratings to show.
I’m dying to see all of these as percentages rather than absolute numbers! I was even thinking that while scrolling down through, and then there the totals are right at the bottom. @KyleOS are you up for normalizing them?
I see from one of the charts that the average movie rating from 2019 is over .5 higher than the previous 5 years, in what appears to be a big jump. Is it possible that recency bias is only relevant on a shorter timescale? It looks like ratings are higher for movies that are a few months old, based on the data. I wonder if this is generally true, or if early 2019 happens to have higher rated movies.
Sure - for example, fans of a particular actor, director, or series are more likely to go see a movie soon after release in theaters as opposed to later via rental or streaming.
Fans are likely to rate higher than the general public, so over the first, lets say, 3-12 months, ratings often trend down.
Fake reviews and ratings must be a factor. For big movie releases it is hard to find IMDB reviews that seem genuine. I assume good portion of the ratings are fake as well.
Great job! I'd love to see a similar analysis for Rotten Tomatoes since there's separate metrics for critics and audience, and I find RT ratings to be more extreme than IMDb. I'm often surprised to see how many bad movies get a rating of ~6 on IMDb.
The problem with your formulation is that 'better' is perhaps the most subjective word there is. Define it in measurable terms, and perhaps you have a reading there. There are many factors to consider which may lead you to find things less clear-cut, for example here's one or two, with some sub-factors
- Modern audience Preference for familiar patterns in: (eg)
-- acting styles
-- music used
-- visual tricks and norms
-- special effects styles
-- acting styles
-- soundtrack loudness
-- colour and high resolution
-- High Concept movies
-- Fantasy and Comic-related films
-- modern movie themes and topics
- Internationalisation of movies, so that fans from abroad prefer modern films which are more universalised and less American-specific (see literature)
Right, I'm not asserting that my alternative title is the correct one, I'm lampooning the assertion that IMDB is "suffering from recency bias" as opposed to a more neutral claim like "ratings for more recent movies are higher."
Answering the question of _why_ they went up is a lot harder to prove.
In terms of shows, Sopranos has long been dethroned, possibly due to this effect. Then again, production values will only get better over time, so maybe the latest shows are of the best quality overall.
Best line:
Fucking Queeahs! (Paulie throws chair at his murder victim spirits)
I've always noticed that when a show is released it gets WAY higher ratings in the first week or so of the release date. I've learned not to trust the ratings until at least a 2 weeks after a release.
It looks like this analysis focuses on the year a film was released. This seems like it would be a lot harder to determine because film quality changes (subjective) and the audience writing reviews is probably changing over time.