A single mediocre experience optimized to work ‘most of the time’ for ‘most people’ is quite contrary to the narrative that has made Google such tremendous amounts of money (“let us surveil you so that you can have a more personalized experience”) though, isn’t it?
Given all of the data collected about Google users, ought not one of the applications of that data be some way to give users specifically what they are searching for if their past behavior suggests that they mean what they type? Couldn’t the “search only for <exact query>“ option be a very good data point on making that determination automatically, or enabling a user setting for “give me exact results based on what I actually typed by default”?
It seems possible to me that this behavior has more to do with the value of ads for “big” keywords than with (poorly) inferring user intent.
I have a sense that this is the dirty little secret of the spyware advertising industry, personalization just isn't that great. Yeah, putting you into a male or female bucket, parent or child, homeowner or renter, that's worth a little bit. But, to find out your name and address and search history and how long your last bowel movement took, just to deliver an ad that's theoretically hyper-optimized to make you buy something... I just don't believe it.
I don't believe that it's worth anything near what they are charging for it, except perhaps in the case of politics, which has always been an extremely efficient use of money. And even then, it's not worth a tiny fraction of the real cost it has to society.
Netflix has achieved the dream of movie studios going back more than a century now. They have the talent, the money, and more than two decades of data. Netflix knows what you watch, when you stop watching, how often you watch, which movie covers work best.
And yet, it's hard to look at Netflix as anything more than a total failure of the promises of data analytics and personalization. Netflix should be putting out nothing but hits. A dozen Breaking Bads or Game of Thrones.
Yet they are not. In fact, they do not even have a single show that is to the level of Mad Men, Breaking Bad, or The Wire. HBO and AMC are running laps around Netflix. Meanwhile, Netflix is making live action Cowboy Bebop and cancelling it before people even know it existed. I'm really curious what the data said about funding that particular project. On one hand, you have the cult following of the anime that will absolutely tear a live action version to shreds. On the other hand, you have to convince the uninitiated into viewing a remake of 23 year old anime.
Then there is the personalization. The fact that there is a meme about spending more time browsing the Netflix catalog than watching content tells you everything you need to know about how little people trust Netflix recommendations. Their new "top 10" feature is just depressing most of the time. It looks like a list of ten random DVDs in the bargain bin near the Walmart checkout line. Oh, and, their top 10 feature is currently the biggest recommendation feature on their site. And it's not even personalized! If that's not a complete admission of defeat I don't know what is.
Netflix has produced a lot of fairly solid content. Not at the level of the all-time great prestige TV shows like the ones you mentioned, but enough to keep a lot of subscribers happy for long periods of time at an accessible price. House of Cards (at least until the Kevin Spacey scandal blew up), Stranger Things, Orange is the New Black, BoJack Horseman, Disenchantment, etc are a few that come to mind that I watched and enjoyed.
I think Netflix has two big issues. The first is the way they drop seasons all at once prevents the natural cycle of pre-episode hype, post-episode interviews, speculation and leaks, fan anticipation, fan arguments (ship wars, etc), etc. Fan culture can't develop around this content as easily because there's never any breathing space for fans to collectively sit with the story so far. The shows that become a cultural force like Breaking Bad or Game of Thrones need us to keep coming back to the conversation every week. They have us talking about what happened last week and will happen this week with our colleagues at work and around the dinner table. How can these Netflix IPs enter stable orbit in the cultural zeitgeist when they're once-a-year events? It devalues the work, positioning it more like a movie that you watch and then forget rather than a story you become invested in over a long period of time.
Full season drops also allow people to binge a whole season of content for a single month's subscription and then immediately churn (ask me how I know). I assume they have some data-driven reasons for doing this, but it makes absolutely no sense to me.
Besides that, I think they should put more wood behind fewer arrows. They've developed a reputation for aggressively cancelling smaller shows with passionate followings which makes a lot of people not even want to bother until something has become an established mainstream success. I think of them now as the Google of content-creation, putting out a lot of solid (but not amazing) products and then cancelling them once people start to grow attached.
Wrote something similar before:
"The writing has been on the wall for some time:
1. Grading system changed from 1-5 to 1-2 (thumbs up/down). They thought that the users where full of crap when rating. I do believe some bosses just looked "bad" when buying in the next Adam Sandler movie. This started a cozy culture where no one in Netflix was wrong. Recommendation engine becomes comically bad, even with the best and the brightest.
2. They started to buy everything under the sun. South park made an episode about it even. All the comedians got their own stand up specials. It was now way easier to get a top score (thumbs up). Bosses where happy.
3. As they no longer focuses on quality which they no longer can measure (measuring time watched and churn is not that useful!), they start to strive for quantity. Which is expensive, very expensive.
I guess that in the next decade Netflix will become the next Comcast and cost 35 USD per month, and it all started in an innocent change to the grading system."
That's not how the entertainment business works. If I had to guess, Netflix's data-driven approach to content production is like card-counting in blackjack. It only gives them a slight edge on the house. A net positive outcome over hundreds or thousands of hands but offering no guarantee over the outcome of any single hand.
> HBO and AMC are running laps around Netflix
Netflix did win the most Emmys in 2021.[1] And they only started producing original content in 2013 or so. That's pretty good.
> Then there is the personalization. The fact that there is a meme about spending more time browsing the Netflix catalog than watching content tells you everything you need to know about how little people trust Netflix recommendations. Their new "top 10" feature is just depressing most of the time. It looks like a list of ten random DVDs in the bargain bin near the Walmart checkout line.
They may also be optimizing for revenues as opposed to recommendation quality (homegrown content being cheaper than licensed)
Part of this might just be account switching issues though. If I’m watching for myself I can usually find what I want quickly. The problem comes when I’m trying to browse with my wife to find something we can both live with. At best it gives the my preferences and the Union of me + wife’s preferences (and vice versa on her account.) But what we’d really want is a separate recommendation feed that shows the intercept of me + wife’s preferences.
But most people are lazy and won’t account switch for different contexts like that anyway, so there’s just no way they can keep the profile data as clean as it needs to be for a television.
Netflix is not recommending what they think you'll like, they are recommending what they want you to watch. Once you're a subscriber, they want to keep you there as cheaply as possible.
This is exactly what their data analytics has told them to do.
Have a few popular, quality tv shows with star-studded casts as loss-leaders to bring in new viewers. Otherwise the model is to produce and recommend the shows that get them the most eyeballs per dollar; the bare-minimum to keep their subscribers there:
- Stand up specials are dirt-cheap, quite popular, and provide never-ending variety.
- Ditto with 'reality' shows, bake-offs, make-offs, expose documentaries, etc.
- Old sitcoms and b-movies that have a proven re-watch-rate.
Throw in a handful of first seasons to keep the FOMO up, and you've got a captive audience on the cheap. Maybe one or two will catch on and become the next loss-leaders.
They may not have the quality shows that are 'running laps around' HBO and AMC, but by any of the metrics Netflix cares about they are simply running laps around HBO, AMC and everyone else.
>And yet, it's hard to look at Netflix as anything more than a total failure of the promises of data analytics and personalization. Netflix should be putting out nothing but hits. A dozen Breaking Bads or Game of Thrones.
I don't follow this argument. Knowing what people like has very little to do with the quality of original creative content; surely you don't expect Goodreads to put out Shakespearean novels, or Spotify to be producing original hits on par with the Rolling Stones? Should ESPN have better pro sports scouting and coaching talent than the professional leagues?
Knowing what people like, however, _does_ have to do quite a bit with selling those people a product - which Netflix just reported 15% YoY growth to $7.7B yearly revenue, they're clearly very successful to this end. I think you actually have it backwards - if anything, Netflix represents a total fulfillment of the promises of data analytics and personalization. Despite mediocre original content, this is a $200B company with 200M subscribers growing revenues by double digits two decades after IPO.
If Netflix paid $450k+ salaries to screenwriters instead of engineers, you'd very likely get better movies on a worse streaming platform. And when Netflix has shelled out for Hollywood talent, like Mindhunter which has David Fincher and Charlize Theron, the results are quite good.
Regardless, to take the fact that Netflix pays for premium engineering and analytics talent, but does not pay for premium filmmaking talent, and then spin that fact into Netflix being "a total failure of the promises of data analytics and personalization" is a questionable criticism.
Strong comment. I agree completely with your negative assessment of the “value” of consumer habits to optimize Netflix recommendation. In my own case I feel trapped in a very shallow local minimum. Yes I watched a revenge flick or two but now I am type-cast for life.
Netflix really doesn't need to produce something like the wire or mad men or breaking bad. There's no reason to make a show that appeals strongly to 70% of the market when they could make 70 shows that 1% of the market is fanatical about. They don't have only one channel that competes for content and they don't seem budget limited.
Ironically, it seems like AmazonPrime is far better at that.
As for then top ten being bottom barrel stuff, I think you overestimate how popular mad men was vs. something like king of queens.
I will say, Netflix seems to fail in many cases, and I don't understand how they think content discovery is supposed to work.
Right, people radically overestimate how much a profile is worth. Someone who owns a house in a rich area is somewhat easy to identify, and you target them... along with everyone else who is also trying to reach that rich slice, so you pay more.
The very high quality pieces of information can be things like "wants to buy a life insurance policy this week" or "just had a baby" or "just bought a plane ticket to XYZ," or "is in the frequent flyer program and spends more than $20,000 per year on travel."
However the majority of information about people, the overwhelming majority of whom have no significant disposable income, is worthless and not worth tracking for the most part. You reach those people through traditional mass marketing means.
It's a great point you have made. I work technically as a data scientist, but my domain is scientific data. I have quite a few GitHub packages and get recruiter calls for data science jobs almost every week, with pretty generous salary offers.
And from what it seems to me, there is a giant bubble. The vast majority of companies doing "data science" jobs are things that a smart undergrad can do with a month or two of training. And this is because I believe C-suites have completely gulped down the data is oil mantra. There are entirely charlatan companies with unicorn, even decacorn valuations now being built on this mantra - for example, CRED in India.
Yet, as you said, and as I believe too, most of the data is worthless.
Right, but unlike oil, most data is worthless most of the time.
For example, I'm about to list my house on the market. The real estate agents who reached out to me last year and the year before that to try to induce me to sell the house had no chance of succeeding. Now is the time if they knew the secret that I'm about the list the house that they should all be competing for my business. The only relevant piece of information is that I'm about to sell the house. That is a valuable lead that many in my region would bid on. My background information is frankly not that relevant to the value of that lead, and isn't something that is readily surfaced by the kind of deep profiling that is supposed to be going on. It can be signaled by me signing up to some kind of list that sells my lead to a zillion people at once, but is never going to be surfaced accurately to the people who can earn the most profits from it by my Youtube habits or whatever.
Zillow sells the logged in user data in the market I'm buying in to the real estate agents listing the houses, but that again is not something being modeled by some kind of big data operation, but is merely the same kind of "little data" provided on things like dating websites or LinkedIn when people browse your profile. There's no modeling going on there that requires sophistication.
> However the majority of information about people, the overwhelming majority of whom have no significant disposable income, is worthless...
I've had a supposition for a while now that the targeted advertising industry should be closing the consumer cashflow loop by advertising effective self- and employment improvement, with the objective of increasing the disposable income people have so they can then make more brokering traditional sales
>I have a sense that this is the dirty little secret of the spyware advertising industry, personalization just isn't that great.
Personalized adverts and recommendations can be incredibly, horrendously dumb.
Here's what I see when I hit amazon's homepage at the moment :
A "buy once again" column that features blackout curtains I bought 3 months ago (no, curtains don't need to be replaced every months, amazon.), USB cables I bought multiples of in the same time frame, a wireless charger (I already bought two before).
An entire line dedicated to showing me backpacks (I bought one less than a year ago)
An entire line dedicated to headphones (I recently bought wireless IEMs)
An entire line dedicated to watches (same)
I don't get it. Supposedly the best and brightest work at firms like amazon and google to brainwash us to buy stuff, but classic, random, non-targeted advertisement is more likely to make me discover products I'd buy than targeted advertisement because the latter only shows me things after I don't need to buy them anymore!
Here's what I would expect actually intelligent targeted advertising to do :
After buying a smartphone, recommend accessories (cases, screen protectors, USB-C dongles, chargers, whatever)
Here's what targeted advertisement actually does :
show me smartphones ads everywhere I go after I already selected and BOUGHT a smartphone. No, I don't need to buy another smartphone weeks after a recent replacement, amazon!
The same sort of phenomenon can happen after google locks on searches I did to buy something. I can't wait to see the internet advertisement industry crash and burn, it's overvalued nonsense.
A "buy once again" column that features blackout curtains I bought 3 months ago no, curtains don't need to be replaced every months, amazon.),
Disagree there. About 75% of the things I buy on Amazon are repeating purchases that I nevertheless don't want to be automatically scheduled. It used to be a real pain in the neck to reorder something manually, so I'm glad they made that easier.
But yes, in general, Amazon is full of low-hanging fruit that's been neglected on the tree for a decade or more. Buying clothes from Amazon still manages to be a worse experience than going to the mall, for instance, which is really saying something.
I think the fact that most recommendation algorithms have seemingly converged on what seems like a really poor and naive implementation - fixation on very recent activity - shows that the sort of deep personalization touted is mostly BS.
Both YouTube and Amazon heavily personalize by recommending primarily the 3-4 things that I've interacted with in the very recent past.
This is not true. For example every time Summoning Salt uploads a video, which happens every few months, it will show up on my recommend feed because YouTube knows I'm willing to watch their ~1 hour documentaries even though I'm not subscribed to them.
Youtube seems to be a rare exception here in that people actually feel like its algorithm is useful. However, even then, their algorithm mostly seems to devolve to "what creators have you usually watched videos from" and (usually directly after you watch such a video) "what videos did other people who watched that video watch?" Basically the same principle as PageRank, just with a lot less spam to deal with.
This could (probably isn't) be a very quick implementation with a heuristic like 'if percentage of viewed videos from channel x (essentially per channel viewed) > threshold ==> show new video from channel x on homepage next time user appears.
Make it fancy and use a multi armed bandit and call it machine learning/AI/data science.
The most common pattern I see relating to personalized advertising as someone being advertised to is that I will often see an ad for something I just bought (or some competitor to it) repeated relentlessly for a couple of days after buying it and this is after not seeing any related ads during the days prior where I was actually doing some research into the product space.
Maybe I'm an outlier but they seem to miss the window of relevance on me often enough that I notice it as a commonly repeated pattern.
I know it seems moronic, but I think it might actually make sense from the advertisers point of view. Some percentage of people who buy a thing are going to return it and buy something similar in the next week. That percentage is almost certainly large compared to the percentage of the overall population who's going to buy that thing in the next week, and it seems plausible to me it's even large compared to the number of people who have been browsing for the thing but haven't bought yet. (Think of it as the ratio of people just browsing vs ready to buy.)
Even so, wouldn't it be much smarter if they kept track of what the expected life expectancy of the thing you bought is, and then years later start feeding you ads for a replacement? Or is it too hard to track people over such a long period of time?
I don't think any advertisers would be able to offer "People who bought a washing machine 3 years ago" as a category that can be targeted without a riot
Same experience. What's even more mystifying is that often it is for items that no human would be likely to be buying many copies of in a short span of time (high ticket items, or items where you probably don't need more than one).
Just because I bought something doesn't mean I kept it. And those 0.1%, or whatever, returning items are very likely to buy another one of a different brand.
Exactly. If I just bought some power tool for a home improvement project, I am the least likely person in the country to want to buy that exact same power tool the next day.
Not if you hate it and want to return it. In fact there’s a calculation to be made - what percentage of people return or dislike their drill? Because that subset of the population is probably more likely to be looking to buy one than any other.
A return rate of say 1% may lead to more people looking to buy a drill who have just bought one in the last week than people looking to buy their first drill.
> In fact there’s a calculation to be made - what percentage of people return or dislike their drill?
If that's true, they left something out of their calculation: What percentage of people will install an adblocker as a result of feeling like they're being hounded for a few weeks? This scenario was mentioned specifically by Tim Cook when he introduced the Safari anti-tracking features.
I haven't consumed significant amounts of ads in a long time, only some logos in sports and the occasional visit to family or the rare times adblock fails (YouTube premium user too). So I can only imagine how hilarious that must be.
I think you're right. I'd like to see an analysis of the effectiveness of personalised advertising based on tracking versus ads based purely on local context. The latter being if you're on a web page about birds then you get ads for bird seed and bird houses. No tracking involved.
It's always fun watching an ad system try to figure out nonbinary people. Spotify ads can't decide whether I'm a successful businessman or Spanish-speaking housewife.
It's always fun checking Google's ad settings and seeing what they think I'm into.
Apparently now I'm into baseball, flowers, boating, celebrities, country music, credit cards, geology, event ticket sales, fishing, and windows OS. Among a couple hundred other things. It even gets some rather basic facts (marital status, company size, education) wrong. I seriously wonder how they generate this profile?
Well... It actually got some categories right, but I don't really feel that's very impressive considering that it put me in every category by the looks of it.
It fascinates me to see how the ad algorithm responds to people who watch content in multiple languages. I study a lot of languages as a hobby, so I often watch YouTube videos that are in Mandarin, like news broadcasts and niche hobby channels. YouTube has now started showing me ads (in Mandarin) which seem to be targeted to Mandarin-speaking immigrant parents of young children who want a way to teach them Mandarin despite my, and my spouse’s, very busy careers. I find this amusing because I am a single, pasty white man in my 20s.
That’s just Spotify. Many years ago they had a little tool that actually reported what demographic slots it pegged you at based on your listening preferences. The top two hits for me were 1.) early 20s, college educated, White, woman 2.) 60+, blue collar, African American, male
At the time I was a late 20s, college educated, South Asian male. I’m very cis and very straight. And yeah my musical tastes are pretty eclectic, but that was a weird profile to settle me on.
Some people fit in convenient buckets, but lots of people don't, and assuming all people do, will make the ad system useless to a lot of people. Even if you're not non-binary at all, you could still be a successful businesswoman or a Spanish-speaking houseman (househusband? stay-at-home dad?).
Better to just follow people's interests, instead of using their interests to incorrectly pigeonhole them and then drawing incorrect generalisations from that.
Something about my actual interests and activity apparently makes youtube think I'm into Fox news and all the crazy shit found there.
Now, who else has this same value judgement about me? This assessment that I neither declared for myself nor even ratified.
It's annoying but ultimately harmless that youtube shows me conservative wackjob stuff.
But is that same profile in someone else's database that marks me as someone to watch or something? Does it affect my insurance rates, my liklihood to get extra scrutiny when travelling, my ability to purchase or register a firearm, my access to jobs that might be extra sensitive or responsible, basically any of the things where someone either private or the state does any sort of background or credit check on you for any reason, and there are really many of those when you think a out it.
I'm guessing, today, it's probably not really affecting my life in any real way, but, there is no way it makes any sense to say that will still be true tomorrow.
There was that infamous case of a retailer figuring out someone was pregnant before they did based on what they were buying and mailing a customized flyer...to their dad's house. I don't remember the exact situation, but it probably wasn't the only incident.
That btw is an anecdote from the association mining community.
I spent a lot of time learning about association rule mining in my AI courses, including the implementation details of competing ways to mine them. The technique seems extremely useful and fascinating (I jury rigged it for on the fly league of legends champ recommendations to maximize calculated win rate change given limited information), but I almost never see it used in the real world or even see it talked about anymore.
I believe you're referring to a rumor (which may be true, I just mean it in the sense that it's out there and not something you or I have verified) about Target.
My experience working in a similar domain (NLP summarization, which leverages methods like text rank which are identical to pagerank but for text summation) is similar.
Personalized page rank is not significantly better at summarization in my experience, even "queryable" summarization, but that also could be a pure implementation problem or a problem of hyperparamater selection...
Does it really work so well in politics? I've read in various places that a lot of political advertising in America functions basically as a means for channeling donors' money to a few K Street firms belonging to party insiders.
I agree with you. I highly doubt that our economy has enough (product, message) combinations to justify the need for personalization based on more than a dozen attributes.
I will buy X, if I need X. And once I buy X, it's done. For example, I wanted a cordless drill last week. Did the "site:reddit.com" thing (I actually have been doing that almost subconsciously now, as Google results are all trash), chose a drill, and ordered one off Amazon.
Then, after that, what's the point in showing drill ads to me for two weeks?
There's a well known effect in advertising that advertising a product to a person that has already bought that product generally increases their satisfaction with the product and the purchase, and may cause them to recommend the product to others.
Probably that's what they are going for if they're doing it on purpose.
> There's a well known effect in advertising that advertising a product to a person that has already bought that product generally increases their satisfaction with the product and the purchase, and may cause them to recommend the product to others.
Do you have a link for further reading on that? That's fascinating if true.
It's not supposed to feel good. If 9/10 people have a brief negative thought about the advertising experience and nothing else happens, but 1/10 people happen to have their friend on the phone at the time and makes a referral, then overall that is a win for the brand.
It's pretty straightforward to understand that when the vast majority of your sites income is generated from ad revenue, that data is being used to optimize for generating ad clicks, etc., rather than actually giving users the best/most relevant/useful/desireable information for their purposes.
From the behavior of ads (that I imagine are highly optimized), all that knowledge is useful for front-running an specific TV model all over your internet once you decide to buy a TV.
It seems to be completely useless for anything else, and specifically harmful for product discovery, that is the one way ads add societal value.
I worked for a healthcare recruitment company in a capital city with some large hospitals and a number of universities. I can't for the life of me understand why they chose to spend so much money on trying to track healthcare professionals online when they could just advertise it on-premise where they actually hang out.
I can't understand why every company want me offer “personalized” service. It never works, and if I can't manually set preferences, then it's not personalized (because “personalization” means exactly that).
Given all of the data collected about Google users, ought not one of the applications of that data be some way to give users specifically what they are searching for...
You're missing what "personalization" has come to really mean. It means knowing enough about the user to give them an experience you can profit from and which they will accept. If there isn't something you can expect profit from, there's no reason to give them anything.
Given all of the data collected about Google users, ought not one of the applications of that data be some way to give users specifically what they are searching for if their past behavior suggests that they mean what they type? Couldn’t the “search only for <exact query>“ option be a very good data point on making that determination automatically, or enabling a user setting for “give me exact results based on what I actually typed by default”?
It seems possible to me that this behavior has more to do with the value of ads for “big” keywords than with (poorly) inferring user intent.