Is anyone trying to solve this right now? It seems like a really interesting problem that would be really useful to solve.
It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback.
Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?
Many services have this issue, that perfection is the minimum requirement. I don't understand why, given this, we don't just go to a thumbs up / thumbs down rating system. We have one option that means "fine" and four (or nine) that mean "terrible, and a risk to your business".
I know I personally gave plenty of three star reviews on Lyft before I learned better, meaning them as "no complaints". Who knows what happened to those drivers as a result. Why even invite the misunderstanding?
This could be a cultural design thing. A lot of these apps (Airbnb, Lyft, etc) are from the US, where getting full marks in school is a frequent occurence if you study reasonably well.
In France for instance, it's very rare for a student to go above 15/20 or so at the college level (I graduated in the top 10% of my class with a 12/20 average; the top ranked student, whom I was friends with, had a 14/20 average).
But when I graded homework as a graduate student for an american university, the professor told me I was too harsh, and after adjusting my criteria to match what she wanted, I found myself giving many 100/100.
The US is also where you insult service staff and deprive them of income by giving them a gratuity which would be considered generous thanks in most tipping cultures. But at least that's baked into one's understanding of the price.
When it comes to ratings used by others to evaluate, no system is more customer and vendor-hostile than one where everything that meets a minimum threshold for adequacy should be rated as perfect.
In a lot of ways, yes. On the other hand, it seems like a hard problem to solve - how do you get honest feedback in situations like this, where you want to use it to both improve things overall and judge your employees/contractors? Maybe you can't.
I guess, at least a small step would be to make it so values less than the highest rating don't count against you so negatively - it should be ok to be a 3/5 or 7/10.
It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback.
Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?