I work on the algorithm for a widely used search engine and can confirm that this line of thinking has been very effective in improving our product over the years.
Rather than trying to generate hypothetical ideas for "how can we make our search better", we spend a lot of time analyzing our data to find where we are failing. Many of our biggest relevance improvements have come from tracking and understanding the types of queries where we consistently fail to generate results or user engagement.
I think it is a very effective approach, but can require some discipline and perspective. When you spend so much time focusing on the failures of your product, it can create this internal perception that the product is constantly failing and broken. So you do need to actively remember what you're doing well and how far you've come as a team/product.
> Rather than trying to generate hypothetical ideas for "how can we make our search better", we spend a lot of time analyzing our data to find where we are failing. Many of our biggest relevance improvements have come from tracking and understanding the types of queries where we consistently fail to generate results or user engagement.
This sounds a lot like the 6-sigma approach of driving improvement by focusing obsessively on eliminating "defects".
There are certainly huge wins that can be obtained by identifying and eliminating bugs or corner-cases with undesired behavior. But it's scary to imagine a world where this is used as a replacement for innovative thinking - ie, "how can we make our search better". If Steve Jobs had focused all his proverbial efforts on minimizing flip-phone defects, the world would have missed out on the smartphone revolution.
The iPhone's competition was not the flip-phone. It was the PDA and the Blackberry and the pocket PC. The iPhone was an evolution of previous similar devices.
I still do not understand why people consider smartphones revolutionary. It is revolutionary that everyone has one on them at all times, but the gadgets themselves aren't all that.
Yea I don't think it's the only principle that should drive product development, but it helps ensure you're always solving real problems for your users.
There is still a lot of room for creativity once you've identified a class of problematic queries too. Especially as a search engine becomes more sophisticated, how you solve clear query failures can be a lot less straightforward, and clever features or machine learning are many times needed.
I will say there are clear exceptions to this inversion rule too. For example, we switched to a Learn-To-Rank system for our core ranking in the past year and we couldn't necessarily point to it clearly being the solution for problematic queries we were seeing, but it proved to unlock a ton of value and drive a lot of relevance improvements and surprising benefits in ways we couldn't necessarily predict for our specific use case and users.
That's why it's important to focus on effectiveness first. At any point in time you should know what you are trying to solve and why. The Inversion Principle is simply a useful tool to helps support that and figure out the how, but is by no means a silver bullet.
There's a huge potential in mining search data, especially if you can group top-of-funnel vs. bottom-of-funnel searches to see where failures are occurring. Segmenting queries like "zm950" against "shoes" or "nike" and seeing where gaps exist against user intent.
When it comes to zero (or near-zero) results, I've had good results using this to identify gaps in the current product offering and what visitors are expecting to be there. Two examples:
1) A seller of custom prescription glasses: top two search queries were "contact lenses" and "sunglasses". They didn't offer the former, they did sell sunglasses (most frames could take a tinted lens as an option) but didn't make it obvious with design, content, or marketing.
2) A seller of cabinet hardware (pulls & knobs): a large proportion of their top 10 search terms seemed to have a door hardware intent. Adding this missing category boosted sales without additional marketing dollars spent (the customers were already there and just bouncing when they realized the site didn't carry what they wanted).
These are all ways to focus on understanding failures instead of trying to optimize successes, which is often finding the local maxima.
Rather than trying to generate hypothetical ideas for "how can we make our search better", we spend a lot of time analyzing our data to find where we are failing. Many of our biggest relevance improvements have come from tracking and understanding the types of queries where we consistently fail to generate results or user engagement.
I think it is a very effective approach, but can require some discipline and perspective. When you spend so much time focusing on the failures of your product, it can create this internal perception that the product is constantly failing and broken. So you do need to actively remember what you're doing well and how far you've come as a team/product.