I'm really glad a16z put this issue to rest because it's the kind of problem that causes insane wheel spin at a certain kind of company. The "Data Acquisition Cost", and "Incremental data value," made me laugh because I had to solve that problem before.
The big mistake I've seen data companies make is they approach their market based on customer vertical, assuming that say a Bank will be like other banks, and a health care provider will be like other health care providers, and this is the fatal error. The bank with the same sensitivity to fp/fn/tp/tn rates will have more in common with the health care company with that same sensitivity than it will with other banks.
Basic problem with any data product is the customer's ROC curve, or where they economically benefit from using your data service. Different customers have different sensitivities to false positive/negative, true postive/negative rates - and the customer categories themselves are defined by this sensitivity. e.g. What they have in common is not their vertical but their risk appetite. I have a blog post 3/4 written on this specific topic.
That sensitivity is specifically an artifact of the customers growth stage as a company, which determines their risk appetite and the economics of the asymmetrical value that the effective ROC curve of your data product describes. (see above link).
This is the fundamental problem for an ML/AI company, where they will go bankrupt trying to find their 2nd or 3rd insurance company customer because they think the value of their product is because their next customer is in the same vertical - not because they have the same sensitivity to fp/fn/tp/tn.
Slight aside, it's so important that an investor like these can weigh on in these issues and other technical economic factors, because IMO when I listen to every technical person I know, the #1 cause of internal suffering at companies is caused by people trying to bullshit their investors, and blog posts like these just wipe away a big source of that temptation.
The most interesting related work on this is a paper covered on HN at some point (http://blog.mldb.ai/blog/posts/2016/01/ml-meets-economics/) about economics' indifference curves and ML's ROC curves.
The big mistake I've seen data companies make is they approach their market based on customer vertical, assuming that say a Bank will be like other banks, and a health care provider will be like other health care providers, and this is the fatal error. The bank with the same sensitivity to fp/fn/tp/tn rates will have more in common with the health care company with that same sensitivity than it will with other banks.
Basic problem with any data product is the customer's ROC curve, or where they economically benefit from using your data service. Different customers have different sensitivities to false positive/negative, true postive/negative rates - and the customer categories themselves are defined by this sensitivity. e.g. What they have in common is not their vertical but their risk appetite. I have a blog post 3/4 written on this specific topic.
That sensitivity is specifically an artifact of the customers growth stage as a company, which determines their risk appetite and the economics of the asymmetrical value that the effective ROC curve of your data product describes. (see above link).
This is the fundamental problem for an ML/AI company, where they will go bankrupt trying to find their 2nd or 3rd insurance company customer because they think the value of their product is because their next customer is in the same vertical - not because they have the same sensitivity to fp/fn/tp/tn.
Slight aside, it's so important that an investor like these can weigh on in these issues and other technical economic factors, because IMO when I listen to every technical person I know, the #1 cause of internal suffering at companies is caused by people trying to bullshit their investors, and blog posts like these just wipe away a big source of that temptation.