Monti's a priori knowledge absolutely matters and should have been stated in the article. If he had no knowledge beforehand about which door the prize was behind, then you would have to expand the probability table to include cases in which Monti opens one of the remaining two doors and accidentally reveals the prize, in which case the user just loses without getting the opportunity to switch or not switch. Monti knowing the door is hiding the prize and actively selecting the non-prize door versus just guessing and sometimes accidentally revealing the prize, are two different statistics problems.
What this article misses out is that the metrics will be present and used whether they are articulated or not. By making them explicit they can be improved, validated, debated, and removed. Standards are consistent. When they're implicit, you get none of those benefits and instead promote an insider culture of unspoken biases that get no scrutiny at all.
The choice isn't between metrics and human intuition, it's between explicit and implicit metrics.
Of course there are bad metrics; to use an example from the article - measuring the output of analysts finding OBL on a binary metric is simply a bad metric. But if we start from there, we can improve it. How many leads do they have at the current time? What stage is each lead at? What intelligence has it generated to this point?
This sounds very much like OCD. One of the nastier traits of OCD that I experienced with a partner was the tendency for the OCD to hide itself, by convincing the sufferer that they were just a terrible person, or that what they were doing was normal. And this was post-diagnosis. My interpretation was that accepting you have OCD is the first step to acknowledging that the repetitive behavior isn't necessary, which of course the disease fights against.
It was very hard at times and there was often hostility but it was always recognizable to me as a symptom of the disease and not an intrinsic quality of the person.
I met another person with OCD who didn't know they had it - 5 hrs of cleaning each day and they didn't know, they just knew things needed to be clean.
I strongly encourage you to do what you can to get to a diagnosis. What helped the person I was close to in the end was cognitive behavioral therapy and a lot of hard work, to get to a point where things were manageable.
Finally - the biggest mistake I made as a carer was not talking to others about it or recognizing and acknowledging the effect on me. Get support for yourself if nothing else.
Almost certainly because it increases engagement. By optimizing purely on metrics, psychologically undesirable effects creep in (guilt, in this case). c.f. online gaming apps and their addiction inducing behaviour.
I think it's important to acknowledge the distinction between chatting to a bot and not being convinced that it's a human, and having two conversations simultaneously, one of which is with a human, one with a computer, and figuring out which is which. Humans too can be surprisingly obtuse, go off at tangents, ignore your points, be boring and repeat themselves.
As for the questions around the utility - automating support functions for various non-critical services seems like an obvious potential application (think of all those "click here to chat to a live representative" dialogs on various websites).
Rather than starting with an assumption on avg revenue per user, start with the valuation, figure out what that would imply about revenue/revenue growth, and you'll probably learn something about their likely avg revenue per user (i.e. it's a lot higher than your estimate).
Aside from medication, for OCD in particular there are therapeutic options that can be (life-changingly) effective. The OCD centre [http://www.ocdcentre.com/about-us] is one such place. If you think you might have OCD (last I read the incidence was around 1/30) talking to someone about is the best first step. Especially as one common and pernicious effect of OCD is that it will convince you that you do not have it.
Extension: what if Monty tosses a coin to decide whether to open a door, conditioned on you having chosen the prize door?
Specifying Monty's behaviour matters, I think.