There are several flaws inherent in trying to scale things by having the computer gather all your data from people.
Solving them almost invariably requires more people, but some can be avoided if you're more careful about what you measure and how you measure it (i.e. making sure you know what the hell is actually going on, rather than staring at spreadsheets of disembodied numbers removed from any context).
* The "best performing" systems are those with the fewest problems, right? But you can get rid of problems in two ways: by solving them, and by making it difficult or impossible for people to complain.
* You only hear about problems from the people who complain. Remove their ability to complain (or make it hard for them to reach anyone) and you'll never know about the problems. The tie-in with my first point should be obvious.
* A broken complaints system that allows few complaints will "perform better" than any other systems (in terms of having fewer trouble reports that get through) and will get selected for, unless you have people who know what the hell is going on.
* When you make people select their problems from a list, you won't get anything not on the list. Sure, maybe you have an "other" field, but sometimes people will just select one of the options even if it's the wrong one. In that vein, your default selection should be some kind of "unknown/other" because unless people make a choice, you can't assume that they intentionally selected anything.
I could go on and on, sadly, because I have a lot of experience seeing this sort of thing go on, even though I've done my best to prevent certain industrial QC processes from feeding total and utter garbage data to managers who see the production floor only when giving tours to prospective customers.
Solving them almost invariably requires more people, but some can be avoided if you're more careful about what you measure and how you measure it (i.e. making sure you know what the hell is actually going on, rather than staring at spreadsheets of disembodied numbers removed from any context).
* The "best performing" systems are those with the fewest problems, right? But you can get rid of problems in two ways: by solving them, and by making it difficult or impossible for people to complain.
* You only hear about problems from the people who complain. Remove their ability to complain (or make it hard for them to reach anyone) and you'll never know about the problems. The tie-in with my first point should be obvious.
* A broken complaints system that allows few complaints will "perform better" than any other systems (in terms of having fewer trouble reports that get through) and will get selected for, unless you have people who know what the hell is going on.
* When you make people select their problems from a list, you won't get anything not on the list. Sure, maybe you have an "other" field, but sometimes people will just select one of the options even if it's the wrong one. In that vein, your default selection should be some kind of "unknown/other" because unless people make a choice, you can't assume that they intentionally selected anything.
I could go on and on, sadly, because I have a lot of experience seeing this sort of thing go on, even though I've done my best to prevent certain industrial QC processes from feeding total and utter garbage data to managers who see the production floor only when giving tours to prospective customers.