The headline is "interview performance is kind of arbitrary," but the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary.
I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?)
The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place)
That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.
Thanks, as always, for the thoughtful notes. We certainly don't mean to imply that this is gospel, and there are limitations to the work, and fwiw, we are thinking about potentially moving to a 5 star scale.
That said, if click on the link in the footnotes to see the original data, you can get an idea of what we're working with.
And lastly, the fontawesome thingy isn't plotly. It was built using http://blockbuilder.org/ by @enjalot
> the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary.
This article's headline reveals the spin the HN community is trying to put on it. Many people in SF like to attack the very idea of meritocracy, because if you admit that meritocracy is a good idea, you're implicitly endorsing the idea that people have different levels of merit, and thus that between-group differences in representation might be due to something other than discrimination.
The real issue I have is that only a few of the other comments in this HN thread are looking at the data analysis presented in the article and are just taking the headline as gospel.
I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?)
The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place)
That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.