Isn't this exactly what you'd expect? e.g. if get a random cohort of developers who all of their peers would say are amazing, if I subject them to a series of tests, they will do differently well at the different tests?
For example, so I set a test where you have to write some Java, if half the candidates haven't written any Java, they'd all surely do worse on the test than the other half?
Or is their a belief in the industry that there's some scale on which we can absolutely rank all developers - front end, back end, full stack, mobile, desktop, embedded? That sounds like a surprising belief which would require extra-ordinary evidence?
For example, so I set a test where you have to write some Java, if half the candidates haven't written any Java, they'd all surely do worse on the test than the other half?
Or is their a belief in the industry that there's some scale on which we can absolutely rank all developers - front end, back end, full stack, mobile, desktop, embedded? That sounds like a surprising belief which would require extra-ordinary evidence?