You end up having no way to evaluate candidates. You have no way to distinguish the following -
1. Those who had seen the question before, decided not to tell you they had, and faked figuring it out.
2. Those who had never seen the question before, but were able to figure it out.
You can, of course, distinguish them from the following, but you can't distinguish the following from each other.
3. Those who have never seen the problem before, could figure it out, but do poorly in the pressures of an interview.
4. Those who have never seen the problem before, and could never figure it out.
And it is -possible- you get the honest candidate -
5. They've seen the problem before, they -tell- you they've seen the problem before, and go on to solve it easily.
#1 looks the best, and you've learned nothing about them (and they have nothing in their favor other than having seen the problem before). #2 looks okay, but pales in comparison to #1, despite having actually demonstrated ability. #3 looks poor, but you have actually learned nothing about them, and they may in fact be absolutely amazing, except not good with interview pressures. #4 looks poor, and is indistinguishable from #3 (unless they do so badly that it's clear they have no idea what they're doing, rather than just being off in the weeds somewhere). #5 you now know is honest, and that they've boned up for your trivia challenge, but nothing else.
You haven't measured anything you set out to measure. You wanted coding ability and/or ability to reason out a problem; all you got was whether off the top of their head they were able to solve this particular problem. If that was what the job entailed, parroting back answers from Cracking the Coding Interview and the like, then you'd have a good test, but that's probably not what you need in a developer.
1. Those who had seen the question before, decided not to tell you they had, and faked figuring it out.
2. Those who had never seen the question before, but were able to figure it out.
You can, of course, distinguish them from the following, but you can't distinguish the following from each other.
3. Those who have never seen the problem before, could figure it out, but do poorly in the pressures of an interview.
4. Those who have never seen the problem before, and could never figure it out.
And it is -possible- you get the honest candidate -
5. They've seen the problem before, they -tell- you they've seen the problem before, and go on to solve it easily.
#1 looks the best, and you've learned nothing about them (and they have nothing in their favor other than having seen the problem before). #2 looks okay, but pales in comparison to #1, despite having actually demonstrated ability. #3 looks poor, but you have actually learned nothing about them, and they may in fact be absolutely amazing, except not good with interview pressures. #4 looks poor, and is indistinguishable from #3 (unless they do so badly that it's clear they have no idea what they're doing, rather than just being off in the weeds somewhere). #5 you now know is honest, and that they've boned up for your trivia challenge, but nothing else.
You haven't measured anything you set out to measure. You wanted coding ability and/or ability to reason out a problem; all you got was whether off the top of their head they were able to solve this particular problem. If that was what the job entailed, parroting back answers from Cracking the Coding Interview and the like, then you'd have a good test, but that's probably not what you need in a developer.