The idea is that the candidate knows they can do the best they can within the time limit, and then it's done. No agonizing over whether it's good enough or they should spend another few hours. No imagining how many hours and days other candidates for the same role might be putting in, wondering if they're going to stack up.
I've used this approach with around 100 candidates hiring for various roles in my own business and for clients. The response I've gotten from candidates has been overwhelmingly positive.
The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field. The responses to a challenge exist in relative terms, not absolute. It doesn't matter whether your response was great or not. It matters whether your response was in the top 5 of all the other candidates responding. It follows that you probably need to spend at least as much time, if not more, than all the other applicants for the role. How much time the other candidates _might_ be spending is basically unbounded, leading candidates to spend vastly too much time working on a challenge and then lying about it.
If the candidate screws up a timed challenge that's also totally okay. It'll be a great conversation to have in the follow-up interview. "I started down this road, but if I was doing it again I'd do X, Y and Z because of A, B and C."
Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.
How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?
You're actively removing the possibility of finding out who would think "meh thats good enough I guess" is the effort needed for production code and people who would go the extra step of spending time analysing and fully solving the problem.
2 people with equally average solutions...would one have done more given the chance? Both? Neither? How can we find out? You've added uncertainty to the process rather than solved any problem.
Two weeks is decidedly not 4 hours, no matter how you try to relate them. I could write the Falcon Heavy takeoff sequence in two weeks. (joking, of course, but the compression factor of 4 hours over two weeks is overwhelmingly and obviously higher)
> How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?
In my job, pretty much every week. Many of my tasks are "implement feature X to a passable amount in 2 days", and then it goes to someone to use and get feedback on immediately. My team are mostly the same.
This sounds to me like the nature of every job. Secondly, the interview process has never, to my knowledge, reflected on the job performance, and I'd suggest it shouldn't.
Many times, markets and business dictate time pressure. So yes, this is very real in startups at least. Like for instance, an inevitable market development might want you to ship code by the next week, however improbable or impossible. I know I'll get a lot of flak for saying this, but that's ok. I've experienced it first hand. In an early stage startup, when faced with the choice of losing revenue/delivering on time, you'll choose the latter. That means your shipping slot is fixed and so are your features. You'll just have to slog it out or work real smart. Even if you cut features, you'll still realize there's a lot to be accomplished in a short span of time. This is mostly a one off thing. If such events repeat endlessly, that is classic bad management. So you'll need to deliver on time, sometimes :).
So, I guess that companies which administer such time based tests foresee at least one such situation for that role. I'm just speculating.
That's more indicative of poor management and planning on the company's side than reflecting on the performance of the candidate if the time pressure is a regular occurrence.
Few people perform well under pressure. Do you want to test what is their mediocre performance under time limits like on a school test or what they are really capable of?
With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.
No real company has perfect management and a lot of them have poor planning.
> Few people perform well under pressure.
Yes, but majority of jobs have some level of time pressure. That being said, there is such a thing as reasonable deadline.
> With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.
Inability to perform under pressure does not imply high skill without pressure.
There's never enough time or resources to do everything you want to do. Believe me; I've gone from one extreme to the other in company and team size. For any but the most mechanical of development tasks (e.g. CRUD apps), it's very common for a developer to make an estimate, realize that it was too optimistic, and then have to reduce the scope. So it's valuable for a developer to spend some time up front thinking about the design, break it down into tasks, prioritize them, and then implement the highest-priority part first. Then, even if they have to cut tasks, they can deliver something of value without having to do shoddy work.
I do think an interviewer should be completely up-front about what they want out of a candidate if they give a coding challenge that's too ambitious to implement perfectly within the time limit.
You have your reasons, but I'll walk from a time limited take-home, just as I'll walk from a screen captured one.
Both methods are ways to tilt the control and convenience back in the favour of the company - defeating the whole purpose of doing the work at home.
With all due respect, but attempts to optimize the interview process invariably end up doing this, due to the priorities of the paying parties (ie - the companies) - and always at the cost of the interviewers. No thanks.
I worry you might be projecting your experience onto the experiences of others. It sounds like you’ve been stressed out about having an unlimited time slot in the past.
But lots of people (myself included) get debilitatingly stressed out when an arbitrary clock is put in front of your face. Honestly, my recommendation to any company that gives timed challenges is to... not.
What if you gave the option to not time challenges? And maybe didn’t make it seem like having untimed challenges are inherently more stressful all the time.
I mean, it's strictly better than the typical code interview, which is timed too. And the reality today is that the majority of people have to be able to deal with a ~45min code interview to find employment in software.
I recently interviewed for my next role. Not a single company was doing the in-person timed code interviews. Where there were code tests, they were all take home and had no time limit beyond "when do you think you can get this back to us by?". If you need a time limit, ask the candidate for their estimate. It has the added benefit of telling you whether the candidate can assess their own schedule in relation to your code test, while not screwing the candidate over.
>Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.
While I agree with what you said about leveling the playing field, I find 40 minutes sound awfully short. In university I noticed I was doing much better on 3h exams than on 1.5 hour exams, often only taking 1.5h for the 3h exam, even though it had more material. I couldn't relax enough in the short exam to do my best, even though the time was long enough for a twice as long exam.
> In university I noticed I was doing much better on 3h exams than on 1.5 hour exams
Same here. I was the person who would do poorly on the 15 min physics quizzes we had in college just to get the highest grade in the class on the final, only because I had more time (half the answers on my quizzes would be blank just because I didn’t get to them). In high school, I would barely squeak by on the multiple choice AMC to then do fantastic on the 3 hour AIME.
But I clearly knew the material, and I knew it very well, so are you going to penalize me for being slightly slower on a test? As an employee, I complete my real world tasks just as fast as everyone else, so why should a time constraint on a high pressure exam relate to the real world productivity and profitability of an employee? I would argue that it doesn’t—at all.
> When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max.
No. Forty minutes is far too little time. Many people take 10-15 minutes to even get settled into coding. I can't imagine a meaningful, applied, realistic exercise that would take 40 minutes, unless you're wanting basic data structures like a linked list, or a basic sorting algorithm, or something.
You're missing a lot of good people that way, but I guess that's just collateral damage in the quest for the perfect hire, right?
Except your path is the one that has lots of long, burdensome take home tests. Some people have time commitments and/or other jobs to apply to but I guess those people are just collateral damage in your quest for the perfect hire, right?
I am guessing fizzbuzz or something similar. I was applied for a job and got asked to do an n queens programming task. It was for a commodity trading firm. I told them no thanks. When I set code exercises I craft them myself and make them appropriate to the job on offer.
All this does is compound the stress. Instead of "agonizing" over whether it's good enough, now a candidate gets to agonize over 1) is it good enough, 2) why didn't I just push the updated code 1 minute earlier before the cut off time! and 3) sh^t I should have put more comments in to explain where I was going with that!
>The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field.
No, it's not. And that's actually fine. I don't care if a candidate took five minutes, or five hours, or five days! In fact, the response time can tell you something about a candidate. Don't take that away from me as a hiring manager. I've actually never asked or been asked how much time I spent on a challenge. It's not an issue to solve!
>If the candidate screws up a timed challenge that's also totally okay.
What follow up interview? They fu^ked up the challenge. You really think a hiring drone is going to proceed that candidate when they have 10 others who did ok? That goes completely against what this sort of canned test is all about; getting past a hurdle. If hiring managers were going to interview all candidates who took the test and have a nice chat about their code, the test is completely redundant. They'll just ask for a code sample to talk about. Saves everyone the time and the money!
All in all, 40 minutes is way too short and artificial cut offs are a nasty way to try and "level the playing field". In fact, you're not leveling it at all! Just making sure candidates who may be taking a bit longer to get into the swing of things that day are discounted. Heck, if the candidate has simple network issues they're screwed!
I like the idea a lot, but I think it is up to the team and hiring manager and not the tool the decide whether to use a time box or not. I haven't tried the tool yet so I'm only speculating here, but as a potential user from the team / hiring manager side I would expect I could disable the time boxing completely if I wanted.
Robust answer and well reasoned argument. I agree, and would go further to say that less pressure and more satisfaction has been had with time restricted homework in my experience as an interviewee.
But then again, a candidate that puts in way more effort is also a strong signal for future performance.
I suppose it depends on the employer, does he prefer candidates that put in more effort but might be less talented, or very talented but might not have much resolve (unproven at least).
If I'm given a time limit on an assignment, I'm going to take the time allotted and it will reflect in my (admittedly easily faked) git history.
If the employer is being honest, then we're all good. But if they're going to go with the candidate that puts in 5x the time allotted, then I clearly don't want to work for them either.
It's an obvious red flag IMO. If they ask for n hours in the interview but expect more than that, then they're likely to do the same thing when you're on the job. Don't be surprised if a few months down the line you're working 12 hour days 6 days a week or you're an underperformer and the first to get cut.
If you’re setting deadlines of forty minutes then I suggest you’re losing good candidates, who like to assess a problem and think through a solution first. It’s just creating pressure. Unless your code exercises are takes on these fizzbuzz exercises? I.e. trivial.
From my experience (and I have a lot) you can tell a good developer by their deliverable. Even better if you see the code developing through multiple commits. Some developers might take a long time, but if it’s bad it’s just bad. I would worry that if you only set 40 - 90 minutes exercises you aren’t giving enough time for the candidate to differentiate themselves at all.
The idea is that the candidate knows they can do the best they can within the time limit, and then it's done. No agonizing over whether it's good enough or they should spend another few hours. No imagining how many hours and days other candidates for the same role might be putting in, wondering if they're going to stack up.
I've used this approach with around 100 candidates hiring for various roles in my own business and for clients. The response I've gotten from candidates has been overwhelmingly positive.
The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field. The responses to a challenge exist in relative terms, not absolute. It doesn't matter whether your response was great or not. It matters whether your response was in the top 5 of all the other candidates responding. It follows that you probably need to spend at least as much time, if not more, than all the other applicants for the role. How much time the other candidates _might_ be spending is basically unbounded, leading candidates to spend vastly too much time working on a challenge and then lying about it.
If the candidate screws up a timed challenge that's also totally okay. It'll be a great conversation to have in the follow-up interview. "I started down this road, but if I was doing it again I'd do X, Y and Z because of A, B and C."
Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.