From being on both sides of whiteboard and take home interviews over the last few years, I’m all for whiteboard interviews, for a few reasons:
- take home interviews incentivize the candidate to spend more time than they’re told. You can choose a three hour problem, but candidates that really want the job will probably spend more time. Less experienced candidates will spend more time on it too.
- I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
- whiteboarding can go badly too. The caveat is that if it does, only so much of the candidate’s time is taken up; there’s a hard ceiling.
- from the perspective of conducting whiteboard interviews, the company loses out if the interviewer doesn’t do a good job. This is really on the company though - it’s worth investing in a good whiteboarding process. If there’s no investment here, it’s only the company that suffers (competent candidates will get hired by a company with a better process).
In terms of conducting a good whiteboard interview, my $0.02 is no trick questions, just solid problem solving with plenty of scope to show creativity and proficiency.
I've had the opportunity to do this fornat twice and I think it works well.
Here's how it works:
They ask you to pick a 3 hour time window to work on the project.
You pick the time.
They send you the project prompt in an email right at the beginning of your time interval.
You have 3 hours to read the prompt, design, implement, test a solution.
If you miss the deadline, project counts for nothing.
Both the times I did this the prompts were absolutely doable in 3 hours.
They were the kind thing you could do in about 2 hours if you had the ability to read the project prompt ahead of time and think about it for a few days.
Personally, I work well with time constraints. This format feels closer to a workday where mid-afternoon you might challenge yourself to finish a task before 5pm. Its a lot less pressure than having to juggle a conversation and having your code scrutinized in real time.
It forces a real time constraint that keeps you from piling time into something you just don't know.
Its hard to cheat, and it also forces the company to make the task achievable in the time frame. This format forces them to realize when the task is too big for anyone to finish in the time.
I hate time constraints. That is why I hate whiteboarding as well.
My brain just not work that way. If I have to modify something in a project I already worked on. That's fine. But interviews always ask tho build something from the ground up.
Once I got the requirements I need a few hours to think about it, before I write any actual code. I usually do something else while I do this. On a job, it's when I home and do some chores and my mind can wander around freely. But I can't make this process much faster. If I'm on the clock I have to sit down and start spitting out code. Also, there is no time for any major refactoring if you went in a wrong direction. This adds a lot of unnecessary stress.
I like it when you have a day or an afternoon to finish a 3 hours long task. I work faster if there's no time pressure.
The real world has time constraints, so I don't see this as unreasonable expectation in an interview.
> I need a few hours to think...before I write any actual code...I work faster if there's no time pressure.
It sounds like you work better but not faster. If something absolutely must be done in 3 hours, you usually will not have 2 hours to do the dishes while you let it marinate.
To be clear, I prefer to let a problem sit in my head for a bit, too. I think the end result is better in most cases, for most people. But the real world has problems with hard and fast time constraints, or huge financial incentives to get something working in ASAP and clean it up later if necessary.
But how many of those involve building a brand new thing you've never thought about from scratch? As the person you responded to said, if they're modifying something they've already worked on, that's different.
If something /brand new/ needs to be built in HOURS with huge financial incentives, that is a failure on so many levels of business that blaming the engineer for their inability to actually get it out the door in 3 hours is ridiculous
Yes, I completely fine with actual hard time pressure, when I know the codebase. When I'm on call and something goes wrong I can fix the code very fast and reliably.
When I presented with a completely new problem I have to think about what is the best tool for the job, how I want to structure my code, how can I test it, what are the edge cases, etc.
Yes. Though in practice I'd give people some grace period after the three hours: the three hour time limit is to help the candidate not commit endless hours to the project, it's not meant as a punishment.
I agree here - given the pressures of an interview are much higher than on the job, enforcing a time limit should only prevent extremes rather than add yet more pressure.
I think I’d be fine if someone gave me 3 hours to complete a typical take home exercise, but I worry that some exceptional engineers I know might not be. Applying excessive time pressure could lead to cultural homogeneity.
Yup I've done a couple interviews like that. They send the email at the time of your choosing and you have to submit the finished challenge at whatever deadline. Works for me! I can do things how I want to, just as if I had been given such a task on the job.
> They send you the project prompt in an email right at the beginning of your time interval
Whoops, your home internet has gone down. Or Gmail has decided it's not having any mail from their domain. Or your mail provider think they're spam. Or your mail client doesn't pick up the email for half an hour. There's a whole raft of problems coming from "email delivery is reliable enough to be a timer."
Take home interviews should be time-limited and proctored real-time. If it's more than 4-6 hours then it's too long. The problem should have multiple parts and just difficult enough to make it unlikely for most candidates to complete everything. The problem should unique enough and broad enough to allow the candidate to demonstrate good code structure, problem solving, creativity, and algorithmic ability. It should obviously not be something so straightforward and simple as to be a question on stack overflow.
The candidate should be encouraged to submit early and often for feedback, and to not be afraid to ask questions. Sometimes a subtle or complex element of the problem statement can help spur this interaction.
Finally the grading should be a blend of both objective criteria such as a series of heuristics and unit tests, as well as subjective measures such as code cleanliness and structure.
Multiple people should be involved in the grading, and a repository of previous submissions should be compared against for consistency.
I and my team used this technique for years to great success, hiring some of the world's top talent.
I assume they mean there are individually relatively self-contained pieces of functionality in the problem statement. Not completing an additional feature would still leave the rest of the project in a good state. Perhaps if you use a vcs you could even leave WIP code in an unmerged branch.
My question still stands. How do you evaluate someone's half-finished work? Do you prefer if I finish more features or my already finished code is better quality or better tested?
It's just an unnecessary source of stress during the test.
It would probably be best if someone was available to answer these kinds of questions for the interviewee when their time period starts. Though I think it could also be interesting to have this be part of the assessment -- how does the candidate trade off between scope / quality when they are given a fixed time period?
I'm not looking gor a dev that knows it all by heart. I'm looking for a dev who knows how to get information about stuff he doesn't know.
A take home interview therefore gives me a much better picture of the real-life skill level of a candidate compared to a very stressful situation like a real-time whiteboard interview.
Also: Give your candidate a real problem to solve and pay them for their time!
There's a famous quote that is attributed to Albert Einstein: “Never memorise something that you can look up.” It is said to be related to a time when a colleague asked him for his phone number, and he reached for his telephone directory to look it up.
At an interview, Albert Szent-Györgyi states basically the same: "You shouldn't fill your head with "szecska" (chaff?). You should keep knowledge in books where you can look it up and use your head only for thinking."
The whole interview was inspiring. It's about learning and exploration. Worth watching if you understand Hungarian:
https://youtu.be/rtqhA4TtBrI
> are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow
Strange reasoning. If you want to cut corners, you use stack overflow.
Unless you know by heart the optimal way on how to reverse a string in C#, you do a 3 second google search and do a 5 sec copy paste of the one liner, from the top answer on stack overflow.
When I was programming as a 14 year old and came across a problem, it sometimes took me a day to figure something out. There was no internet back then. Now the most obscure issue is probably nicely documented with a top 5 of solutions.
I agree with googling stuff but this gives me anxiety, even though I can create relational databases, use ORM/stored procedures/queries, write web apps, deploy to IIS/Docker. I still worry about looking for the next gig. I did write a complete mission critical solution for a mid-size business, used by 100+ employees but then I am asked about algorithms and I could answer 3 years ago but now I would get stuck on it.
Also, when I interviewed a year ago I was asked to find if something is a palindrome. This is programming 101 and it caught me off-guard. I got so used to all the methods available in C#, that going back to pen/paper and writing a simple for loop felt odd.
There are two good reasons to look things up on the internet:
1. Obviously, its faster. I, personally, get paid for results, not for how "clever" I am. (Which is good, 'cause I'm not very clever.)
2. Frequently, the answer on the internet is better/more complete/tested. Take implementing a Singleton class; its so simple that anyone could make one. Most home grown implementations would have race condition though.
> Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
One of the ideas that is always brought out about the awful interviewing process is that there are just a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down. I'm not sure how many this is and not just bad interviewees based on experiences, but I do also accept that they exist (My guess it's 10-20% of programmers).
At one project I was on we had a guy who was pretty bad (he was then hired away by a competitor to their super programmers team so whatever) but I know he spent days on something that should maybe have taken 1 day, and it looked pretty bad (I mean just looking at it)
I can say that no amount of time would have allowed him to write something that did not look messed up (based on his habit of using map to update multiple arrays at once and not using the returned array from map anywhere)
Maybe they've used extra hours to get a good result, but if they are able to recognize a good result they are not in that small group of programmers that must be kept out at all costs.
> a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down
I suppose it depends how you define "quality" - in my recent experience, it's always been the "rockstars" that are the worst developers. Because they generally write code which ends up being a nightmare to support and maintain when they're gone. Sure, it's great you wrote your own ORM in Perl but 10 years later it'll be crippling the company because no-one can make any changes without it breaking 27 other things.
(Ok, it's often Perl people...)
For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
Ok I've had one experience with a guy that I guess thought he was a rockstar, in my opinion he was my level (from looking at the code, although he was much better than me at DevOps), but he did have one awful failing, he didn't want to figure out why something was the way it was and just decided to rewrite everything.
Also he thought I had done something using ElasticSearch because I was an ElasticSearch guy, instead of having learned ElasticSearch because I thought it was what was needed (and no amount of explaining this simple concept could penetrate)
> For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
Agree. But it should be noted that the needs of a startup may not always make that easy. I'm personally responsible for transmogrifying an existing Perl ORM for a company, because it only supported a single database handle / class. Which is a pain when you're in a master / slave environment. This hack allowed the company to meet the performance challenges that came along each year by summer. It also allowed growing the company from ~35 people in 2004 to about 5000 people in 2012 (when I left).
It also froze the version of the ORM in question, until it got rewritten from scratch at great expense. But that expense could now be easily carried by the company.
10-20% seems really low in my experience. Do you mainly hire out of the industry?
Hiring heavily from colleges results in 60%+ of applicants that somehow have a B.Sc. from a university that produces amazing applicants with great (3.5+) GPA's but stall at either fizzbuzz or (just in case they memorized fizzbuzz) i-can't-believe-it's-not-fizzbuzz.
Furthermore, some candidates (20ish percent?) can't even answer basic, non-trivia questions about their language they self-report to know best. e.g. Java programmers who either don't know or cannot articulate what the static keyword does.
I'm convinced there's a path through every university, no matter how good, where you can avoid all the good professors and only pick babysitter professors and squeak through your four years with a piece of paper and a well-tuned sensor for which profs actually grade homework instead of just giving 100's so people don't complain.
My experience is only as a programmer, maybe HR has kept most of the bad ones out, but I doubt it based on my experiences with HR. So 10-20% is my experience of how many programmers I encounter are bad in a big organization, and bad is a relative thing of course.
But often when I am brought into a big organization there is one out of every 5 programmers that just should be something else. They're not necessarily stupid, just shouldn't program.
on edit: in my experience small organizations don't have quite as bad because for the organization to be able to exist their programmers must be able to program. (for any organization heavily focused on tech of course)
I would say that while only 10-20% of programmers might be literally not worth paying any amount for, it might be as much as half of the field that isn’t worth the current, major US market compensation.
> I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
If you just evaluate them based on their code you doing it wrong. You should organize an in-person code review. Where you ask: Why they do what? How would they improve their code? If something is not right, point out. Look at them how they react. Good candidates usually try to fix it. Bad team players usually defend their implementation until the end. It really easy to find out if they just copy-pasted some code without understanding it.
- take home interviews incentivize the candidate to spend more time than they’re told. You can choose a three hour problem, but candidates that really want the job will probably spend more time. Less experienced candidates will spend more time on it too.
- I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
- whiteboarding can go badly too. The caveat is that if it does, only so much of the candidate’s time is taken up; there’s a hard ceiling.
- from the perspective of conducting whiteboard interviews, the company loses out if the interviewer doesn’t do a good job. This is really on the company though - it’s worth investing in a good whiteboarding process. If there’s no investment here, it’s only the company that suffers (competent candidates will get hired by a company with a better process).
In terms of conducting a good whiteboard interview, my $0.02 is no trick questions, just solid problem solving with plenty of scope to show creativity and proficiency.