> All our technical hires, whether in engineering or product management, go through a work sample test of sorts, where they are asked to solve engineering problems during the interview.
I disagree that a whiteboard session can be considered a work sample test. Whats wrong with 'complete steps 1,2 & 3 and i'll be back in 30 minutes'?
For engineering we actually found giving a 2-3 hr 1 problem test has been a better predictor, and 3-4 30 minute face to face (usually 2 on skype prior to onsite Test, then 2 follow ups). Alone time gives the candidate time to digest and solve the problem on their own which is more 'real world' like and shows what can be done on their own. Some questions are intentionally vague to compensate for 'able to make decisions with little information'. We have no right answer but will ask why you made these decisions. This frees up our staff time tremendously and weeds out those that may need a lot of hand holding early on.
This is exactly correct. A work-sample test needs to capture the normal parameters of the work. If team members aren't routinely called on to solve programming problems in a high-stakes stick-the-landing-or-you're-fired exercise on a whiteboard in front of an audience, then the prediction the test is trying to make is confounded by all those factors.
* Eliminate live audiences
* Let people work in the environment they'll be able to choose on the job
* Ideally, let people work in their own comfortable environs, even if they won't be able to do that on the job
* If you're worried about cheating, build that assessment into your in-person followup interview
Originally he'd simply asked if I could be there whilst he did the test so that he could talk aloud and bounce ideas off of me as it was timed and he gets nervous in tests and was afraid he'd not think clearly.
He froze. Totally.
I took over and did the test for him, and he aced it and was offered the job at the top salary band.
He is a perfectly good engineer and the company were very satisfied with their hire, but he never did that test. The test in it's entirety was completed by myself.
I cannot imagine this is such a rare thing with remote technical tests.
I believe both of you, but our process did/does nothing to overtly catch cheaters, we hired directly off its conclusions, we hired at a rate faster than most VC-funded cash-flow-positive YC companies, including in SFBA, and we never let anyone go (nor did anyone ever quit while I was there) once we made an offer.
(Matasano is also not a company where it's easy to duck attention and coast; the tempo is 2-3 week engagements that wrap up with metrics that everyone cares deeply about).
The conclusion I draw is that cheating just isn't as big an issue as people think it is.
I've seen it several dozen times. A solution to one of our sample tests made it out to Github and that was all she wrote. For a while we kept the same sample test as a honeypot, but disqualifying half of the candidates for cheating was tiring.
Meh. IT unemployment has been about 2% for YEARS. I can get a good job on a team I'll enjoy working with in less than a week. If you're a top tier IT name then it might be worth jumping through hoops, but I will wind up at companies with the least hassle in the hiring department. 15 years in the industry has told me that there's no strong correlation between hiring practices and team quality.
Precisely why I hit the back button if my employment application is through something like Taleo. My time is valuable, too valuable to waste on a resume rabbit hole.
actually, that brings up a really great point. This could be an interesting article. Considering how much the web focuses on 'conversion of web leads' and making the entire process seamless, you'd think you could quantify the candidates lost (and put a dollar figure on it) simply because the interface wasn't friendly and turned away candidates. However, all too often i feel that the goal for everyone involved but the hiring manager, is a seat filled.
Anecdotally, I was prompted 'Zip code required' on a multinational company talent portal. Tried to fix and submit for at least 60 seconds before finally had to modify the CSS to show the input so I could put in a value, as Zip input was set to display:none. I figured at least there would be less competition for the role.
If you deliberately made it that way and told your candidates, it might work well. If you're a web dev and can't figure out how to un-hide a field, then you don't deserve an interview.
Similarly, I send my resume out as a PDF. If they ask for it in Word format, I know it's going into that kind of system. I may or may not send it in Word format, but it's definitely a mark against them.
I have questions for both you and IndianAstronaut, then. I see statements like this frequently. I also see people talking about desiring to work at places like Intel, Qualcomm, Nvidia, and other such large tech companies that violate one or both of your rules. I assume you are not in that group? Would you really turn down one of those companies of a recruiter approached you and asked for a Word resume or told you to fill out a Taleo profile?
I kind of hinted at that in my original post. If it's a named company that I explicitly desire to work for then I might be willing to jump through a couple of hoops. Generally speaking, though, the name on the sign is not a strong indicator of whether or not I'm interested in working there. I'm usually looking for some combination of salary, technology stack or cool project, and strong/intelligent team. Having a recognizable company name on my resume is a very very low priority.
True, but theres a difference between paid in-house training and being able to investigate problems on your own without constantly bothering everyone on the team to do your thinking.
The latter is a candidate I try to avoid like the plague.
I'd agree with everything but "alone". While having an interviewer (or more than one) staring at you while you solve a problem is unnerving, I'd also expect to be able to elicit requirements during the exercise, either before, during or near completion - because that's how it has always happened on my past assignments.
Do you let your interviewees reach out (possibly via IM) to your interviewers with specific questions?
Yes, we all have skype on our machines and I show them how to dial my phone extension if they need anything.
Edit: also for clarification, we discuss the problem ahead of time to make sure they understand it and ask any questions. For my group it's always a basic full stack problem. Here's a mockup and a basic install of visual studio and MSSQL, show me you can CRUD some data and present a decent UI. then we'll talk about how you did it.
I disagree that a whiteboard session can be considered a work sample test. Whats wrong with 'complete steps 1,2 & 3 and i'll be back in 30 minutes'?
For engineering we actually found giving a 2-3 hr 1 problem test has been a better predictor, and 3-4 30 minute face to face (usually 2 on skype prior to onsite Test, then 2 follow ups). Alone time gives the candidate time to digest and solve the problem on their own which is more 'real world' like and shows what can be done on their own. Some questions are intentionally vague to compensate for 'able to make decisions with little information'. We have no right answer but will ask why you made these decisions. This frees up our staff time tremendously and weeds out those that may need a lot of hand holding early on.