I think its the opposite. I would much rather had only my and the candidate's 5 minutes wasted, versus a whole hour, when its very apparent in those first 5 minutes they don't have the technical skills we need for the job.
Presumably you consider how drastically your whole process has failed if you bring someone in for an on-site interview who demonstrates within 5 minutes that they lack all the technical skills required for the job.
Well mistakes happen - a phone screen can only get you so far, especially in a seller's market where your talent pool is pretty shallow. Though I think a shared google docs interview is probably the best way to go for the initial screen, I just haven't given or taken one like that yet so it must still be relatively rare.
We came to the conclusion that phone screens are pretty close to worthless. We'd still have been very upset if we ended up in a face-to-face interview with someone who had no business interviewing for the job. You need to do more than just phone screen (actually: you should eliminate phone screens).
(I don't know who's downvoting you but I upvoted.)
I'm really fascinated by your replies on this thread; seems you build a really solid process there.
I'm curious about the phone screens - how do you screen out the candidates that are grossly unqualified so that they don't progress to the interview stage?
By my last year at Matasano we were literally discarding the results of phone screens. We did them largely as a pro-forma exercise, and because they made team members more comfortable with the process as a whole.
We virtually never selected out candidates based on phone screens. We had a work sample process that kicked in after phone screens, and that cost us almost nothing to run. So we never had an incentive to prevent a candidate from going through that process.
Unlike phone screens, the work sample results were strongly predictive. You could bomb phone screens, ace work samples, and end up coming in for in-person interview.
(We did "nudge" candidates we felt wouldn't do well on the work sample tests, solely out of concern for not wasting their time, but anyone who wanted to proceed was able to).
Phone screens are a waste of time.
Spolsky did a really good job of documented the best practices within the framework of unstructured interviews as they were practiced 10 years ago. The problem isn't Spolsky's tactical suggestions; it's that the strategy they're a part of is being discredited.
I'm curious how often you had people voluntarily drop out of the process without completing the work sample, and whether that frequency would change if you didn't have a phone screen.
(I take it from your questions that you never actually stopped doing phone screens).
If my first contact with a company is them asking me to put in hours of my time, when it costs them "almost nothing to run" (your words) then I'd be inclined to pass.
The company I work for has a work sample exercise, and we intentionally place it after a phone screen and first interview, because we feel it is (and appears) more fair to the candidates.
From our point of view, we'd love to put it up front, as it is the best source of information we get, but if it caused us to lose good candidates before we even started (and we believe it would) that would be a show-stopper.
Most of our candidates drop out before the work sample. On the other hand, almost none of our candidates are qualified to work for us when they initially apply.
We make it really clear that there will be work samples before people even apply. I think the real question would be "how many qualified applicants don't bother applying because of the work sample," which is a question we can't answer. Given the paucity of unemployed qualified infosec folks, we're comfortable with the tradeoff.
I’m curious exactly how many times a candidate gets to apply to Matasano.
Thomas describes the Matasano process as costing “almost nothing,” but that includes running a couple exploit training websites and sending an 852-page book to applicants. Which I’m not inclined to go through immediately, because I need money right now.
Do I have to do the microcorruption.com and the cryptopals.com and The Web Application Hackers Handbook before even trying the technical screens and challenges? Or should I try seeing if my existing web application security best practices and rusty MIPS assembly experience are enough to get to where I have enough breathing room to do these exercises?
> I’m curious exactly how many times a candidate gets to apply to Matasano.
Undefined. Personally, I applied twice, ended up coming to work here the second time through.
> Do I have to do the microcorruption.com and the cryptopals.com and The Web Application Hackers Handbook before even trying the technical screens and challenges?
Nope. But they might help, and you may want to anyhow; they're fun :)
On the other hand, if you made something you could give to anyone who applied and was little to no work for you how many good candidates would you pick up that you wouldn't have given a phone screen to?
I think that varies quite a lot based on company profile and role-type.
We phone screen anyone who has a resume that's vaguely relevant. If you're applying for a software engineering role and have never done any development, then we'll reject straight away, but that almost never happens.
We simply don't have a high enough profile to receive a flood of applications that we need to filter before we phone screen.
And for a software engineering role, evaluating the work test takes as much as the phone screen (sometimes more). We're not simply checking whether you can produce the right output for FizzBuzz (checking that could be automated), we're looking at your choice of algorithm, design trade-offs, unit testing approach, etc - things that require a human to be involved.
Different companies and different roles will have different time investments.
A CTF style task can be evaluated more simply (at least on first pass) - did you capture the flag.
And if you are looking for a few amazing candidates form amongst a mass of poor ones, then you can afford to optimise your process so that you test every candidate at as low a cost as possible, even if that causes some candidates to self-select out.
I have heard much the same about dating, so I don't think it is necessarily about a process failing, as the way we judge people based on first impressions.
Curious how you could judge that a candidate doesn't have the technical skills you need for the job in 5 minutes? Can all the tasks your employees do on the job be completed in 5 minutes?
Sometimes you get applicants that literally don't know what a compiler is, don't know what a variable is. That sort of thing. That kind of person can probably be ruled out in the first 5 minutes for sure.
Of course I think everybody would agree that if such a person progressed to the interview stage, there was a failure somewhere.
