Oh, wow, 1000x this article. Forget about the crying thing, which makes it vivid and more interesting to read but might obscure the fundamental point, which is that interviews make people incredibly uncomfortable. Some of the calmest, most unflappable people I work with shook visibly in their interviews with us.
What's important about that is: you are not getting an accurate read of a candidate's state of mind in a high-stakes interview. I think you have just a couple of options to improve your accuracy given that fact:
* You can take deliberate steps to lower the stakes in your interview process by changing the kinds of transactions that happen in the interview (for instance, by constructing interviews that involve collaborating on a problem instead of challenging the candidate).
* You can "discount" the signals you get from the interview, particular w/r/t already nebulous concepts like "confidence", "social skill", "enthusiasm", and the dreaded "culture fit".
* You can push more of the recruiting process into situations where the candidate controls their environment (for instance, by doing stuff at home, and on a fixed but flexible time scale) and less on in-person timeboxed interviews.
I'd love more ideas. We're trying all three, but it's a lather-rinse-repeat kind of situation. I'm getting good, I think, at spotting signals that are probably the result of jitters. And I think I've learned pretty conclusively that those jitters have zero correlation to on-the-job performance. But I have so much more to learn.
I like your ideas, particularly collaborating on problems and giving candidates the ability to control their environment while problem-solving (such as giving them a week-long assignment). We've had good luck with the latter, and it's provided several valuable employees who, frankly, would probably not have passed a "live" interview.
Another tactic we employ is to keep the technical interviews very conversational. Rather than asking a very difficult question and putting the candidate on the spot, we try to talk through problems and do an over-the-phone "white board" session. We like to choose problems with many answers; some of these answers are easy to arrive on, while other explanations may be trickier. The candidate gains confidence getting the first part of the question correct, and seems to be more at-ease while completing the more difficult tasks.
There's no way that we can completely cut out the stress of a technical interview--no matter what, it's nerve-wracking to put your technical capabilities on the line--but it doesn't need to be a belittling, terrifying experience like the ones we so often hear described.
My first few gigs during/after school, the interview was one of the hardest, most stressful parts - the actual work tended to pale in comparison. The interview rarely seemed to have any tangible connection to the actual work being done, either; it was essentially like some sort of bizarre hazing ritual where I proved that I had what it took to be an employed professional, or something.
At this point I've done thousands of phone screens and hundreds of in-person interviews over my career, so that stuff doesn't get to me like it used to. It's much easier to disengage and ignore all the high pressure tactics and shitty interview questions and bad interviewers and just write it off. Given that, I could probably easily end up doing the same thing to candidates when I'm interviewing them.
But I shouldn't! Because oh my god, those first few dozen interviews were emotionally and physically exhausting. Every single one left me a nervous wreck regardless of whether it went well or not. That shit simply isn't healthy; why would I ever inflict it on someone else?
'High pressure' tactics are probably the worst in this regard, out of all the things that go on in interviews. I've dealt with a few places that went with tactics like verbally abusing candidates during interviews, or making a subpar offer and giving the candidate 24 hours (or less!) to decide, in the hopes of getting them to panic and take it. One place had their recruiter call me up at 7am and verbally berate me for turning them down! Fucked up shit.
One thing I consistently hear people complain about is whiteboard coding. Nobody likes it, everyone finds it stressful. Note that the actual task of coding in an interview isn't something I hear everyone complain about - just doing it on a whiteboard, specifically, vs doing it as a take-home 'programming test' or in a shared etherpad or something. Maybe at some point in the future, eventually our industry can phase out the whiteboard.
The problem with interviews is that you have to make a decision and the candidate may not be themselves regardless of what you do. They may freeze up in a collab. Or collapse on a take home or be overly stiff doing a 2 month contract knowing the job is on the line.
I personally think technical questions are useful. Not hard ones, but softballs. It is never used to say someone is a hire, but is a requirement.
Who cares why the stakes are as high as they are? They're confounding the results of the interview process, which means they're the employer's problem, whether we like it or not, think it's fair or not, or whatever. The fact is that "we", as an industry, have a counterproductive habit of screening candidates for "ability to gracefully metabolize semihostile job interview experiences", which turns out to be not all that valuable an employee attribute down the road.
That's not an industry problem, it is a society problem where people need jobs to eat. Until you fix that problem, everything else is just dancing around the problem.
I interviewed a guy at a movie theatre 20+ years ago when I asked if he liked movies. He started sweating... And eventually said he had to cover medical expenses for his son and he was running out of time. He needed to work and would do what it took.
Until you fix the stakes, all you're doing is changing to another interview form that someone else will freeze up on.
I'm not hiring any programmers at the moment but if I were, I'd probably hire high-level people by sitting down with them to discuss some piece of open-source code they've written. I'd take a few hours before the interview -- again, assuming the position is important enough to justify it -- and familiarize myself with their publicly-visible work. At that point the 'interview' can just be a casual conversation over lunch -- a two-way one, at that.
Lower-level people with less experience? I don't really know how I'd evaluate these candidates. I might have to fall back on grades/employment history and the dreaded Fizzbuzz-on-the-Whiteboard routine.
But that's probably OK, because the higher-level people are the ones where you really need to get the selection process right the first time. One fresh-faced college grad is just like another, presumably, unless s/he has taken steps to stand out from the crowd by contributing code to the community. At which point I'll treat them like a higher-level interviewee, and we can skip the Fizzbuzz stuff.
> You can take deliberate steps to lower the stakes in your interview process by changing the kinds of transactions that happen in the interview (for instance, by constructing interviews that involve collaborating on a problem instead of challenging the candidate).
If you want to evaluate all candidates on a level playing field, this can be really hard to pull off.
I work in the defense sector. We have some pretty strict guidelines on how we're supposed to evaluate people. Because of those guidelines, we often try to give each batch of candidates the same general set of questions/challenges, etc. We do try to be collaborative during our interviews, but it's really tough to collaborate without leading when you've worked a particular problem 1,000 times.
Further, even when you do this well, the candidate often interprets your help as their failure. "Oh crap, he pointed me in a different direction, he must think I'm an idiot."
> You can push more of the recruiting process into situations where the candidate controls their environment (for instance, by doing stuff at home, and on a fixed but flexible time scale) and less on in-person timeboxed interviews.
I think this is really key to our success. Our interview focuses on two programming problems which require you to interpret imperfect requirements, ask questions if you deem it necessary, provide an estimate of when you think you'll complete your work, code/document/test two full applications (one per problem). We ask the candidate to do this well before the interview, on their own time. Typically we don't discount for how long it takes a candidate to complete their work. I've had awful submissions which took someone 40 hours, and excellent submissions which took someone 4 hours, and vice versa.
The problem is real. For many jobs that don't require working with strangers, it is unrealistic to the work required to be able to perform under stress in front of a stranger. I think this may be an introvert/extrovert issue.
There are many solutions, but a couple that come to mind are also imperfect:
- Exams: They seem fair, but reflect the bias of the exam writer, have legal (discrimination) implications, and don't show what's relevant to the job.
- Take-home work: This is closer to real work experience, but can be fudged too. I know someone who landed two jobs based on take home work that he paid someone else to do. It's also someone who visibly cheated in school, so it's a pattern of behavior. Despite being the OP's favorite method, it's still imperfect.
The only things that I've seen work even decently (but still closer to 50% than 100%) are word of mouth recommendations from trusted advisors, and internships. But how to judge potential interns makes this sound like a recursive problem...
Totally agree. One more thing I'd add is: tell your candidates up front what to expect of the process. An overview after first contact is nice, but interview-day schedules specifically seem to help alleviate the jitters. We've started giving candidates a schedule with names and rough content before they arrive: 10a-11a - pair programming on a TI Launchpad with Jerry; 11a-12p - planning and organization exercise with Sam; 12p - group lunch; etc.
I'd also humbly request that employers hurry up and tell a candidate that they're no longer in consideration for a position as soon as that's decided. It doesn't have to be a involved production — a minimally polite two sentence e-mail is all that I ask.
As a job-seeker, getting strung along or getting the silent treatment after interviews is the worst thing in the world, especially if the candidate really wanted the job and/or has other opportunities on the table.
One thing that helps is to think of yourself as the interviewee while you interview the candidate. While you would have a better picture of the open position, still imagine the candidate asking you questions backwards within your mind. This helps in making sure you are not testing the candidate just on your own experience.
Imagine someone walking up to Maya Angelou, handing her a white eraser marker, and asking her to write you a poem on a white board. She's obviously not going to immediately write you a poem, at least not one thats any good. Is it fair to judge her poem writing abilities by this off the cuff creation?Creators can only do their best creation when they are in their comfortable space. Asking a programmer to write code over the phone will result in substandard quality no matter what.
Sorry, if I ask you to write a function that determines whether two given words are anagrams of each other, "I need to get into my comfortable space for my creative process to work" is an evasion. No one's asking for a masterpiece, just a demonstration of basic competence.
What kills me with Do Work While We Stare At You interviews is my process is all over the place. I tend to draft something, try it, find my five mistakes, try more things, etc.
That doesn't fly in an interview. At least, I don't feel it does. Interviewers expect mostly right and mostly very fast. I end up standing there trying to pre-think the entire problem so something correct comes out the first time, which, to them, is just dead time with me staring at the wall.
The other option is to "dive in" and backtrack a lot, but the people watching can't tell the difference between actual incompetence and just figuring it out.
I've been interviewing a lot lately, and I'm totally fine with people trying something, changing their mind, fixing mistakes, etc. When they get stuck, I help them out. All decent interviewers know that being interviewed is stressful and that you need to cut candidates some slack.
Speaking personally, I don't care much about speed. I care about things like how much fluency do you have with your programming language (is writing a for loop something you have to struggle with?), do you know how to use a hash table (you'd be amazed at how many people don't), can you explain what the tradeoffs are when you make design choices and compare them to the alternatives, etc.
Also, since I can, I'm going to complain about my pet peeve: candidates who want to write a solution in C++ or C or java (I let them choose any language they want) but who refuse to use a container library. They'll insist on using arrays, even when the problem pretty clearly requires something like std::vector. And then they'll tie themselves in knots to make static fixed size arrays work. Now, if you can justify why you don't want to use a @#$! container library, that's fine, but usually the justifications are garbage ("it will be too slow...no, I've never measured that"). What really gets me is that a good sw engineer should be able to see that rewriting stl::vector by hand in every application intertwining its code with your application code (vector's client) is a maintenance and performance nightmare.
And here's the rub: Running an interview this way is hard to do RIGHT. The questions have to be tractable for the given time, not too oblivious that the candidate blows through them, not too domain-specific that the interviewer ends up giving a lecture to education the interviewee on the domain. Then there's the proctoring of the question.
Show of hands here: Who here has worked a companies where they used white boarding as part of their process? Did any of the following happen?
* Interviewers asked pet questions instead of using a vetted list of questions
* Interviewers never tried any of the questions in interview conditions
* Interviewers were never trained/practiced giving these sorts of interviews
* Random developers were pulled into interviews to proctor a white boarding.
Guess what? The development team is learning how to interview candidates by interviewing candidates! Not prepping a team on how to interview seems very common. Maybe candidates are expendable and ultimately are the ones that need to prove themselves.
I have to wonder how many times qualified candidates get bypassed because they didn't whiteboard just so to the interview's liking. More concerning, how many times have unqualified candidates buffalo inexperienced interviewers with hand-waves or by luck of having seen the problem before, and end up weighing down the team with incompetence.
> What kills me with Do Work While We Stare At You interviews is my process is all over the place. I tend to draft something, try it, find my five mistakes, try more things, etc.
Well yeah, that's exactly how those interviews generally go. You have to kind of narrate what you're doing, and that can be awkward at first but coming up with the optimal answer in the end is not the only goal of the interview. By all means show your work.
My process is similarly all over the place, but DWWWSAY suffers from the additional problem that 80% of my thinking power is taken up with social crap in a situation like this, so the interviewer is essentially seeing me at 20% speed. This may be why I have never gotten a job after being asked to demonstrate coding something on a whiteboard. :)
Essentially, I just checked to see if the characters in the first string exists in other string and exits out of the code. It probably won't cover every use case (in other words, probably very buggy), but it is a start.
However, I have the luxury of relaxing while looking up the functions online, writing the code by my lonesome and then verifying it by running it directly. Last I checked, not any of these things are likely not present or available in a white board interview.
There's a much easier way to solve this problem than what you've done. And this is sorta/kinda the point of the article, where you will do well at a technical interview if you know some tricks in programming, whether it be through academic study or just on-the-job stuff.
To find if two words are anagrams of each other, do this:
sort(word1) == sort(word2)
done.
The trick is knowing that an anagram has a signature, and that signature can be found if you sort a word's characters alphabetically. Then you do the same to the other word and compare both of their signatures together. Voilá.
"genuine class" == "alec guinness"
because
"aceegilnnssu" == "aceegilnnssu"
In Ruby you'd write your own alphabetic sorting function first, then apply it to each word.
def sortAZ (word)
word.downcase.unpack("c*").sort.pack("c*")
end
def is_anagram (word1, word2)
sortAZ(word1) == sortAZ(word2)
end
puts "is_anagram(ARGV[1], ARGV[2])"
I recall seeing this trick in a Ruby book (probably PickAxe or something) and thought it was cool because I had never thought about finding an anagram in this way. Regardless, it stuck with me so, if this was asked in an interview, I'd be able to answer the question easily and look like a rockstar...when really it's just knowledge that stuck with me.
By the way, the alphabetic sorting here is done by unpacking each character into its number representation and then sorting the resulting numbers.
One of the comments mentions: "Just had a phone interview with Amazon and this was the exact question they asked. Interesting problem if you have more than 10 minutes to solve it. – Javamann Jun 14 '12 at 20:46"
So....do you store this knowledge in your back pocket in case you need to check anagrams at work (which will be never), or do you toss it aside knowing you can hit StackOverflow if/when you need it?
It seems like the only time this stuff is "used" is on technical interviews.
I readily concede you can't substitute StackOverflow for general competence, but an algo for checking for anagrams doesn't seem like general competence.
If the interviewer expects a correct answer off the bat, or rates candidates based on the time they got the solution, then the interview process is bad.
But the question is legitimate. If you're a programmer then you should be able to come up at least with a naive (but correct) solution. You don't need any tricks for that. Then you estimate the efficiency of your naive solution and see if you can improve it.
That said, there's still the problem that OP and several comments here mention, that the interviewer might get nervous, freeze, etc, even for problems that they would normally have no problem solving.
Another easy solution is to just make a histogram of characters in each word and to compare the histograms.
Anyway, if anagram questions were all that were asked in technical interviews, I really doubt we'd have so much critique and analysis here. I've never been asked such a simple question in any technical interview. The closest would be the first phone screen I had for a Google internship, which had (arguably) simpler questions (like raise one integer to an integer power efficiently) but there were like three or four of them that you had to get right.
The anagram question is a phone screen question, yeah. I believe the histogram solution is better performing than sorting, because storing the histograms as hash tables is O(n) on both words while the sorting solution is O(nlogn).
I think the general approach in an interview would be to ask the person being interviews "can you explain what time function that runs in".. "can you improve it", and see if they work out the "sort" trick by themselves.
That's how it was done to me anyway (although I talked it through before I started writing, and by the time I got actually writing anything down I'd worked out the trick).
Remembering the standard library isn't the competency being tested for, but rather the soundness of your solution. Your first solution has two problems from an interview-question standpoint: first, it's incorrect (it misses the case where two words use all the same letters but aren't anagrams, like "banana" and "nab"), and second, you're looping through each character of the second string for each character of the first string so it's a slow algorithm. I would probably try and get you to figure these out with probing questions, but the expectation is that you can figure it out and fix it.
What an interviewer will usually be looking for on a whiteboard interview is not knowledge of a library, but that the interviewee can reason that there's a better solution that works much faster than this solution.
You might want to try something like 'return set(x)==set(y)'. Or, a better solution might be 'return set(collections.Counter(x).items())==set(collections.Counter(y).items())'.
The point of the question isn't to see if you've memorized every single stdlib function. If you can describe a stdlib function but can't remember it, a good interviewer will tell you about it.
The OP's solution didn't deal with letter counts at all; my first expression was to show him a similar solution that was more succinct to illustrate the principle at work. My second solution deals with multiplicity properly.
Using Counters is actually pretty cool... it's basically a sparse histogram. Can't believe I've never used 'em before -- all I've ever used from collections is OrderedDicts.
Actually... I prefer the parent's solution using set(counter.items()). This way, you know that the equality tests are ordering-agnostic. It's not clear if that's the case for counters... even the docs are unclear AFAICT: http://docs.python.org/2/library/collections.html#collection...
Counters are effectively building a sparse histogram by going through each element in x, and then each element in y. Ignoring the test for equality, the Big-O for histograms is O(x+y). For sorting in general, the Big-O is O(x·log(x)+y·log(y)). In other words: the sorting solution is correct but less efficient.
>> No one's asking for a masterpiece, just a demonstration of basic competence.
Once the candidate demonstrates the basic competence, what would be a good next step? How should one evaluate the higher level of thinking like creativity, (higher-level) problem solving, etc., which may be just as important for the position at hand as the basic competence?
If you do have a good answer to the above, the next question could be if you could have tested this higher-level thinking without the basic coding competence if (and only if) so were the nature of the job.
These kind of FizzBuzz stuff is not an issue I believe (although, "going blank" during the phone call or in similar situation is as well legitimate).
The main problem is that interviewers (BigCo's) ask questions that require inventing Knuth-Morris-Pratt algorithm or K-D trees on the spot, whereas researches spent months... Of course I assume that the interviewee is not aware of these a priori (or it's been a decade since he last heard about these).
I think I'd get yelled at in an interview for this one since I didn't punch down to String manipulation (which is probably what they're really looking for).
>> if I ask you to write a function that determines whether two given words are anagrams of each other
I've no college degree, I've been writing software earning me a comfortable living for 2 decades and for fun for 3 decades. I've solved a lot of problems in software and never heard of Big O until recently on HN. If you ask me about college quiz stuff like fuzzbuster (fizzbuster?) or anagrams I have to Google what an anagram is in the first place. I've never seen anyone paid to write them, never had a manager ask me to write one.
Anagrams aren't some sort of classic CS problem domain, it's just a relatively simple example. One would start the question by making sure you knew what an anagram was. Two words are anagrams you can rearrange the letters in one of them to make the other, like "Mary" and "army".
Phil, the point is who gives a flying f in production about anagrams. Noone has to deliver a meaningful abstraction in < an hour (or 15 minutes as a "simple" example of "puzzle" testing), and interviewing for that is truly ridiculous.
You and others might find it meaningful but after two decades of development I really have to question those who do.
Like you - I've never needed to solve an anagram in production.
I think many are missing the point. In an interview situation, the anagram problem gets the person to write some basic, naive code (ie, proves that can track loops) which I think we would both agree is reasonable.
Then the interviewer gets you to look at how to improve it. Hopefully you'll notice that iterating over both strings isn't very efficient, and reason about ways to improve that.
It's that reasoning process that is important, not the knowledge of how to write an anagram solver.
To be fair, though - most programmers should be able to solve it with a couple of hints.
If anything, not being a real problem is a feature. Very few candidates will have ever actually had to solve anagram problems, so most candidates will be equally unfamiliar with the question, making it a fair measure of the candidate's ability to solve simple problems they haven't seen before.
Here's an example of an interview question that doesn't have this feature: "How would you serialize a tree?" The Lisp programmers will get that one right away. Even people with a passing familiarity with XML will get it right away. You don't know if they would have been able to figure it out themselves, all you know is that they've seen serialized trees before.
Tree serialization seems reasonable - basically you end up looking at tree walking algorithms and some recursion, right? (I can't remember the costs & benefits of depth first vs breadth first traversal, but I'm sure I'd work something out).
OTOH, I was once asked to write a Sudoku solver in an interview. I'd never really solved a Sudoku myself at the time (although at least I knew how it worked), so I had to try and work out an approach to solving the puzzle as well as how to write the code. I found that pretty tough.
The purpose of a question like this, and of the classic FizzBuzz question, is to be a lightweight test of whether a candidate can actually write code or not. That's all. It provides a negative signal if they fall completely to get anywhere close, and nothing more.
There are alternatives, you could ask them to explain a piece of code they wrote on their own before the interview, any piece of code would do. I suppose that could be gamed which is why most companies opt for FizzBuzz.
Personally, once I started thinking about these kinds of questions like this, I had less issue with them. Interviewing people who failed to write FizzBuzz opened my eyes too.
Since a reply link isn't available for spacemanaki's comment, I'll stick it here:
"The purpose of a question like this, and of the classic FizzBuzz question, is to be a lightweight test of whether a candidate can actually write code or not. That's all. It provides a negative signal if they fall completely to get anywhere close, and nothing more."
I disagree. Vehemently. Provide me an email address and I'll mail you tons of code I've written. You cannot, cannot conclude with any accuracy that the inability to code up a method/function to check an anagram means that the person cannot write code. What you can conclude is that they cannot write up a method/function to check an anagram.
You're making an assumption. If a = b and b = c, then a = c. That works in math class but not here. The test is not one of syntax or knowledge of the language but of knowledge of a specific problem. Some here have used the word "trick". I don't even need to go there. Once you have the answer to this anagram exercise, you can back into why it's a good question, what it supposedly shows, etc. At that point you're justifying the question and defending it.
But you missed the entire point of the exercise: you're supposedly screening candidates for knowledge of a language and how they might have used it in their past experience, but letting it all come down to whether they solve your specific problem on the spot with all of Pamela's concerns about stress, etc.
It's easier to ask about an anagram than to ask about a candidate's past and what they did on their projects. If not an anagram, there are others. There will always be "solve this now" rather than "tell me about..." That's just the game.
> You cannot, cannot conclude with any accuracy that the inability to code up a method/function to check an anagram means that the person cannot write code. What you can conclude is that they cannot write up a method/function to check an anagram.
If you can't check an anagram given the definition of an anagram and the programming language of your choice, what problems can you solve? It's not a hard question and if you can't come up with a correct solution at all, I dare say you lack the ability to program.
So, I understand and am incredibly sympathetic to the stress problem some folks have. I also occasionally freeze up when put on the spot, though I fortunately tend to recover after a minute or so of verbal stumbling.
That said, hiring quality engineers is HARD, and often ends up coming down to a numbers game. If I've got 100 applicants to go through and know (from past experience) that over 80% of them couldn't code their way out of a wet paper bag, then I need short, deterministic interview techniques that will identify those candidates who are actually worth my time. Thus: fizzbuzz. I've met lots of folks who talk a great game, have all kinds of experience listed on their resume, and yet somehow can't write up a linked list or fizzbuzz solution in any language.
So, I accept that there will be some false negatives in order to more quickly narrow the focus down to those few rare folks that can actually code. It sucks: it isn't entirely fair to folks who struggle with stress/nerves, but I need a 5-15 minute litmus test that can be done remotely and I haven't found anything else more effective.
Short of wasting everyone's time with take-home projects (which I often employ in later stages of the interview process), what other options are out there?
That's the $64,000 question. If I knew the answer, that would be my startup. It's a tough problem. It almost makes one beg for some sort of one-off certification (bar exam?) so we can put that to rest for ever after. I'll go out on a limb and say that most lawyers, especially the senior ones, probably couldn't pass the bar exam again. Developers have a mini bar exam every time they interview (hence the studying).
That said, there's one thing you wrote I took interest in: "I've met lots of folks who talk a great game, have all kinds of experience listed on their resume, and yet somehow can't write up a linked list..."
Umm...that would be me. I've never written a linked list. Guilty as charged. Java has a LinkedList class and I've never bothered to look at the implementation nor have I ever been curious enough to (re)write one from scratch.
I wish you would have said "...and yet somehow don't know when to use a linked list." That moves it from an academic exercise mostly applicable to CS students who might have written one 6 months ago, to a meaningful real world scenario and more than legitimate. Choosing the appropriate data structure, knowing when and why, is both realistic and important. I don't know how a candidate can bullshit their way thru a discussion of data structures where they can articulate, accurately, which one to use, when, and why.
> I've never written a linked list. Guilty as charged. Java has a LinkedList class and I've never bothered to look at the implementation nor have I ever been curious enough to (re)write one from scratch.
Good! You're right: the JDK's implementation is perfectly acceptable, and you'd be crazy to write your own. I certainly haven't, either.
But if you have any CS education, I'll bet you know what a linked list is at a high level. And I can probably verbally walk you through turning that into a "spec" that's good enough for a basic whiteboard implementation of one that supports a couple of basic operations. It's (hopefully) just another fizzbuzz: a simple problem to demonstrate that you can code something/anything. The overwhelming majority of applicants can't.
I'll ask you a slightly different question then: when is it appropriate to use a linked list? In which of those cases might it still be inappropriate and why?
These things can be memorized of course but I find that if one has a reasonable understanding of the underlying machine and knows how to implement a range of data structures and algorithms the answer to these questions (for most of the less specialized data structures) becomes quite straightforward. Whether that is relevant very much depends on the job being interviewed for however.
I really have to question those who find the question so difficult they would rather rationalize why they shouldn't have to answer it or handwave about how they can't possibly solve even simple programming problems in anywhere less than an hour.
HN seems to lack reply links in some messages so I need to answer you here.
"If you can't check an anagram given the definition of an anagram and the programming language of your choice, what problems can you solve? It's not a hard question and if you can't come up with a correct solution at all, I dare say you lack the ability to program."
I'm not trying to be obnoxious, but your answer is not optimal. Which is not a big deal, and it would give us a chance to talk about ways of optimizing code. Sooner or later we'd figure out if you knew what a hash table is and how to use it (no real points off if you never really got, under the pressure of an interview, the hash table solution to this problem), how you would go about profiling code, how you might solve this if they weren't strings but social security numbers, and so on. It's a launching off point to more interesting discussions. At least that's how I'd use it in an interview. Before long we'd be talking about a, I dunno, search engine or something.
Lol!! You're killing me. I need to work tomorrow and you're gonna have me up all night thinking how to optimize this thing. It's only 5 lines of code.
Hashtable, huh? I don't like them because they're synchronized, but let's use a HashMap instead. Regardless, I'm stumped.
Let me think out loud. You'd probably want me to stick two entries in there with the words being the values. I suspect the keys are irrelevant. But I need the values themselves sorted prior to being placed in the map. Lets assume I stick the char arrays in the map. I still can't see how the map helps. Unless, I abandon the HashMap and go with a TreeMap instead. That negates the need to explicitly sort and maybe that's the time savings, but....
My head hurts. :-)
I don't know. I'd have to code this out and experiment a bit. But I like the push. You're forcing me to dig deeper.
PS. I can't build a search engine.
EDIT: A quick search of the net yields at least one answer (don't know if this works):
"We can solve this problem in O(n) time by using hashtable. That is, create a hashtable which record the times that the characters a-Z appear in S1 and create another hashtable for S2. If the two hashtables have the same content, then the strings are identical. This solution requires O(n) time to create two hashtables and compare them, and O(52) = O(1) space to store the hashtable."
I'm so disgusted. I was way off. Not even close. I wasn't even thinking in this direction. Does anyone have the number to that truck driving school?
I'm pretty sure the sort solution would be an order of magnitude faster on any reasonably sized string. Big O's are Big O's : they are irrelevant when n is small, and for anagrams we're talking something like n < 45.
Reply links take longer to generate on HN as comment threads get deeper in order to deter people from losing their temper and flaming each other. In your case I would say that didn't quite work. My comment was not meant personally but as a general statement. If you had kept your composure and waited for the reply link to show up instead of flying off the handle about it and replying to an unrelated comment, perhaps we would be having a more productive conversation right now.
phil already answered you, but to expand a bit, programmers are asked to do anagrams all the time. Except it is not 'does this word have the same letters as this other word, just perhaps in different order', it is "is this list of address the same as this other list?" or what have you. In other words, do two differently ordered sets of items contain the same items? Asking the problem as an anagram just reduces it down to its essence. And, like it or not, there is an important efficiency issue here. The decent solution will be a bit slow compared to the optimal solution, and sometimes that can be important. The naive solution will not work for anything but toy cases. But, you can set the bar for the follow on discussion according to the kind of work you do.
I'm not sure if I think it is a great interview question or not, but it gives the interviewer a chance to see if you can do something basic in your language, and then it opens the discussion to broader contexts where you don't have to code - efficiency concerns if these are string or emails, but 1MB documents, or dealing with Unicode, terminations of strings and so on, algorithm performance, and so on.
It's normal for people to be upset when they're stressed and frustrated and it's good for interviewers to be kind and help them through it. Freezing up shouldn't be a disqualifier by itself, it's just a barrier to finding out whether the candidate has the skills and knowledge you're looking for and one that the interviewer has a vested interest in helping to overcome.
On the other hand, stress and frustration are part of the experience of being a software developer. Lots of developers, myself included, would be better off learning to handle them productively.
As a self-taught developer these stories scare me. I don't know the ins and outs of programming fundamentals, but I know how to program and I am really efficient in what I do as well. I've got big ideas, I have drive and vision and I think technical interviews can have the reverse effect of attracting real people.
For what I lack in low-level understanding, I more than make up for it on a higher level. Ask me to build something as a whole, don't ask me to solve mathematics questions and other unrealistic junk questions.
The biggest takeaway from this post is definitely this:
In my experience, when we are programming on the job, we're given the problem and we have time to think about it. We have time to research possible solutions, we have time to try stuff out that we know will most likely fail, and we can wait until we have something decent before we show it to our colleagues
^THIS. One hundred times over. How is putting someone on the spot to build something and build it perfect a good test of their capabilities and skills? That's not even how it works in the real world.
>For what I lack in low-level understanding, I more than make up for it on a higher level. Ask me to build something as a whole, don't ask me to solve mathematics questions and other unrealistic junk questions.
Without the math, how do you know that your solutions are any good? How do you know that the algorithm you sketched out on the whiteboard will work on the 1,000,000 records that we have in production in addition to the 5 record toy example that you worked out?
I mean, sure, I agree that brainteasers are in appropriate. Knowing why manhole covers are round or how M&Ms get their candy shell tells me nothing about your programming ability. But basic math and algorithms? They're a lot more important than you think. Knowing Big-O notation, various kinds of basic algorithms and a good assortment of data structures lets me know that you can evaluate a solution and find potential flaws and bugs.
How is putting someone on the spot to build something and build it perfect a good test of their capabilities and skills?
There's a difference between perfect and working. I've never had an interviewer ask me for perfect code. Not at Amazon. Not at Google. Not at Facebook. I have, however, been asked for working code. The code doesn't need to be 100% bug free, but it should be syntactically correct and should solve the problem in a reasonable fashion (e.g. without using inordinate time or space).
Frankly, I like technical interviews a lot more than the alternative, which increasingly boils down to, "Make me a webapp and we'll evaluate your code." I'm sorry, but if I wanted to make a webapp for you on my own time, I also want to get paid for it. Even now, when I'm single and living alone, I have a hard time motivating myself to do homework problems. I can't imagine having to do something like that when I'm married and have children.
If someone is married and has children, then it's likely they've been working for a while (for example, assuming a typical graduation at 22 and kids at 30, that's 8 years in the industry). In that situation, technical interviews require a lot of time preparation. It would require looking at old college textbooks/notes, doing practice interview questions, etc. The amount of time required is probably about the same as a "make a webapp".
Yep, that's exactly how I do it. I've seen people use whiteboards and pens and paper to solve problems, but nothing solves a problem like tactfully tackling it head on and persisting until you solve it. Having said that, some people work better writing down their problems on a whiteboard and others physically solving them.
In my day-to-day I try and avoid writing as much code as possible. If someone else has solved the problem, I'll read through their code and understand it, then implement. Developers at the end of the day are hackers. I think as long as you know how and what to Google, then you're more than capable. The difference between a good developer and a bad developer isn't knowing all the answers it's knowing where to find the answers when you need them whether that be documentation, a Google search or even a physical book.
Because lets be honest here. In the day-to-day life of a developer what percentage of developers actually solve their problems manually using their brains as opposed to finding the solution on Stackoverflow, forum or blog post? I'd be willing to bet a large percentage do. When I get stuck on something I hit Google, in an interview if they were to take into account your problem solving skills in the form of a Google search, then that would be more than fine. But I am willing to bet most interviews don't allow you to search for answers.
nothing solves a problem like tactfully tackling it head on and persisting until you solve it
The problem with this approach is it very often leads to you getting stuck on a local optima. For many problems that is good enough, but there's a good chance you might miss an entirely different approach that will lead to more optimal solution.
I'm torn on this. I get what quanticle is saying, that Big-O is very important. I've developed an appreciation for it even if I didn't take CS in college (I took the redheaded bastard stepchild major of CIS).
quanticle is suggesting there is a purposeful, formal means of discussing and identifying performance, irrespective of how it's measured/profiled or addressed.
Here's the rub: why don't job openings eliminate the other requirements and just put Big-O (and a language) as the two must have items on the list. Keep it short. Eliminate the History majors, the non-college types and the CIS (almost but not good enough) types.
That, more than any FizzBuzz, should weed out the chaff instantly. If it's that important to that specific job, then just say so explicitly. Help folks self-filter themselves out.
Let's just hope getting in the door with that Big-O knowledge isn't wasted on run of the mill CRUD apps.
Or he realizes he doesn't understand the hard core mathematical basis, but knows to go looking for the most efficient algorithms for a given problem type.
The history of computing is largely dominated by software which got done, not done perfectly.
You're implying that low level CS knowledge plain isn't important in a developer role. Perhaps in some cases it isn't, but lacking low level CS knowledge is very limiting in a technical role and there are valid reasons employers might emphasize technical chops.
There exist very good developers with drive and vision who have a strong grasp of technical fundamentals and can still manage to pass technical interviews. Why wouldn't you want to hire them? No one expects interview code to be "perfect", especially not for your first answer. Thinking about it aloud and going down a false path or two rather than trying to write a perfectly optimized solution first time is what you're supposed to do and a good interviewer will encourage that. On the other hand, you should also be able to get onto a fruitful path rather quickly, or at least when you get there, you should be able to follow it easily.
"In my experience, when we are programming on the job, we're given the problem and we have time to think about it. We have time to research possible solutions, we have time to try stuff out that we know will most likely fail, and we can wait until we have something decent before we show it to our colleagues"
^This * 2
:-)
That's exactly how it played out on this thread in real life. Let me explain.
philwelch mentioned something about using anagrams as simple example to test for coding knowledge. My first attempt was a solution to palindromes. (Seriously...I totally botched the interpretation of the problem).
Elsewhere in the thread, typicalrunt sketches out some Ruby to show the answer but the crucial piece was him mentioning "sort(word1) == sort(word2)". I didn't know that. Language aside, I get it now, but I didn't then. If you don't know that, all the knowledge of language syntax in the world won't help.
So, just sort two strings. Cool. Except, I had never sorted a single string. It just never came up in day to day work. A quick peek over on StackOverflow yielded two bits of information I needed (I'm doing this in Java).
1. Convert the string to a charArray (something else I never used).
2. Sort the charArray using Arrays.sort()
With those pieces in place, the solution became evident.
But....I'm doing this at home with the TV on, relaxing, with nothing on the line, no one looking at me, the ability to research things, and generally no pressure at all. I was just curious on how to get the answer.
Change that to a job on the line, typicalrunt's advice/pseudocode not available and folks staring at you, and it's not going to end up the same way. I drew a blank at home. I'd definitely draw a blank in an interview.
Elasped time to solution (actual coding): < 1 minute.
With solution in hand, am I now somehow "better" as a developer? I think not. I did what I would do on the job: research when needed and apply it to the problem at hand. I could be faulted for not ever having used toCharArray(). Fair enough. But in the grand scheme of things, did it matter? Heck, there's plenty of pieces of the Java language I don't use daily or not at all.
Anyways, I just wanted to share how this played out.
Saw Pamela speak at an HTML5 Meetup at Yelp a couple of months ago. She was smart, funny, engaging and genuine - exactly like she sounds in this article. Love her honesty and how she wears her heart on her sleeve.
Curiously enough, this interview happened 6 hours before that talk. And it made me wonder, "why am I giving a talk tonight?? how am I qualified?" Thankfully, I recovered my confidence by the time that talk came around, and you guys were such a cool audience. :-)
I really feel for the author. I had a similar problem, except on the other side of the table - evaluating whether candidates' solutions while interviewing them over the phone was excruciating.
I built https://coderpad.io to make the whole programmer-interview phone screen process easier. It's like Stypi on steroids, because it comes with a live REPL. You can play around with, say, JS, right there in the window. I've found it much more accurately simulates coding alone: the experimentation, quick validation, and verifiability that editors alone don't have is sooooo important.
Unfortunately, there are way too many unqualified people out there applying for jobs and ruining it for the rest of us. As a hiring for manager for years, I've seen a huge number of people try to fake their way through interviews. They can talk vaguely and grandiosely about all the technical problems they solved at their past jobs or in school. But the breakdown always happens when you ask them to solve a problem for you.
Sure, they may be undiscovered geniuses who are too nervous to think straight. But where there's smoke, there's fire, and in this case behind the smoke are ten other people lined up to interview for the same job. All things being equal in terms of general personability and hygiene, the one who displays technical competence gets the job. Just imagine if the odds weren't so skewed in favor of job hunters. When the economy slows, when your jobs are really subject to outsourcing to a lowest bidder overseas for a fraction of the price, these sympathy hires will be a thing of the past.
A few months ago, I decided that I really wanted to
join Khan Academy. I would longingly browse their
jobs page in my idle hours, read their intern blog
posts from top-to-bottom, and fork their repos on
Friday nights. It became a bit of an obsession.
(Shh, don't tell them how creepy I am.)
I realize that she's joking here with the creepy bit, but I have to ask. Does anyone honestly think this kind of behavior is creepy?
To answer my own question, as someone who interviews people quite often, I'd love to hear that a candidate did this sort of thing. Honestly, if you're worried that you're going to be weak in your interview for a company which you really want to work for, do this.
Oh, and also - if interviews make you nervous, please tell your interviewer. For some people it's really obvious that they're nervous, but others do an amazing job hiding it. If there's any chance at all that you think your interviewers aren't aware of your nerves, just tell them.
First off, we're used to it. I'd guess somewhere between 60 to 80% of my interviewees are visibly nervous. Further, quite often the people interviewing you experience the same sort of nervousness when they're on your side of the table. If they're good at what they're doing, they'll totally empathize, and clearing the air helps a lot.
Unless you're interviewing for a sales type position where it's essentially your job to be interviewed, nobody in their right mind should view nervousness as a sign of weakness. Personally, about 9 out of 10 times this happens I err on the side of "this person really wants this job" rather than "wow, this person must be nervous because they don't know their shit."
> To answer my own question, as someone who interviews people quite often, I'd love to hear that a candidate did this sort of thing. Honestly, if you're worried that you're going to be weak in your interview for a company which you really want to work for, do this.
Guilty as charged. There are a handful of companies that have provoked this type of reaction from me. I have yet to actually get an interview from any of them, though.
I'm actually surprised how much Pamela sounds like me when it comes to interviewing. Almost exactly like me. I just have crippling anxiety on top of that which I only recently got taken care of.
One approach I've tried recently is getting a digital ocean box and giving the prospective candidate access to the box 24 hrs prior, just so they can install whatever tools they like if they so choose, and then giving the candidate a simple problem to build on the box for 30 minutes.
Then we mute the phones for 30 minutes, so that people can work in peace, but we're around if they want to ask us any questions.
And at the end of the 30 minutes we discuss how far they got, and what they would do differently, etc..
We've only done this twice, but I think it was fairly useful both times. Haven't done it enough to know whether or not this is a good approach.
I had the same thing happen to me while interviewing for the position I hold currently.
I had gotten through 6 interviews without a hitch. They were all culture related, personality-based, "how will you fit with our team" type. Young, scrappy kind of team, who didn't believe in big corporation-type interviews.
Well, the interview in question came. I'm a rockstar when it comes to interviews. I'm personable, confident, funny, and I carry myself really well in conversation. But what was different?
The guy I interviewed with previously held a position with Microsoft.
And he did things the Microsoft way.
I'm nothing but a loose guy. I don't have the college education to back me. I hadn't worked on any famed projects. I didn't commit to open-source projects, have a network to vouch for me. I had a few side projects, but they were nothing more than me just building things. I owned a few companies in my mid-late teens that sold for a good amount of money, but nothing I'd write home about. I'm a utility player that can wear many hats: front-end, back-end, sysadmin, marketing, advertising, copywriting, legal stuff, financials, client management, databases... I don't do any one thing particularly well (except for maybe back-end) but I do all of them pretty damn good.
And when I heard this guy was from MSFT I thought "well this is going to suck." Why? Because I thought that he may not care that I'm as scrappy as I am.
The position I was applying for wasn't anything specific. It was just that utility player. I wasn't expected to write gobs of code, I wasn't expected to manage systems. I was just expected to do what was asked of me, and take ownership of it.
So the interview began. He asked me some really, really low-level questions. Questions that you'd be asked on an exam if you're a CSCI major at MIT. I had no idea how to answer them, so I tried to explain them high-level and ignore the fact he wanted low-level answers.
That got me nowhere except him thinking down on me.
When it came down to me writing code, I just blanked. He wanted me to write a class for a shopping cart and he wanted to see how I'd apply a coupon code, how I'd manage the items in the cart, how I'd sort and group them together, etc. It lasted all of five minutes when I just disconnected from the internet, took the battery out of my phone, and sat there.
I suffer from crippling anxiety, and this was no exception. I've learned how to manage it better since, though.
About twenty minutes after my meltdown I emailed the CEO of the company. I was in tears, because I wanted the job so badly, and I explained that to him. I said that the guy was an asshole, really belittling, and didn't reflect the values of those who I had spoke with prior.
I still got the job, and I'm still with them today. It's the butt of a lot of the jokes we have with our team of five. "Josh, is this someone you'd consider hanging up on?" or "Well it's okay, because Josh once hung up on someone that was interviewing him. Thus, this isn't that bad in comparison!"
"About twenty minutes after my meltdown I emailed the CEO of the company. I was in tears, because I wanted the job so badly, and I explained that to him. I said that the guy was an asshole, really belittling, and didn't reflect the values of those who I had spoke with prior"
- To me it's perfectly clear who was the real asshole there. Bitching about technical problems to the CEO, haha..what a loser. You didn't have the caliber to handle, you should have not blamed anyone else but yourself.
The fact is that he is still with that company, so he obviously was a good fit. So it is the interviewer who didn't have a caliber to do his job - determine if the person was a good fit.
I wouldn't go as far as to call me an asshole, and I definitely didn't <blame> anyone. But to each their own.
Full-disclosure: I admitted that I panicked, and that I was unable to complete the interview. I'm pretty candid about things, and in our first interview (which was with our CEO), I told him that I was naturally nervous, and was the captain of my team in a battle vs lifelong anxiety.
To me, though, being employed is to be part of a team. Sometimes, you have to pick up your teammates. Everyone commits an error from time to time, nobody's perfect. You're not expected to bat 1.000. It's not about that you bat .250 coming into the season, but where you end up as a team and individually at the end of it. To this day, and 9 months into employment, I'd like to think that I've offered a lot of value despite my shortcomings. I'm confident that anyone on my team would say the same, too. Which, to me, is what matters.
I recently had an interview with Google London, Oh boy did a screw it up, it's like I'm reading my experience, my mind simply went blank ! The interviewer was like the nicest person I've been interviewed by (he's pretty famous and I recognized his voice btw). The task was hilariously easy (I've done the same thing like 2 days ago), but when I was asked to work on it on the phone I bombed ! Basically I was thinking of everything else BUT the problem at hand. The guy kept being nice telling me that this happens all the time (you know people freezing). At the end I wasn't lucky enough to proceed to the next level and frankly I felt that I deserved to fail and I should not be a programmer and what not. Very depressing - I took at as a BIG personal failure and missing an opportunity of a lifetime. I think the thing that makes it worst is the fact that I honestly knew this, after the interview was over I looked at the problem and couldn't believe that I wasn't able to do it. Complete mental block. I think I've recovered since, and I'm trying to improve. Just want to say (because I've heard complaints from other people) - Google were very nice, and worked very quick with me, I've submitted my CV and got a call the same day (maybe just lucky), then I basically chose the date of my interview and they were spot on. Also they gave me good feedback.
"If you ever suffer from Impostor syndrome, you should probably prepare even more, because the standard interview situation will make you feel like an Imposter in the worst way."
It may also help to think of that Dunning-Kruger study [1], according to that it would be normal for the competent to be full of doubt while the less experienced would be full of confidence.
To share my experience, the kinds of "make you cry" interviews which test a lot of CS knowledge seem relegated to the Bay Area companies which I interviewed with. All my interviews with companies in Texas (where I live) would interview more to ask about your experiences with a technology, challenges you'd faced when working with a similar stack, and maybe ask how you solved some tricky performance problem. They tended to focus more on the real world experiences than the academic knowledge. Once in a while someone would throw a framework- specific curveball your way to make sure you knew what your resume said you knew.
This isn't a knock on the companies in the Bay Area, because clearly what they are doing works for them, but it's just a drastically different style that feels foreign to me.
When those of you in the Bay Area interview every few years for new jobs, do you just brush off all your CS textbooks, study/review for a month, and then start the process?
That's what I've always experienced. I've only had one company do a technical phone interview with one company - Google - and I told them not to call me back.
All the other jobs I've only had an in-person interview and it's been about experiences and the occasional "Why is Prototype a bad Javascript framework?" for example.
Anything more complex than FizzBuzz will take me a while to figure out, because I never start writing code without actually understanding the problem that's being solved.
I happened to have a conversation with someone about this recently.
Software developer: the career where you study to get the job, toss much of that away when you actually do the job, then study for the interview for the next job, where you toss much of that away when you actually do the job...
It's a rhetorical question, but I can't understand studying for a job. You either do it on the job or you don't. Studying, however, puts a ton of stuff in short term memory where it gives the appearance of brilliance.
See that's where I want to dig deeper with those that really believe in this style of interviewing. Clearly it must have some value other than the weed-out factor.
Are there engineers who actually keep the algorithms/data structure stuff in their head and find small ways to apply it while they work on a standard web or mobile app? That's the only way I could see the knowledge having any value, but I've never worked this way nor have I observed anyone who works this way.
Pretty funny to see this, because you've interviewed me before (not super technical, though!). Even more so, I think I got the same question when I interviewed for KA ;) Small world.
Great post, it's really refreshing to see someone with your hacking chops experiencing the same anxieties as a little college student like me.
I seem to remember that while I was interviewing you (for Coursera), I was browsing your Github repos and checking out your cool libraries. I didn't feel a need to test your technical chops when I could see them in front of me. I'd rather get to know other sides when I can. :-)
I'd also recommend - if you're an interviewee normally, try to join an interview at your current job (if possible). I was quite anxious about being interviewed some time ago. Being on the other side of the table once changed everything however. The amount of preparation, ideas, planning, time needed to interview someone is just crazy if you want to do it properly. And you're still left with: what if I didn't ask the right questions, what impression I left on them, are they going to be a good fit or was their behaviour just an act for today, etc.
Now I know that the people of the other side are either just as anxious to do everything right as I am, or they just don't care much. One way or another, I'm just not able to get nervous about an interview anymore. The best ones turn into pleasant chats about tech anyway.
Rather relevantly, this article [1] was posted a few days ago.
“In improvisational acting there is this great rule that I’ve used in my life: ‘Act As If’. Act as if this is completely normal. Of course I’m supposed to be interviewing Barack Obama. Of course I’m supposed to be playing guitar with Bruce Springsteen and of course there’s a big part of me inside that’s saying: ‘What are you talking about?’”
It's been in the front of my head since I read it and, while I can't say I've passed any major coding interviews, it's certainly helped quell the screaming voice inside me that is the source of my constant anxiety.
You know, there's a lot of these interview articles. Where whiteboarding doesn't work. Or whiteboarding works but filters out a vast majority of applicants. Give homework problems instead. No, applicants can easily cheat or resort to copy pasting code when given time away from the interviewer. The future is GitHub repos to see candidates' prior work. No, open-source code is not verifiable and easily falsifiable.
What about determining if an applicant has solid fundamentals, not overthinking it, and having him or her learn on the job? Whatever happened to providing training? Is there really no room for apprentices or novices in modern tech companies?
This post hit a nerve! This is exactly what got me working on http://www.phoneinterviewme.com/ to help fellow developers prepare better for their phone interviews.
Here is how it works. You get a call at a time you pick on your phone line. You are asked some automated questions related to the interview you expect. Once you are done, you can go back and listen to your session to see how you did. I was hoping this would help developers practice and get over the fear during an actual interview. Would this help you be better prepared?
I recently had a couple of skype interviews with a well-known startup. First one went fine and I got good feedback but during the second one I was asked to share my screen & work on an IDE of my choice. That being something new, I didn't feel comfortable coding and though I did answer, the interviewer felt I was a bit (iffy) - I had written a conditional check & then changed it later. Interviews can get pretty hectic and it is way easier to say no to someone you don't even look at.
For me, what helps a lot with tech interview stress is to already have a job. It turns the tables so that I don't really need them, they need me. The change in attitude is powerful and gives confidence.
What's important about that is: you are not getting an accurate read of a candidate's state of mind in a high-stakes interview. I think you have just a couple of options to improve your accuracy given that fact:
* You can take deliberate steps to lower the stakes in your interview process by changing the kinds of transactions that happen in the interview (for instance, by constructing interviews that involve collaborating on a problem instead of challenging the candidate).
* You can "discount" the signals you get from the interview, particular w/r/t already nebulous concepts like "confidence", "social skill", "enthusiasm", and the dreaded "culture fit".
* You can push more of the recruiting process into situations where the candidate controls their environment (for instance, by doing stuff at home, and on a fixed but flexible time scale) and less on in-person timeboxed interviews.
I'd love more ideas. We're trying all three, but it's a lather-rinse-repeat kind of situation. I'm getting good, I think, at spotting signals that are probably the result of jitters. And I think I've learned pretty conclusively that those jitters have zero correlation to on-the-job performance. But I have so much more to learn.