I hate code-writing puzzles during interviews. I don't use them at all. Why? Precisely because of methods such as this one where someone might memorize or learn a bunch of patterns to common puzzles going around. They come in and ace the interview test. Does it really tell you anything about the person other than they can train themselves like an intelligent monkey to produce the correct output given an input?
My preference, by far, is to spend time discussing ideas and approaches to solving problems. I don't want to see code. I want to see into his/her brain. How do you think? How do you slice and dice a problem to try to get to core? How good are you at intuitively choosing a good data representation? Do you bring-up alternative approaches to optimize for size vs. raw speed or other parameters? What questions do you ask?
Anyhow, whether the person can actually write two lines of code or not can be ascertained by looking at prior work (if available). The most interesting aspects of a programmer --the real reasons you might not want to let him(her) get away-- have nothing whatsoever to do with how quickly they can write a bunch of loops or if they have memorized five sorting algorithms. Nah, the real value is in how they think. That's what you want to hire them for.
EDIT: Some of the best, most creative and versatile programmers I have known could not write code without a set of reference books next to them. Why? Because they don't have encyclopedic knowledge of the various languages they might use. They'd excel at dissecting data to reveal structure and representation and then choosing a good approach to solve the problem but more often than not had to have a library of various CS and language books around them to support their work. My point is that these people would have failed interview puzzles yet they contributed to and sometimes single-handedly drove projects that generated millions of dollars for the companies that employed them.
We get our interviewees to sit in front of an IDE of their choice and write some simple code. And I mean simple! No fancy algorithms or tricks.
I ask them to write a method that takes a list of Strings and returns a single String which is all those Strings concatenated together with spaces in-between. And oh my word, how people fail. It's truly frightening.
This is interesting. I'd like to know more. Are you targeting young inexperienced programmers? Are they just out of school? What kind of filtering do you have prior to bringing them in?
I have never hired a recent grad for programming.
I used to own an electronics manufacturing company near UC Riverside (California) some time back. We had pick-and-place machines and the usual assortment of equipment you'd have for the assembly and testing of circuit boards. We made microprocessor-driven industrial motor controls.
During that time I thought it'd be great to hire engineering students. As someone who was a capable electronics hobbyist before I even touched college I assumed that these kids would be hell-bent to work in electronics as I was when I was in college. Not so. I probably went through 15 or 20 of them before I found a couple that were worth the effort.
When it came to programming I never hired anyone with anything less than five years of demonstrable experience.
We were looking for experienced Senior Java Developers. I'm talking about people who have 10 or more years of experience on their CV, working for well known companies - large banks etc. We give the CVs a fairly good going over before deciding to get someone in. They've also responded to a short questionnaire about their experience with agile methodologies. (And of course, the job agencies have supposedly already done some sort of filtering.)
Interview nerves are one thing but even if you're nervous you should still be able to do the equivalent of multiplying 5 * 10.
Just for some clarification, when I say fail, I really mean fail to do it well.
Most of them at least manage to get it working but don't necessarily cover all the problems that can arise and can get it to 100%, with some prompting.
It's a real joy when someone comes in who obviously loves programming. Like night and day.
Or something more complex? I can't really imagine it being more complex, but I can imagine people failing. I too am sometimes shocked at how people end up getting programming jobs who don't grok extremely basic concepts.
I do understand how some people end up in programming roles - often just because they show some ability to get stuff done - they're the least bad in an org, and no one else wants to do it. But that's different that someone intentionally applying for a job as a "developer" and not understanding the concept of a loop (I've met a couple over the past 15+ years in the world of paid developers).
We were interviewing for Java devs so basically a method with this signature:
public static String concatenate(List<String> strings)
And I have a StringUtils class ready to go with an example method and say "imagine there are a bunch of methods in this class and we want production ready code"
There's a lot you can learn.
- do they decide to write unit tests up front (note that our questionnaire asks about their testing experience and everyone rhapsodizes about their strong testing background)
- if not, once they've written the method, how do they respond to the question "How do we know if it works?" (cue people writing main methods within the StringUtils class)
- what kind of unit tests do they write? Can they come up with good tests that cover all cases
- what names do they give to their methods/parameters/variables. (One guy called his method bangThemTogether)
- what bugs do they have - what cases do they miss
- if a unit test fails how do they go about fixing it (interestingly only 2 people fired up the debugger)
- do they ask questions if they need clarification
Minor things
- keyboard/IDE skills - do they laboriously type everything or have they taken the time to learn their preferred tool
- can they explain what they're doing - do they speak at all...
There's a second part to the practical - a bit of a code review/refactoring exercise. Sometimes I skip it because they've taken more than 30 minutes on the first part or their just so bad it's not worth continuing.
BTW we've interviewed about 30 people with this technique. They first meet with the dev manager for culture fit and general knowledge of dev processes. Then they meet with a tech lead for a more technical interview - explain your last project etc. Then I do the practical. Many people get through the first two just fine but fail badly on the practical.
You are setting yourself up for failure using these methods.
- You're being misleading by asking them to write a concat method when you really want unit tests for that method. Just be straightforward. You're being too clever for your own good.
- People rarely code in a linear fashion. For example, I often write the meat of my methods first. Then do edge cases / error handling in a second or third pass. I have been dinged for this in interviews before. But imo, it says nothing about me as a programmer.
- You want them to be comfortable with tools, but they are behind enemy lines and under enemy fire. Dont expect them to achieve any kind of comfort level (and I would put 'thinking to use debugger' up there as something you would forget/ignore while being uncomfortable).
- When you are in extreme concentration mode, do you talk to others? Probably not. If you want them to talk, ask them questions. Again you're expecting them to read your mind. (also, keep them away from a keyboard if you want them to talk. Put them in front of a white board instead)
As others have said, talking to people and getting to know them and how they solve problems is much better than giving them mechanical interviews.
It's really not as black and white as you seem to be imagining it. There's lots of prompting and asking questions. If they don't think of unit testing straight up, that's fine. It's just a nice indicator if they do think of it. Things generally flow as you describe - in a non-linear fashion. They'll get the basic method done and then I'll ask them about possible edge cases and error handling. The problem is that even with this prompting, many people can't come up with what those may be. And then, if I suggest what they may be (eg how should we handle nulls), how they go about handling that in their code is when it can sometimes become really messy.
It's definitely not a failure. When a good programmer comes in, it's immediately obvious. They have no trouble. I mean, it really is basic stuff - writing a loop with a few if statements - not some fancy egg dropping puzzle or the big O notation stuff that Google gets you to do.
Good list of how you interview/test and what your expectations are! I'd have to admit, I don't often even consider writing a test for a method that would just return a join() - I trust the join() method in the underlying language 'just works', although there are other reasons for writing tests (error handling, etc).
I do some PHP teaching, and one thing I got some good feedback from was giving students assignments that had asserts in them. Not quite unit-testing specifically (not using a full testing harness), but I'd give them some shell code with assert statements and they'd need to 'make it work'.
The sharp ones actually made it work, and gave me feedback that they appreciated that a lot, as it showed them how to think about breaking down things in to testable parts (even trivial stuff). The not-so-sharp ones... easier to spot when they'd give me back code that obviously was never even run in the first place.
Reading your description below (test cases et. al.), it doesn't seem as simple as you first make it seem. I start with something like, "Write a loop that displays the numbers 1 to 100." About 80% of the people I phone interview fail this test. Almost 100% of newly graduated students fail. I'm not sure if they are overthinking it, if schools are no longer teaching anything practical, or what.
> Some of the best, most creative and versatile programmers I have known could not write code without a set of reference books next to them. Why? Because they don't have encyclopedic knowledge of the various languages they might use
You don't have to know everything about a language to solve problems in that language; if you need a set of reference books next to you for the languages you use every day, you don't know those languages very well.
And a good interviewer will help you with discussion, if you have a reasonable approach to the problem: "I think there's a function to do foo, but I'd have to check the documentation to find the details." "It's called foomod.foo_doer; it takes a baz and a bar." "Ah, thanks. scribble scribble". (Or, alternatively, "I'm assuming a function foo_doer to do foo, which I think exists in the standard library, but if not I could write it easily enough.")
The best example I have against this goes back to physics and calculus classes. I have to try to be very sensitive here.
There was a certain ethnic group who overwhelmingly aced calculus tests. I mean, they were machines. Got it done faster than everyone and just aced the tests. No contest.
The amazing thing was what happened in Physics class, both tests and labs. They were completely lost. It was really an odd and unbelievable thing to behold. Some of them would get tripped-up by the simplest things.
I won't venture to guess as to why and how this happens. I'll just say that, in my experience, mechanical memorization of anything isn't learning at all. It's just memorization with no value whatsoever beyond that.
I am trying to think about a CS example.
Let's say that someone memorizes the implementation of a binary GA algorithm or a NN. Great. They can type it out in an instant when asked. Do they really understand what's going on? Do they understand when to apply it, how and why? Do they understand how to optimize it? Do they know how to make good chromosome encoding decisions? Or how about dealing with cases where the GA is lost looking for a solution? Do they know how to fix it? Or how to deal with it?
Or how about the choice between writing a class with a bunch of methods and properties vs. a simple routine with lookup table to solve a problem? I've seen people who think that every solution requires a class and a bunch of objects. You end-up with code that is fifty times slower and ten times larger than it needs to be.
Some of this only comes from experience, of course. If available, I like to look at code that the candidate wrote and explore it. It's hard for someone to bullshit you if it isn't their code. Another interesting thing to do is to put code in front of them and see how they go about understanding what's going on. After all, if you are hiring them to plug into an existing project that is precisely what they'll have to do. Someone who can't code will not ask intelligent questions about the code they are looking at. They will have a hard time getting into it and understanding what's going on.
Now, I do understand from other posts that there seems to be a failure in the system in that schools are graduating people who can't write a program that prints numbers from 1 to 100. I can't figure out how someone can come out of a reputable university with a CS degree and not be able to write that program, but that's besides the point. If this is true, then, well, I hate to say this, puzzle testing may be a necessary evil. The problem is that this isn't enough because you don't know if you have a memorization machine in front of you or someone who really understands the topic.
EDIT: I should say that I couldn't imagine being a company like Google or similar and trying to hire large numbers of developers on a regular basis. I would imagine that at that level you don't really have an option but to apply puzzle type filters as a first line of defense.
It all depends on how you define "puzzle", of course. However, I find that "code writing puzzles", as you term them, tell me far more about the candidate's ideas and approaches to solving problems than abstract discussion. Even relatively simple problems (like the hash-table example from the OP) often have enough edge cases and follow-up topics (cache locality is a particular favorite of mine) to give you insight into what the person is thinking about when they solve a problem. You say that you don't want to see code, but instead "see into his/her brain." But how do you do that without looking at code? One thing I've found out when I was interviewing for my replacement at my last job is that people are very, very good at bullshitting. Asking them to actually write code strips away the bullshit and forces them to actually show you what they're thinking. Sure, prior work is great if its available, but I know of many really good programmers who don't code all that much in their free time, either due to family commitments or because they have gasp have interests other than programming.
I accept that when you're programming "for real", you have reference books, Google, StackOverflow and all sorts of other resources to help you out, and I accounted for that. I wasn't picky about syntax. I let them use whatever programming language they were most comfortable with (as long as it wasn't something terribly obscure, like Brainfuck or J) I even accepted the occassional algorithmic error (e.g. greater-than/less-than being flipped around) if it was clear that the applicant knew what they were talking about and was quick to fix the error once it was pointed out to them. But I still insisted on some code. I would be willing to bet that the programmers who you are referring to -- the ones surrounded with reference books -- would be more than able to bang out something like a quicksort or a binary tree without references. Sure, it might not be a perfect, production-ready implementation, but I'd expect something that at least resembles the correct algorithm, along with a discussion of the tradeoffs and edge conditions.
Frankly, I think memorization is, if anything, underrated in computer science. Our access to knowledge is quite similar to the memory hierarchy in a computer. Working memory is like RAM. Long term memory is like disk. Books and references are like the network. Having algorithms and data structures memorized is like having a local cached copy of a network resource. It allows you to be faster and more fluent when you're programming because you're not context switching constantly as you look things up. And just like in sports or martial arts, it doesn't matter what advanced techniques you know if you don't have solid fundamentals. Basic algorithms and data structures are the fundamentals of programming. When I ask a candidate to demonstrate their knowledge of such by writing code on a whiteboard, I'm assessing how good their fundamentals are. The rest can be taught.
EDIT: And of course, I forgot the most important fundamental of all: Big-O notation. Knowing algorithms is useless if you don't have solid basis for determining how and why one algorithm is better than another.
No Terms of Service or Privacy Policy. I can imagine having a list of email addresses with people interested in getting coding interviews could be interesting to the right people with enough money.
There being a TOS or Privacy Policy doesn't actually mean a company won't sell your emails. It just means they're going to say they're not going to sell your emails. Harsh, but true.
I'm also looking for guest posters with real interview questions from their company, or anyone who would like to guest post a CS topic overview. I can be reached at my username at gmail.
This is an awesome idea bcjordan! I signed up, I am really excited to get the first email. If this stays CS only, and avoids the pitfalls of 'the industry' this might quickly become my favorite thing of the week :D
Best of luck with it!
Also, any plans to monetize it? What are your long terms with the project, hopefully it will be something to keep you interested for a long time.
Except for Gayle Laakman's extensive marketing and the heavy user base that followed, I don't see why careercup is popular in the first place. It is a terrible website with ads for her two books taking up half of the home page space. It would be great if someone could start a stackexchange site for discussing CS interview questions; I'm pretty sure the quality of questions and discussions would be better then.
um.. I think it's popular because it essentially is like a stackexchange for CS interview questions..... it has a ton of questions, some of which _actually_ popped up during interviews. As for the ads, I had no trouble ignoring them.
I created something similar a few months ago to send out a interview question every other day. The time constraints of maintaining it got to me though (I'm a university student myself) and I recently stopped sending out question of the day. I have plans to revive it later but for right now its in 'hiatus' mode so to speak.
Ahh, after I signed up for PyCoder (curtailed Python newsletter every Friday), I thought of doing something similar but tailoring to college students. You beat me to it! Awesome project bcjordan :)
The signup flow was really well done. A minor nitpick: put the social buttons at the bottom of the model after the (see sample -> signup -> congrats) process.
The confirmation e-mail doesn't have any link back to the website. Hopefully your weekly messages do, it's where most of us will look for a link if we want to share it with others.
Looking forward to this- the last couple similar services (periodical design/coding challenges) that I've signed up for fizzled out after 1-2 weeks. Hope this one sticks around!
seems like a very cool idea! i'm exciting to use it and see where it goes.
You do have some bugs to work out though, the UI isn't very consistent throughout the site and somehow I got redirected to here: http://interviewpractice.herokuapp.com/ and now can't see anything. I also had trouble the signup process and had to reset my password to join.
My preference, by far, is to spend time discussing ideas and approaches to solving problems. I don't want to see code. I want to see into his/her brain. How do you think? How do you slice and dice a problem to try to get to core? How good are you at intuitively choosing a good data representation? Do you bring-up alternative approaches to optimize for size vs. raw speed or other parameters? What questions do you ask?
Anyhow, whether the person can actually write two lines of code or not can be ascertained by looking at prior work (if available). The most interesting aspects of a programmer --the real reasons you might not want to let him(her) get away-- have nothing whatsoever to do with how quickly they can write a bunch of loops or if they have memorized five sorting algorithms. Nah, the real value is in how they think. That's what you want to hire them for.
EDIT: Some of the best, most creative and versatile programmers I have known could not write code without a set of reference books next to them. Why? Because they don't have encyclopedic knowledge of the various languages they might use. They'd excel at dissecting data to reveal structure and representation and then choosing a good approach to solve the problem but more often than not had to have a library of various CS and language books around them to support their work. My point is that these people would have failed interview puzzles yet they contributed to and sometimes single-handedly drove projects that generated millions of dollars for the companies that employed them.