I’ve hired dozens of Sr. engineers and given coding tests to almost all of them (after a 2nd round interview, never earlier).
However, my tests are time-boxes (1hr) and are about greenfielding an app (like a ToDo app) to see how they approach a problem rather than programming knowledge.
> Why?
About 33-50% of devs that pass the two interviews fail the coding test. Of the ones that fail, they’re great at interviewing but probably oversold their skills. Some are really poor at time management. Some are really poor at simplifying problems. And some are really poor at following directions.
It’s possible that they were on teams that carried their weight or possible they were exaggerating their capabilities or prior responsibilities.
The opposite has happened too where I’ve been blown away by the code quality and approach.
But overall it’s been a very helpful datapoint in hiring.
This is a hypothetical, but how would you suggest engineers that have issues with simplifying problems, managing time or maybe don't realize they're overselling their skills work in a positive direction to correct some of these things?
I've noticed many companies will give technical feedback if you don't pass a round, but seems like there has to be some kind of path to help people like this. Always been curious to me that "ideal" practices for managing time, communicating technically / in a business context never seem to really be written down. I learned some of this (albeit badly) at a few internships, but if there'd been some source to read even if it was only the first 25% that would've been immensely helpful out of college.
Practice rewriting personal projects. Once you have a working app, try to make it even simpler, clearer, more efficient, etc.
> managing time
If you’re on a tight deadline do the easy stuff first and if you’re falling behind get help with the harder parts or ask the team to scale back the scope. But if you bite off more than you can chew you run the risk of having nothing to show for it.
If you’re building a proof of concept, then start with the hard parts to show the team that it’s possible and worth additional time to finish it off.
> overselling their skills
Be honest about your role in a project. Don’t hide the fact that another dev was doing the primary research or making the technical decisions. Just take credit for the things you did from start to finish, without help (because those people that helped might not be there at your next job).
>About 33-50% of devs that pass the two interviews fail the coding test.
I don't mean to sound insulting, but if your organization can't tell if someone knows what they are talking about after an interview, much less two interviews, what the heck is going on?
All you have to do to be indistinguishable from someone who knows what they are talking about in two interviews is to say the things that someone who knows what they are talking about would say, in the way that they would say them. This is obviously a nontrivial skill, since it involves memorizing a lot of declarative knowledge, but for extraverted people with good memories it can come more easily than actual coding, let alone debugging. An interviewee who biffs it slightly can be very hard to distinguish from an interviewee who knows so much more than the interviewer that they identify errors in the interviewer's knowledge. And at this point there's a cottage industry of books and YouTube videos specifically designed to help people fake this stuff.
Basically what it comes down to is that doing something and explaining it are different skills. True of riding a bicycle, true of fixing a car, true of programming. Karl Popper never discovered shit, Harold Bloom never wrote anything worth reading, and plenty of bestselling novelists are utterly terrible as creative writing instructors.
>All you have to do to be indistinguishable from someone who knows what they are talking about in two interviews is to say the things that someone who knows what they are talking about would say, in the way that they would say them.
Sounds like a supervillain or something. Maybe you're asking the wrong questions or maybe I have a knack for it, but the only thing I can't detect is motivation, which is probably more important than current ability. One of my worst hires was a guy who could answer all the trivia; sharp as a tack, but didn't have the discipline to do the work. He was always slacking and I couldn't count on him to get things done, even with loose deadlines.
I mean most hiring managers don't bother calling references either. That yields a lot of insight; those references were hand picked by the candidate. Maybe call their last employer, even if it wasn't listed as a reference.
One of my first questions is, "Tell me about your favorite project and why." If they answer in a sentence or two, even when prodded or show little or no enthusiasm, it's a pass. You have to like the work to be good at it. Ask some follow up questions to their answer. A phony can only go so deep.
Second question is, "Tell me about your least favorite project and why."
Maybe you're just better at interviewing people than I am. (That sounds sarcastic, but it's not.) I'll try to take your advice to heart if I'm interviewing someone for a job in the future.
It's true that if someone has memorized the details of a "least favorite project" from dailywtf.com, or a "favorite project" from some other dev blog, they won't be able to go very deep into it; and if they don't have at least deep declarative knowledge about the technologies involved they won't be able to invent anything plausible either. Deep declarative knowledge, though, is still very different from debugging ability or the ability to find simple solutions to complex problems.
The enthusiasm thing is discussed pretty often in all kinds of job interview advice contexts, not just programming. I think many people make an effort to project enthusiasm in that situation, and in some other social situations like dating. Hopefully you're filtering for real enthusiasm instead of ability to emote, but realistically it's probably a combination.
>Deep declarative knowledge, though, is still very different from debugging ability
90% of debugging is reading (and understanding) whatever exception was thrown. I can't count the times when a really good dev asks me for help and I ask, "well what does the error say?" Once they read it, they're like, "never mind."
If reading/understanding the error doesn't work, it really requires pretty deep knowledge of the application architecture and possibly auxiliary systems and infrastructure to figure it out. If your application is custom, then they can't have that knowledge coming in and can only get it with OJT. That extra 10% is really the hard part, but you really only need one or two people who can cover the whole team on that extra 10%, and the process will help improve the people who aren't as good at it.
>or the ability to find simple solutions to complex problems.
That's where the genius really is. Reading the first few chapters of Donald Knuth's first volume really blew my mind. Probably won't be able to discover that talent in any interview process.
>Hopefully you're filtering for real enthusiasm instead of ability to emote, but realistically it's probably a combination.
Yes, it's just a piece of the puzzle. Lack of enthusiasm is more a red flag than putting a lot of weight in enthusiasm itself.
Making the simple complicated is commonplace; making the complicated simple, awesomely simple, that’s creativity.
Not parent commenter, but the first two are often designed to be rather trivial. No point wasting hours of the candidate/interviewer's time if the candidate just doesn't work out. We've found people who straight up lie on the resume (claim to use a technology then backpedal when asked about it). And there are the kinds of people who just don't show up or can't speak passable English. And there are some who have a history of making CRUD apps for gov and giant corporations, but can't tell when to use an Array or LinkedList even when allowed to google it.
The real coding test later is to actually see that the person can code. Normally it's about following instructions, isn't afraid to get their hands dirty, bangs their head on the wall thinking of a solution but actually tries something.
There are also lots of people who will build everything except what you ask them to. You tell them to build a To Do List app and they go straight to building logins, adding profile pictures, and then complain that the test was too short.
We also often list technical requirements in the job description. The tech stack is always some niche things. It's okay if you don't know it all (nobody really does) but you have to be able to work with the code once you're working on the code.
A lot of people list buzzwords for everything they watched a tutorial on. Someone may have 3 years doing something but are actually bad at it. Someone may have learned it last week and want to be given a chance to prove that they're fast learners. Great, if you've been working with that tech for 3 years, this should be muscle memory. If it's new to you, well, let's see your learning process.
1) People lie. I found out one person had someone listening in and chatting the answers to them.
2) The interviews get progressively more technical. We do a non-technical first round (some people don’t show up on time, aren’t great communicators, etc.), then have a technical interview about high level concepts followed by a technical test.
3) Efficiency. We can learn more from a 1hr coding challenge than a 1hr verbal interview. And it requires less involvement from the team.
What you're measuring as "bad time management" is likely to some degree people being frustrated and dismayed and giving up because an arbitrary timed test isn't fun or interesting or rewarding, when there are companies probably paying just as much as you with an easier interview process. So yeah, anecdata can be tricky.
What I mean is if we ask them to spend no more than 1hr on a ToDo app, I’ve seen people spend 1hr on the folder structure, importing a ton of packages, abstracting logic, etc. and don’t build the basic CRUD functions.
So you basically end up with a giant empty shell of a project.
Commits are not verifiable, any half-competent git user can google how to change committer and author dates. I used to work for a company which did these "self-governed" tests and most candidates who bothered with these spent more time than they actually were supposed to to get an edge.
Fair enough, but we ask the dev to send us an invite to the repo when they first create it, so then we ask them to Email us an hour later and we check the repo.
I’ve hired dozens of Sr. engineers and given coding tests to almost all of them (after a 2nd round interview, never earlier).
However, my tests are time-boxes (1hr) and are about greenfielding an app (like a ToDo app) to see how they approach a problem rather than programming knowledge.
> Why?
About 33-50% of devs that pass the two interviews fail the coding test. Of the ones that fail, they’re great at interviewing but probably oversold their skills. Some are really poor at time management. Some are really poor at simplifying problems. And some are really poor at following directions.
It’s possible that they were on teams that carried their weight or possible they were exaggerating their capabilities or prior responsibilities.
The opposite has happened too where I’ve been blown away by the code quality and approach.
But overall it’s been a very helpful datapoint in hiring.