I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job.
On the other hand, take home challenges eat up a lot of time and credentialist filters like going to a name brand school (or even having the right major) filter me out.
My go-to for when I'm in a hiring position again would be recommendations from my personal network, but that has its own flaws and doesn't scale very well.
I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.
One of the things I've done is to hand a candidate some existing code and ask, "What do you think of this code? Do you see any obvious problems? Are there things that work but could be done better? Is there anything exceptional in there? Tell me what you see." This allows you to test that they know how to read code, which is what they'll be spending the majority of their time doing. You can ask them how they'd improve it, which should give you an idea of how they write code, too. And of course, you can see their experience by checking out their resume and if possible running some of the apps they worked on. And if they have any code on GitHub, etc., you can check that out, too.
One of the companies I interviewed at had me do this exercise at the end of the interview and to be frank, I bombed it quite badly. One issue was the syntax was so weird compared to the type of code I'm used to writing and reading from others in library code that it was hard for me to comprehend the logic without the benefit of looking it up.
I was filled with anxiety working under the time pressure to understand a document of code which I had very little context of, and no previous relationship with the reviewer so I kept thinking to myself whether or not the question that came into my mind was stupid obvious.
In contrast, when I had take-home tests, I scored exceptionally high on all of them, leading to offers. I recognize that a lot of people here hate them because of the time involved and it's akin to spec work, but IMO it felt much more like my usual day-to-day life as a developer.
One of the take home tests had existing code to work from but again, working at my own pace and having access to online resources to read docs was much less anxiety provoking.
These sound like very valid issues. It might not be good for every candidate, but having a conversation about the code seems less stressful to me than having the interviewer bring out a whiteboard and essentially say, "Please perform for me!"
You say:
> One issue was the syntax was so weird compared to the type of code I'm used to writing and reading from others in library code that it was hard for me to comprehend the logic without the benefit of looking it up.
From the point of view of an interviewer, that's feedback I'd be interested in. One problem I've seen in some of our code is an engineer using a completely valid but very obscure method of doing something, and then the rest of us having difficulty maintaining it. I would want to hear, "this looks hard to maintain." But I understand that in an interview, it can be difficult to assess what the interviewer wants to hear and that can be nerve-racking. I personally think take-homes are a reasonable compromise, but not everyone does.
You can work on that anxiety, just like you ace programming tests with enough time & effort. Therapy, especially group therapy might be really helpful.
Yes! I've been on the interviewee side of this a couple times and absolutely love it. It has so many nice properties:
1. It fosters meaningful conversations about code that is not dependent on the candidate solving a toy problem.
2. As you said, it tests a candidate's ability to read and understand code.
3. It's actually representative of the type of work you _will_ do at any given company, i.e. code review.
The major downside is if you give the candidate code in a language they're totally unfamiliar with then that candidate will likely bomb. I think this can be mitigated by sticking with mainstream languages like C/Java/Python, or domain specific languages where not-knowing would be a deal breaker anyway, i.e. must know Swift for iOS dev.
I do this as well. It's the best signal to noise ratio. Developers spend much more time reading code and understanding the system rather than writing code, yet interview questions don't acknowledge this.
It also works even if the person doesn't know the language because you'll get to see them mapping this new language to something they already know and it can give a good understanding of how well they think of things beyond just the code in front of them.
I have create a git repo specified for the question, with some code working in way, but buggy, such as logical bugs or race conditions...etc. with a lot of not business relevant assumptions. I usually told the candidate what this code supposed to do, and ask him to review the code first, then tell me what was his impression on the code base, and what were the problems, and how he can fix it. By doing this, I can judge the candidate if they are good at general coding, debugging and and comfortable to work on projects which they are new to, as well as their mindset on writing maintainable software.
A good candidate will not just fixing the obvious issue, but propose some architectural changes, then I will ask them what are the strategy to do so as the project has existing dependents, this can give me signal on how the candidate coordinate with other teams.
Then I will ask him some more questions related new features, and scaling..etc. to measure his design, communication skills and more.
It never failed me for senior level positions hiring, that either the candidates were hired and proven to be a great match, or they got a better offer elsewhere.
This is pretty similar to a ML Engineering interview question I got once. Basically, you get some mildly obfuscated code implementing a statistical concept and you have to tease out what the code is doing. Then you have to fix a few bugs, the variables names, and make it easier to read.
If I had to do interviews I'd do this and something complementary to this: bring a piece of code you're especially proud of. What problem were you trying to solve? How did you try to solve it? What options did you discard? Why did you discard those options? What resources did you use? How do these concepts/solutions relate to the CS theory you learned in school/your own?
I don't really have code I can show you. Most of my code I write is in my employer's VCS. I have some throwaway scrips on Github, but those not production grade.
LeetCode could be a good source for code for this.
Pick some interesting problems from there, and without asking the candidate to actually code a solution, discuss the problems with the candidate. Ask for ideas on how to approach the problem, and suggest some yourself if at any time they appear to be getting stuck.
After they, possibly with guidance from you, have reached an approach that if correctly implemented would solve the problem, including meeting time and space requirements, discuss what issues might arise in actually implementing it, such as edge cases that might require special handling. This would be a good place to talk about how they would go about testing an implementation.
Finally, go to the discussion on LeetCode for that problem. There you will find various solutions posted by others. What I have found looking at those is that a lot of them don't meet the time/space requirements, or miss edge cases, or sometimes are just wrong. Go over a few of these with the candidate and have them criticize them for you.
Anything leetcode related comes down to rote memorization. If LC is part of interviewing process you are hiring people who good at repeating things they’ve memorized, have a lot of stamina to spend hundreds of hours doing meaningless work and are very good at following orders. If that’s what you need than it’s a great filter.
You can save some effort though and just hire anyone who passed Chinese or Korean grueling school system with emphasis on standardized tests. LC-based interviews follow the same tradition.
It’s very company-dependent, but most are not wanting a regurgitated answer. If you have a solid grasp on data structures and traversals, you can solve a lot of those puzzle problems. Sure some have tricks, but they’re not completely pointless and certainly don’t indicate anything about the person being willing to “spend hundreds of hours doing meaningless work” or are “good at following orders”.
One strategy that's been successful for me is skipping the technical screen and jumping to a narrowly-scoped paid assignment. If they do well, I give them a larger paid assignment, and we work our way up to regular work together. If they do poorly, I pay them for their time but tell them it's not a match.
I'll add the caveat that I've only used this to hire junior to mid-level developers for part-time or short-term work. I've never tested it on a full-time hire, and I imagine that would be more difficult.
I've never written in detail about hiring developers, but I've shared similar strategies for hiring writers,[0] illustrators,[1] and editors.[2]
I do gig-to-hire too. Works a treat! I've found loads of talent this way, up and down the skill chain. But, for a CTO like role the gig has to be 3mo or more.
I'm confused. You are saying that you have had CTO interviewers that did a gig role for 3 months as a CTO? Did they also have another job at the time or was this their only job?
This one is a bit harder (naturally). A recent/typical case was a CTO type candidate, they participated part-time in some meetings, had some exposure to the team, talking through issues with other CxO. During this time candidate is also shopping around too. It's basically buying consulting while you decide to hire. It does not generally take 3mo - but you'll add a probationary period that gives you maybe 12mo room.
Thanks for the clarification! I had no idea C-suite hiring was at all like this. Naively, I figured there would be clean breaks and legal 'firewalls' about such dual responsibilities and obvious conflicts of interest.
It really explains a lot about the C-level people and their lack of engagement with work (for my company at least!); they don't seem to be doing their full jobs as, well, they aren't. I really need to figure out how to get on this carousel.
I realized I wasn't super clear, let me add more. The case is where the candidate is NOT currently on W2 wages anywhere else. Typically, these candidates have enough cash reserves to be able to take a few months to find the next gig. A little highriing bonus takes the edge off.
>One strategy that's been successful for me is skipping the technical screen and jumping to a narrowly-scoped paid assignment. If they do well, I give them a larger paid assignment, and we work our way up to regular work together. If they do poorly, I pay them for their time but tell them it's not a match.
Kudos to you. You have worked out one of the fairest ways to gauge someone. May your tribe increase!
This is replacing a 1-hour phone screen with a 6-hour project. You're still going to rule out a bunch of good people who will refuse to spend that much time on a first-round interview.
You don't simply fire off a 6-hour paid project to everyone who applies. You do some screening and conversation before-hand.
It always seems strange to me that people won't bat an eye at taking a day (or more) off work to come on-site for technical interviews, yet the idea of giving someone a 6-hour project do at home, at their leisure, is somehow a bridge too far.
In real-world hiring experience: I gave candidates a choice between on-site interviews and take-home projects for a while. Zero people chose on-site interviews when given the option, so I stopped offering it.
The bottom line is that interviewing requires time. If someone isn't willing to lift a finger or allocate a few hours to an interview, how are we supposed to gauge their competency? The "just trust me" model doesn't work at all.
> It always seems strange to me that people won't bat an eye at taking a day (or more) off work to come on-site for technical interviews, yet the idea of giving someone a 6-hour project do at home, at their leisure, is somehow a bridge too far.
One of the key differences is the company has "skin in the game" at an interview, but not in the 6-hour project.
When you interview, you know the company has included you in a shortlist and they will put time into getting to know you. For the time put in, you have a chance to pitch yourself, to at least get across your general abilities and personality, and some people will pay some attention. Also you can learn more about the company and the people you might work with.
When you get a take-home project, it's like the way spam is cheap: It takes almost no effort for the company to send the same project to a lot more candidates at low cost.
You do the work, but there's no guarantee the company will really assess it. Sometimes the company drops the ball and just "forgets" the application, or discards it the same way résumés are filtered: By skimming though 6-hour project submissions and casually throwing some of them away. It's not uncommon to not hear back anything from a company after sending the result of working on their project.
Some candidates will enjoy doing a project more than an interview, but that's not the only consideration in determining their value to the candidate.
Personal opinion: take home tests are way less work for the employer and than the candidate in an already asymmetric relationship (or can be).
Personal experience: I have never had an interview last more than an hour. I have also been hired twice on a "just trust me" model and it worked reasonably well.
Right, but it's a 1-hour unpaid phone screen vs. a a 6-hour paid project.
I can understand not wanting to do that if you're a full-time employee, but if you're a freelance developer actively looking for jobs, why would a paid project be worse than an unpaid phone screen?
They are getting paid, though, for the 6-hour project which seems like a positive over a 6 hour take home test (used by some companies) where you're not paid.
Wouldn’t this completely exclude anyone on a visa (since they need another visa to work for money) and also partially anyone with a lot of family obligations?
I'm not sure what you mean. If they're applying for paid developer positions, they would need a work visa anyway. I've only ever hired for remote freelancer positions, so I've never been involved with any of my contractors' visa details.
Why would this exclude people with family obligations? When I say "narrowly-acoped project," I mean 5-10 hours of work. I've never gotten an offer for a dev role where the application interview process took less than 5 hours, and those were unpaid hours.
Most US work visas are limited to a single specific employer. Some can be transferred to another employer, once that employer extends an offer - but this transfer has costs in processing time & fees.
In other words, some visa holders in the US are very much allowed to work, but not contract work.
Typically these are candidates, most of whom are already working right? So their visa would be tied to their current employer and they are not allowed to work for someone else.
5-10 hours would be similar to a typical interview loop yes, and if that was the entirety of it then it's not more time consuming than the typical engineering interview.
Hiring manager here. Top-tier candidates do this all the time.
The alternative is to have them take time off of their dayjob and travel on-site for technical interviews.
In the real world, candidates are greatly appreciative of take-home assignments that let them work in the comfort of their own home, on their own equipment, at times that work best for them.
I feel like this is the solution that values candidate time the most. They don't have to do an interview, and I'm paying them for their time immediately.
If you're "top-tier" and you want another gig because you're not happy with your current one doing a 6 hour paid gig vs. doing a full day interview somewhere (which isn't paid) actually seems like it would be a lot better. It's not like you don't lose a day (or days) when interviewing. No reason it would have to be a weekend day.
I'm a "top-tier" candidate (at least, tech companies offer me a lot of money to work for them) and I don't think I'm alone in not really caring much about the process, as long as it's sufficiently technical that I'm not worried about working with people who can't program. If the full time job at the other end is interesting enough, the 6 hours of whiteboard/takehome/code review is just an annoying cost of doing business.
And, if we're being really honest here, there's a ton of informal backchannel talking that goes on, so your interview "performance" regardless of medium is affected by that at many places. I think if you aren't actively working to make your interview process objective, it's being strongly affected by many biases and informal relationships.
That's all to say that on the balance, taking up "some" or "all" of a candidates time isn't that interesting of thing to think about because the process is so broken in so many other ways and this one doesn't have a huge impact.
Adding to what the other person said, I've leveraged interviews to get flights to cool places. A free day trip to Seattle, nyc, or sf is a nice perk, and in my experience companies are flexible on travel arrangements (eg. Can you put the return flight a few days later, I'll handle any extra hotel costs).
How about trusting candidates' experience? If they say they have 20 years of experience in X and you hire them, you'll know very quickly if that was a lie. Those tests are so insulting to the relationship you're pretending to want to build. Why would I bother putting false information on my resume that has the potential to get me hired and get tested daily? Isn't that why there's a probation period?!
I have 20 years of experience in X but prove it? Why's the burden on me to prove what I've already told you? Are you saying I'm a liar? If my date of birth is on my cv do you expect a birth certificate to validate that information too?
Anyway, that's always been my philosophy - hire on facts and fit, fire quickly if anything was wrong - and it's been pretty smooth. The great thing that it does is it prevents testing for futile thing that you will never use in your position and that are just elitist criteria or vanity wishes.
The reason this philosophy is misguided is not that candidates flat out lie all the time (though that happens too), but more that resumes are crafted to inflate, exaggerate and obfuscate to make the candidate seem more qualified than they might actually be for the position upon closer scrutiny. I say I have 20 years of experience in X, but 4 of those years were just hobby projects in high school, 4 studying in university, another 5 in entry-level positions in a semi-related industry, etc etc. It would be a huge waste of time in obviously unnecessary hirings+firings to blindly trust stated experience on a resume without verifying its accuracy first.
This is something that only makes sense after you've spent significant time on the hiring side of the table.
Engineers who are 100% honest, overly humble, and fortunate enough to have only worked at top-quality companies tend to not understand the need for verifying the skills that people put on their resume.
In the real world, some of the best candidates have the worst resumes. Some of the worst candidates have the best resumes. And perhaps most dangerously, some of the worst hires tend to be the best at lying and exaggerating because they've used their charisma to get them through life.
Making bad hires is a very costly mistake for the team. Not only do you lose out on months of team-building and project progress, but it's bad for morale to see people hired and fired quickly. In reality, bad hires are going to stick around for 6-12 months or more at any company of moderate size company.
Also, not all experience is created equal. It's shockingly common for applicants to claim 5+ years of experience in something, only to discover that they only touched that topic a couple times per year for 5 straight years, for example.
> If my date of birth is on my cv do you expect a birth certificate to validate that information too?
US hiring practices expect to see something that validates your birthdate, so kind of.
Fast firing can work, but none of the places I've worked were prepared to do it. Similar to what a sibling commenter posted, years of experience is something, but what you got out of it is another. There are plenty of people who have spent a lot of time doing something and not gotten much experience. In my mind, a good interview process figures out where the candidate actually is with regards to experience and capability, if they fit with the company, if the company fits with them, with as minimal a time commitment for both sides as possible.
It's not great for either party if someone is hired and fired in a month.
Yes, and the way that I know this is by performing the tests you're arguing against. I've interviewed plenty of people that could converse fluently in buzzwords, but could not write code. In the actual job, it's necessary to be able to write code.
You can also be straight up lying about having 20 years (or any) experience. Most people won't be doing this, but it seems like a basic feature of a hiring process to have some kind of check against this.
Also common is someone with 20 years at the same company that's has been coasting. Maybe they still work on a decades old enterprise app and have no relevant recent experience.
Yeah, but you know when you obtain the information - which would be reading it on their resume or talking through it with the candidate. Asking them to confirm what you've been told is my issue with this whole thing
Honest question: How do hospitals hire doctors (industry hire)?
Do they quiz them like they did to the residents on Scrubs? Do they pull up a list of symptoms and ask what they would test/try next? Do they show them x-rays and ask them to spot the coronavirus?
Seriously? Or do they look at your degree and experience and conclude that if you worked at General Hospital in Atlanta for 5 years you can work at Metro Hospital in Philadelphia?
I honestly don't know how other professional industries hire industry-hires (not college grads/interns) but I only hear about quizzes and riddles and online tests and shit like that in programming.
Hell, do they do the same shit for electrical engineers? Chemical engineers?
I really want to know if they do and if not, how is it Boeing can hire a EE without a whiteboard puzzle but can't hire a CE without that bullshit?
Medical doctors have a professional organization enforcing strict standards. People cannot practice medicine without passing a series of standardized exams that only recognized universities can administrator, they have and a lengthy internship process, review board etc. If they break the rules they can have their licence suspended.
I think it would be a shame to bring this sort of thing to software development, but it would allow us to get rid of some of the hoops people have to jump through to ensure they can actually do the job being hired for.
I posed the exact same question below. I have the exact same thoughts, and genuinely curious on the process they have.
I'm not that opposed in general to whiteboarding if you feel like you can't assess the candidate.
However I do feel that it sometimes feels like credential and experience similar to what you describe in the medical field: if a doctor had 5 years at an inner city hospital, I don't think they would doubt his credentials to the point the want to make him put cell structures on a whiteboard?
Doctors do have professional education, harsh exams, and they can get their licenses revoked for mistakes.
Do you really want the same for programmers? "only people with Ms or PhD from accredited universities can apply, and diploma is invalidated on some mistakes"
For an EE, it might be analyzing or designing a simple circuit. Or writing an state machine in Verilog or VHDL. Or demonstrating their understanding of a industry standard.
In some industries an engineer might need to be licensed. That requires passing multiple tough exams and working under another professional engineer to get their license.
Most of those industries have board certifications. An EE or architect may take months studying for a board certification exam.
Physicians are similar in that you have board certification and residency which provides a baseline level of implied compatency. As I understand it, your residency can essentially set your career. You won't be able to change speciality, and moving up in terms of institution is difficult.
In practice this might be "you can only get hired at faang with faang on your resume", so your options if you don't get it right out of college are to do masters or hope to get a job at Amazon and possibly pivot that into a position at a different firm.
I've done my good share of recruiting interviews and you can measure if the CV has been inflated just by having a conversation with the candidate. Just asking "What did you do in your last job?" and following from there works wonders. No coding exercises, no whiteboards, it's all about working a conversation with them. I can confidently say that the recruiting process was faster --less interviews--, less stressful for the candidate, and finally the number of hire employees that flopped was not that much worse.
I did whiteboards for Architect positions, but once again, I just asked the candidates to draw architectures to have a friendly discussion.
> I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job.
This is the fundamental part that is broken in my experience.
They are often challenges/coding puzzles, when rather white board exercises should focus on collaboration, and solving problems with code.
Having a "gotcha" that shows rudimentary CS understanding is fine (such as understanding why the obvious O(N^2) is a bad solution). But having a gotcha that requires some crazy leap of faith guess (or having seen the problem before) is not.
I admit, my fondness for a candidate increases with the level of complexity of coding problem they can solve; however needing hints on the harder problems doesn't mean they are not qualified.
I would highly doubt he would get rehired today at google first try. Leetcode is a different beast and things might not be all that fresh. Someone on the hiring committee might have said yes too many times this month.
Former Google engineer and technical interviewer here. Google doesn't use Leetcode for interviews (at least as of 2018).
I'd be extremely surprised if Jeff Dean (at any point in his employment) would fail a SWE interview loop at Google. The Google interview is heavily based on data structures and algorithms, and Jeff Dean likely knows them cold.
Sure there's always a degree of luck, and Google errs on the side of false negatives, but there are certain people that are so high above the hiring bar that luck becomes too tiny a factor to be relevant.
Yes, Google doesn't use Leetcode per se for interviews. What the parent commenter was referring to was that Google interview questions are of the type for which Leetcode has become famous for.
> The Google interview is heavily based on data structures and algorithms
That's pretty much what Leetcode is. When I interviewed at Google I was asked tree traversal and hashmap design questions that were very similar to leetcode questions.
In eastern europe interviews are mostly experience-related.
Questions are mostly about technologies that hiring team needs, the more the better, and the deeper the better. I never had a problem getting the extra experience not found on previous workplace from my pet projects and blogs before moving to US.
Funnily, I've seen people who moved back from SF Bay Area, and struggled to find a job there, despite being very good at whiteboard and CS problems. Turned out eastern european companies don't care about CS problems really.
> I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.
This is absolutely why whiteboarding exists, and persists.
I have interviewed a lot of people at this point in my career, and it's just not possible to tell if candidates are good by looking at their credentials or job histories.
I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions. I'm not talking about Leetcode-style brain teasers; I'm talking about being unable to construct a working for loop and do basic array operations in a language of their choice.
I'm going to continue to do whiteboarding for candidates for this reason.
> I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions.
That honestly sounds to me like you found some people who lied like crazy about their credentials and job histories.
I work somewhere in the middle of biology/neuroscience and ML. Interviews for essentially the same job (e.g., building brain-inspired robots) are hilariously different depending on how the org views itself.
In a "science" organization, the interviews mostly involved talking about what I've done in the past and how I'd approach new issues (technical and behavioral). There were certainly a few probing questions about things that seemed designed to check up on claims, but my experience seemed mostly taken at face value. A coding test, for example, was skipped because an interviewer had already seen some of my code and thought it was fine.
"Tech" interviewers seemed to want everything "proved" on-site during an all-day slog. I reversed strings and traversed graphs on a whiteboard. Someone quizzed me on facts and figures about the human eye (which they were reading out of the same book I learned them from). To me, this seems to test immediate recall and stamina/stress tolerance more than anything actually related to my R&D skills.
I think adversarial is the right word for the current style of tech interviewing. It wasn't always this way. Back in the 80s when I first got into tech it was a lot more conversational and in the spirit of "we really need help, what can you do for us?".
I think something more akin to a thesis defense would be better than what we're doing now. Prior to the interview have the candidate write up a description of a project they worked on and particularly liked along with a set of slides. Have the candidate send this presentation to the main interviewer who will distribute it to all of the interviewers. The interview then becomes a presentation where interviewers can ask questions about the presentation. I've seen this done in industry, but it was only when we interviewed folks who were PhDs coming from academia - I don't see why we couldn't do this for all kinds of candidates, not just those coming from academia.
I interview, on average, two people a month for non- or semi-technical roles. Everything from financial analysts to contracts specialists and project managers. Interviews are generally structured to give interviewers a good idea about four items:
1) What did you, personally, work on previously that's relevant to this role? This usually means pretty in-depth discussions about previous roles and projects, accomplishments, tasks, skills gained, etc.
2) How do you approach tasks and problems at work? This is usually a combination of asking about why the candidate did things a certain way previously and how they would approach theoretical situations in the future. The goal is to uncover the mindset the candidate uses to actually get stuff done in a changing business environment.
3) How do you interact with teammates and managers? This is the concrete version of "culture fit" and includes asking about various interpersonal situations likely to have been encountered in the past as well as discussions about how the candidate would handle quirks of the office environment at the new company (ex. some companies love people to make decisions quickly on their own while others hate it and mandate team consensus).
4) Do you have the base-level skills needed for the role? This is often far more than the technical aspect. After some basic questions to establish that the candidate knows the basic processes, software tools, artifacts, and terminology, it's much more important to establish if they have the chops to actually be successful rather than a "task completed" robot. Can the candidate communicate ideas and opinions clearly? Did they previously work in a collaborative way rather than an adversarial way with teammates? Can they present findings and issues to management effectively? Have they previously worked with supporting teams and vendors outside their main role to solve problems? Etc.
In most jobs the technical skill part is a small piece of what the person will actually be doing so the interview is a means to collaboratively establish if the role is a fit for the candidate as a whole and vice versa. If I only looked at technical skill and immediately disqualified people for failing concocted and unrealistic tests, I'd end up with mostly shitty candidates who aren't actually very good at the wider job.
This is not a rhetorical question: How does other industries hire high knowledge well educated candidates ? How does doctors, mechanical engineers and space flight engineers get interviewed and assessed ?
As have already been reiterated in each and everyone of these threads (about leetcode/hackerrank/whiteboard coding) this is what the companies seem to think is the most proven, and produces most consistent results. But I would love to understand how other industries does this as well; does a propulsion engineer start to draw up a de Laval nozzle and calculate the exhaust gas velocity on a whiteboard? (again genuine question)
The problem is that the software industry does not have the same standards as other industries. People graduate CS programs with little to no coding ability in many cases. There is no certification program like there is for, say, accountants or CPAs. Companies set the hiring bar all over the place. At a lot of companies, bad hires stick around for years.
Other industries have higher and more consistent standards, and certification processes. The software industry is the wild west, and that is partly why it has succeeded and why so many self-taught programmers have found success in it. The other side of the coin is that the quality of candidates varies more widely than in other industries.
I've never seen someone with a CS degree and no coding ability. Lots of them weren't taught anything else about software engineering. But most interviews don't test anything else anyway.
Depends on the company.
Google can afford to reject great candidates. A small company might have to settle for a mediocre candidate 6 months after rejecting a good candidate.
Imagine you're a team of 2 top performers. Your workload is increasing, so your manager gets approval to hire a 3rd person so you can take on more work. Unfortunately, your manager is fooled by a charismatic person who greatly exaggerated their experience on their resume, to the point that they work about 1/4 as fast as the rest of the team and tend to code by trial-and-error.
Suddenly, the expectations for your team's output have increased by 50%, but the 3rd person can only really contribute another 10-20% at most. By extension, your job just got harder.
Worse yet, the average talent of the team has now gone down significantly. You spend more time fixing bugs and chasing bad work. You're also faced with the fact that you're probably getting paid the same as the over-leveled person next to you. This breeds resentment, wastes time, and ultimately leads to good people quitting for other jobs.
Alternatively, imagine if the company hired quickly and fired quickly. Every time a new person showed up, you had no idea if they were actually good or if they were just rushed through the interview process. You also don't know if they're going to be around after the 3 month probationary period, so why bother building a relationship? High turnover like this tends to destroy morale.
That looks good in a conversation but under scrutiny, unless you know someone in a company mentioned, all you get is an opinion or you need to do a deep economic check of the company in question, which is nothing more than a statistic at best.
Why would you need to put so much judgement into a single company. If the person has only worked for one company than their employment history won't be much of a factor.
If someone has worked at 6-8 places over 15/20 years covering a range of skills required I don't see the risk.
You tell them what the job is like, what is expected of them, and what the best way to work with you is, then you you ask if that's what they want, and how much it's going to take to get them in the door. That's it.
I wouldn't ask an accountant what the Corrodium 3 Deduction is; I'd tell them I need them to file my taxes, that I need them to own the process and chase whoever they need (even if that's me).
> I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.
I think you're probably right, but if someone lies to you and says they can do a job that they clearly can't, you should fire them.
This should be unusual because getting a new job can be stressful, and nobody wants to put themselves into a job they can't do, so it's rare that you even have to consider this -- but it does happen, and unless you're a sociopath, it's going to make you shy about ever having to go through it again, because you're not going to want to fire anybody. It sucks for you, and it sucks for them, and it sucks for the team who see the revolving door, but nobody benefits from the wrong fit in the long run, so you just need to suck it up and do it.
So be firm, and if it turns out you're firing a lot of people straight in the door, look inwards: you're probably not being honest with them (or perhaps yourself) what the job actually is. Figuring that out instead of some new programming puzzle to torture grads with is going to pay dividends.
It's Reid Hoffman's idea, not mine—a referral from a trusted source is a better indicator than an interview.
He's previously said that if he could only decide based on one or the other, he'd hire the candidate referred by a trusted source and skip the interview. In practice, this choice never comes up, but you can weight referrals much more strongly than interviews.
I concur with you. This debate rages on endlessly. In-person whiteboarding is the worst way to assess candidates' tech abilities... except for all of the other alternatives.
How is this a job board? There is effectively no information about the position you would be applying to. The only offer is asking for “A typescript developer with experience using React, Apollo, Node, GraphQL”. This tells me nothing about the position. How much experience do I need? How much do you pay? What other benefits are available? Where is your office located / what’s your stance towards remote work (and what will it be post-covid)? What will my typical day look like? Do you use source control and CI/CD? Do I get a company laptop or do you expect me to bring me own? I need to know answers to questions like those before I even think about sending a job application.
Side note: Why is everyone into job boards in 2020/2021? Was there some event or article to push this as an untapped market? Seems like so many people recently came up with the ever same variation of job listing services.
> Why is everyone into job boards in 2020/2021? Was there some event or article to push this as an untapped market?
There is/was a global pandemic making a lot of people unemployed.
If you're a coder and unemployed, what better way to get a job than make a job board? Potential employers will come to you and tell you about their jobs, and then you have a great leg up by saying, "I built the job board you posted to".
> Potential employers will come to you and tell you about their jobs, and then you have a great leg up by saying, "I built the job board you posted to".
I recall Google rejecting Max Howell despite creating Homebrew for not having as solid of a comp-sci knowledge as they were looking for. Despite flaws of the product that he'd be the first to acknowledge, creating it should've been evidence that he has strong potential to grow into a person that could benefit the company.
And I guess I'm not surprised. Some of the interview questions I've seen people post online from their experiences with FAANG are so detached from the code programmers write on a day-to-day basis.
A lot of college grads go into the recruiting industry as a means to get entry level tech sales experience, a lot of people are also unemployed and thinking about unemployment solutions, finally job boards are a good means to collect data that is interesting: the resumes of everyone.
Job boards are a major part of the zeitgeist of collapsing middle class America.
Why now? And I am seeing this in Germany as well, which of course suffers a social divide, but much less extreme, or existentially threatening, than in the US. I assume the "job exchange" market is a natural monopoly, at least for its respective sections, and now there seems to be a race for it. But weirdly it's for the most part all the same: Listings. There doesn't seem to be a sustainable model, which avoids the problems of the past's iterations. Therefore I assume it's driven by business folks, not people with ideas.
I don't know, but apparently they weren't perfect or otherwise there wouldn't be such a rush for it, no?
I guess it's still a tedious experience for all parties involved.
I wonder why there isn't an OkCupid for jobs, but I think the problem is an asymmetrical interest in a "good match", unlike dating. As usual people fail to come up with good metrics and incentives and it's Goodhart's law all the way. Some problems just can't be solved by technology. But that never stopped anyone from trying over and over again... hyping their overfit tech to non-tech people.
I think the actual problem is with management on the employer side, not the job boards. They need to bite the bullet and accept they need real human interactions to measure productivity and reward it case to case.
Job board industry veteran here. Job boards are pretty easy to get started. You can write a nice job board in a weekend so a lot of boards start when a developer doesn't like the job search experience. This seems to be an example of that. That said, getting traction and making money can be hard because it is very competitive with Indeed, Facebook, LinkedIn, and Google gobbling up traffic.
That said, there are lots of good sources for backfill (job posts) and there are still lots of really great ways to monetize users and applicants via direct sales to employers and pay per click or pay per application programmatic networks.
The tricky thing here seems like customer acquisition of a two-sided marketplace: you need job searchers to get job posters, but you need job posts to get job searchers.
Job board industry veteran here. You are right it's hard to scale up a job board because of the two sided market. Once you get to a certain size, you are suddenly competing with Indeed, Facebook, Google and LinkedIn (and all the other job boards). Getting started is pretty easy right now because the employer side is very much into pay-per-click or pay-per-apply syndication and there is high demand for jobs on the job seeker side of the market. The key is to have a source of traffic that you can convert into click and applications. You'll see all kinds of niche job boards spring up right now fueled by traffic from communities like this, product hunt and even appropriated email lists that people take when they are laid off (yes, that's probably unethical, but lots of job boards get their start when a developer or recruiter starts one up in anger).
This is a very unfortunate name indeed. BPD is a serious personality disorder that causes a great deal of suffering, not only in those affected by it but especially their families and loved ones.
To the senior engineers who are already happily employed and hate leetcode interviews: An effective way to fight against this is to start saying no to these interviews. Senior engineers have the power to change this if it becomes a trend to say no to these interviews. Anytime a recruiter contacts you on Linkedin, ask them does your interview process include leetcode style whiteboard interviews? If they say yes, then politely decline and make it clear that's the reason for the decline.
All the leverage is on the happily employed senior engineer side. If they're not happy or unemployed, then it's different. But for the happily employed, it's the recruiter/company trying to convince them to leave. And it's very reasonable to reject these interviews. An employed senior engineer already has a demanding job that requires at least 40 hours a week. And since they're older, they're likely to have other responsibilities like family. There's simply no time to grind for months on pointless leetcode problems.
If more senior engineers start immediately rejecting any recruiter messages with a simple "No, I don't do leetcode", the signal will be sent. Recruiters will start getting frustrated and tell hiring managers, hey many of the great engineers I contact keep saying no because of this leetcode thing, can you change that part of the interview? I can't find any good candidates because of this. That will force hiring managers to really evaluate if this leetcode interview is a good idea.
Don't be fooled if they say it's not about getting the problem right or they just want to see how you think about a problem. That was somewhat true a long long time ago when these types of interviews started and leetcode didn't even exist. And it's what they still say to make it seem reasonable. But the reality today is that the interviewer only cares that you can write out the optimal solution within the interview time. That's it, that's all the interviewer will care about. If you don't have the right answer, it's an immediate zero. They might care about other factors (like are you friendly, code quality, etc.) after you successfully write the optimal answer on the board. But writing out the optimal answer is the baseline, they don't care to evaluate any other factor until that baseline is satisfied.
There might be a few interviewers who still try to stick to the old way of these interviews, the original intent. But statistically the odds are against you. These interviewers are generally older and becoming fewer and fewer. The new generation of junior engineers are becoming senior and they only know the leetcode grind. In a typical interview loop, it will become more and more probable that there's more and more of these interviewers who expect the candidates to have done the leetcode grind and base their evaluation from that assumption.
I can only speak for myself but as far as the first two the problem is that given the pressure and time constraints your success is generally dependent upon whether or not you’ve churned through similar enough problems in preparation so it’s not as much a test of your coding ability as simply pattern matching ability, which is important on some level, but not nearly as useful as, say, being able to research a solution to a novel problem and having a general sense of where to start. But for some reason we’ve equated the ability to do leet code problems with the ability to do the latter and now I’ve got to spend a significant amount of time churning through a bunch of coding exercises to prove myself instead of proving myself by doing real projects and being able to speak about them. As for take-homes, my main problem with them, other than the risk that you’re doing unpaid work, is that you often just submit your solution to the void without any chance to discuss your solution and the trade-offs you made, improvements you would implement given more time, etc. so you can imagine it quickly becomes an onerous time drain that doesn’t seem like a fair trade of time on your part if every company asks for one.
Personally, I don't mind take home assignments or even live coding challenges as long as they are testing skills I use for my job (front end web dev). I've spent a lot of time getting good at what I do, and I am actually happy to show those skills off to potential employers.
But hackerrank/leetcode don't test for those skills. They test your ability to memorize solutions to trivia questions, typically things you'd never be asked to do in the workplace.
There's enough to learn to be good at software without having to also memorize arbitrary trivia.
>But hackerrank/leetcode don't test for those skills. They test your ability to memorize solutions to trivia questions, typically things you'd never be asked to do in the workplace.
That's not really true. Most of them are variations on a relatively small set of techniques. Dynamic programming, back tracking, tree traversal, sliding window, greedy algorithms, and minimal connected tree will solve 90% of them. And those are all techniques that are occasionally useful.
Sure, some of them will have some esoteric math or algorithm that makes the solution trivial if you know it. Or sometimes there will be some confusing wording. But so long as they're not like that it's a pretty fair test.
> And those are all techniques that are occasionally useful.
Shouldn't we be testing for skills that you will be using 90% of the time in your day job rather than skills that are occasionally useful (and thus could be looked up / learnt on demand if necessary).
I love algorithms, but in all my years of coding I've never had to use DP on the job. IMO, it's not useless knowledge or never used, but it's not something I think most developers are going to use at their companies to accomplish their tasks so it feels like a weird thing to test and dismiss a candidate for.
I know developers who have written books on subjects related to development and are 100x the developer I am, but have no knowledge of DP beyond what was taught to them decades ago because they never used it when helping to build systems used by millions of customers.
HackerRank is a platform. You can design and implement your own exercises on HackerRank. We do this at our company; we design real-world problems and even test our own engineers to make sure “we” can pass our own evaluations.
I understand that at many companies, they just use pre-canned questions and all the above criticisms apply. But I don’t think this is the fault of the platform; it’s a conscious choice of the company to do so.
> Why is there so much aversion to proving one's abilities? Programmers today feel they shouldn’t have to do whiteboards, or hackerrank, or take-homes.
Because it doesn't necessarily prove one's abilities. Each of those items have pretty obvious flaws:
- Whiteboard: Usually idealized problems requiring specific knowledge that is easily looked up, but not necessarily in one's short-term memory. Whiteboard is generally not the tool one uses to write code. It's very easy to bomb these sessions despite being competent.
- Hackerrank: Easy to cheat. Usually toy problems with idealized properties, i.e. they don't have the dirty edge cases you see in real world development. This also tends to be just a first-pass filter for getting an phone interview or maybe on-site.
- Take Home: Easy to cheat. May run afoul of existing employment agreements. Requires a pretty large up-front time commitment, and IME does not actually avoid the on-site whiteboarding sessions anyway. It's more of a first-pass filter.
> How does a prospective employer evaluate someone’s problem-solving and programming chops?
That's the million dollar question. As an industry we just don't have a good system for identifying competent developers. My personal approach is to stick with folks I've already worked with, who have a long track record of delivering. That doesn't scale at all and is not very good for folks just entering the industry.
I truly wish we had better answers and I support anyone trying to build something that explores the problem space. It's nice to have diversity in hiring practices.
Cheating on the take-home assignment is easy to fix: have an interview after the assignment is turned in, and ask the candidate to walk through their code, explain their choices and design decisions. (The task should have a few places where making a decision is necessary, eg. whether to use a relational or a NoSQL database.) This can expose cheaters that have no idea about their choices, and help the interviewer understand or accept some decisions they might dislike or disagree with.
My problem with the whiteboards and hackerrank tests is that it is an entirely different ability.
Writing algos from memory on a whiteboard is not what I do in my job. It isn’t a subset of what I do in my job. It isn’t something I have ever done in a job at all. You may as well be giving me a first year chemistry final.
So it is just something I drill, forget, and then re-drill when I want a new job.
In my experience as a hiring manager, Hackerrank is useless now. There are too many cram schools which teach you how to pass a hacker rank test and many junior developers will beat the crap out of seniors just because they know how to game the system. It ends up optimising your hiring pipeline for people who have time to cram.
The same problem happens with take home tests. It pretty much optimises for people with more time on their hands (i.e. people who don't currently have a job, people without kids etc).
In both cases a lot of good people will be filtered out. Some people think it's good to avoid bad hires at all costs but I don't think wasting people's time is particularly ethical.
because doing b-tree traversal from memory on paper correlates only so much with building successful business applications.
there's market leaders breaking new grounds every day, and for those I'd say it might be appropriate.
then there's your average bookkeeping application that is at most transforming existing data based on problem already solved by the customers, and these are like the majority of jobs on the market, but still pretend to hire the best 10th percentile - except they don't provide a 10th percentile pay and the job is boring and soulless so they probably won't even have a 10th candidate apply to begin with
> How does a prospective employer evaluate someone’s problem-solving and programming chops?
how do factory evaluate the capacity to tighten bolts?
this problem was solved by Ford a century ago.
if one's building run of the mill business software with spaceship grade engineering, you're trying to solve a hiring problem you shouldn't be having to begin with.
I don't care about the tests, but I do care about my time. That's why I'm not taking anything home. They either take the time to look at my GitHub / StackOverflow / Whatever profile or make me do some test, but I'm not spending more time on these things than what's absolutely necessary.
If we had an agreed upon credentialing system with a unified standardized test series, under a non-profit consortium, like engineers, doctors, lawyers, actuaries, etc. we might get somewhere. But that's a huge hill to climb, and it's debatable what if any technology specific content should be included.
I don't think most jobs actually depend on formal testing: there are a lot of people doing science and engineering work that aren't duly-accredited "Professional Engineers" and, of course, there are many other fields that also don't have accreditation.
If you're 40+ and have dozens of large projects delivered in the real world for both commercial and government clients, Hackerrank and its ilk can still make you look like a beginner just because you couldn't solve one of their toy problems in the ridiculously short time available. Additionally I get annoyed when some toy SMS message problem barfed up by an online coding tester has an incorrect solution when I have a correct one.
I'm slow and methodical. I'm proud of it. My code runs for years in demanding environments without breaking and I'm famous for it, at least within my own network. My code is boring. The tech I choose is boring. The solutions I deliver are predictable.
Hiring an experienced engineer is not difficult. Can the person do the work? Can we see some examples? Will they get on with the team? How quickly can they get up to speed? Will they deliver value to you?
>If you're 40+ and have dozens of large projects delivered in the real world for both commercial and government clients, Hackerrank and its ilk can still make you look like a beginner just because you couldn't solve one of their toy problems in the ridiculously short time available.
Is this a flaw or a feature? What I'm really asking is: is it a coincidence that some of these tests may tend to filter out an older demographic? It may be yet another one of those ways that some people can be filed away as not being a "culture fit" or whatnot.
If you're hiring a senior engineer, it doesn't really make sense to filter out the older demographic. Do you expect a 25 or 30 year old to have 15 years experience in a work environment and 10 years in using XYZ stack/language/tech?
A lot of companies don't care if you have x years of experience. They hire based on what they think is talent, and then train them. Even for senior-level positions, they may only care that the person is senior, not necessarily, "has x years in language x".
To be sure, there are companies that do care about domain experience - and I personally value it as well. But that experience often carries a price that some companies will jump through hoops in order to avoid paying. I've seen that first hand, unfortunately.
The problem here is that for lots of big projects, there are typically only so many people who really drive the project to the completion. If you work for a big bank, you'd actually have a couple of people like you and a dozen of bystanders who lend a hand here or there, but are barely competent. They will have exactly the same resume as you, and it might even look more impressive. But their 20 years of experience in development is really just a couple of years played over 10 times.
So when hiring, I don't care if you're slow and methodical, or super fast. What I care about is that you understand basic algorithms and data structure (lists, sets, maps, trees) and can apply them to simple problems without me telling you that using a hash map here will speed up the solution tremendously, or using a heap will simplify the code. Those simple problems are also taken from most common patterns we see in our codebase, which consists of lots of geospatial data processing and analysis.
And if you need more than an hour to solve a couple of those simple problems (let's say, find duplicates between two lists), you'll definitely have a problem fitting in with the rest of the team - since everyone on the team has solved those problems and agrees that we need people with at least this basic algorithm knowledge to be able to make a significant contribution here.
These questions also a first hour of the interview, the rest goes deeply into discussion of previous projects you made, different architectural patterns, etc.
>And if you need more than an hour to solve a couple of those simple problems (let's say, find duplicates between two lists), you'll definitely have a problem fitting in with the rest of the team
result = set(l1) - set(l2)
Time taken: 20 seconds. I take your point that you need people with at least a certain level of competence. I found I saved those who were hiring an awful lot of time by sending them example code or asking them to give me a challenging weekend project as a test, perhaps even something they needed doing for real. But I dread these exam-like online tests where I have to have a right answer in 20 minutes. I don't work like that. Few people over 40 do any more.
If you're talking about python, I believe you - where you should have &.
And there are many developer position applicants that wouldn't be able to do it. If you actually solve it in 20 seconds, I presume you passed the test.
Tell the interviewer that if you work on the L33T code problem, that you will bill them $250/hour, and 8 hours minimum. Since that is your standard consulting rate.
This will make obviously make them balk. But likely the company isn’t worth working for.
Provide work samples, and discuss them. If the interviewer cannot put in the time to investigate your work product, then they’re just being lazy, or they’re really just stupid. And they need to get off their high horse.
Genuine question here: what's the better way?
I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job.
On the other hand, take home challenges eat up a lot of time and credentialist filters like going to a name brand school (or even having the right major) filter me out.
My go-to for when I'm in a hiring position again would be recommendations from my personal network, but that has its own flaws and doesn't scale very well.
I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.