In this case, Bezos saw firsthand how a brain teaser could be answered by "seen something similar once before" much more easily than "worked on it on a team for several hours". Isn't this exactly the sort of brain teaser that we hate seeing in tech interviews?
This one was bad enough that Bezos left the field, yet Amazon (like every other software company) still asks these style of questions of its engineering candidates.
I love the Feynman Algo. I hold over 60 patents (not bragging) often times colleagues will ask me if I can teach them to be innovative. After being asked this question a number of times and giving it a lot of thought, the answer I give now is the following:
"No, no one can teach you to be innovative or creative, you already are. Humans are the most innovative creatures in the history of this planet, innately, look at all we have created. All you need to do is to un-learn to not be creative. We are all constantly taught, either in school, the military, by our family or by our peers, our company we work for, whatever... to NOT be creative; to in effect comport. First un-learn that. It will unleash your creative/innovative abilities."
Jeff Bezos may not be a physicist but somewhere along his path he learned to stop NOT being innovative.
The Feynman Algo relied on him putting a lot of work in. His wife's divorce complaint:
>He begins working calculus problems in his head as soon as he awakens. He did calculus while driving in his car, while sitting in the living room, and while lying in bed at night.
This is not that atypical if you are a physicist, mathematician, or a developer on a non-rote job. Not the wife complaining part, but the 'keep thinking part'.
I think what Bezos says in the interview is funny, but also very directly clarifies further what you're saying. The smartest guy at Princeton solved this problem ...
Not because he was actually smart (ok, perhaps a good memory). Not because he was able to do math really fast, or really accurate, or in his head. None of that. NOT EVEN because he figured out this problem on the spot. That was quite simply not it.
But because he had solved a metric fuckton of problems before, and then got lucky.
I agree with your statement (just got one patent to my name though), but a very important caveat is that you have to learn something (and preferably more than 1 thing) thoroughly before unlearning it. And learning first and then unlearning is vastly different from not learning it at all.
The guy didn't get lucky, the guy has experience and has worked hard. The same thing happens in software, the developer with extensive experience across many areas, that has studied tons of different things is more likely to solve a novel problem than the developer that is one dimensional and master of one tool.
I often see people say things such as, "I have tons of experience with language Y and framework Z and that's all that matters, who cares about trees, graphs and all that ridiculous interview questions..." The big tech companies are looking to innovate and lead, and that's why they ask these questions.
If someone is new to programming and begins by telling me what framework they are considering learning, I often tell them to forget about the framework and learn the language. The language is the foundation that frameworks are built on. Algorithms & Data structures are the framework on which software is built on.
The only time I frown upon such questions is if someone is testing you for exact syntax of the language instead of making sure you understand the nature and solution of the problem.
You pretty much argued the opposite of your point.
Asking algo and whiteboard questions completely ignores the "extensive experience across many years". I can spend years gaining deep expertise in an area, and that means absolutely nothing. All that matters is I practiced for 2 months white boarding algo questions under time pressure.
Or to put it another way. Spent 5 years getting a PhD? Who gives a f*ck. Can you code a DP problem in 20 minutes? You're hired!
I am sorry but this is insulting. You should practice what makes you better at your job, not jump through hoops for someone else’s entertainment. So yes, some of us, are logical enough to not join the hazing ritual. Just cause you want to do it doesn’t mean others should.
Just think of it as part of the job. The trade off is minimal, while the benefit is clear. If you still don't want to do it then its cool, less competition for those who want.
My point was that the interview process is flawed. It's really just a proxy for an IQ test (a gameable one). I think for a company like Google, it is somewhat understandable - they have very mature processes, strict coding guidelines, and overall very little choice in the tech you use. They are hiring for a "smart cog" rather than "innovative/experienced engineer".
I never thought about algorithms this way. However I would argue it’s much better to study functional patterns and understand why those patterns are useful.
Architecture is also another part that most programmers fail to grasp.
I think of algorithms as good to know but rarely used. It’s has an answer to every question. So what you get is a bunch of people cramming questions and answers until they memorise it; defeating the entire point.
> All you need to do is to un-learn to not be creative
Can you please elaborate on this? What exactly are we trying to unlearn. I have never been able to think creatively and this has cause me self respect issues.
I don't have any patents, but I believe the key is to continue to look at the world like a child would.
Be curious and playful (in the domain you want to create in), experiment. Try to figure out things on your own before reading the manual or other learning materials; then learn the existing theory and compare notes. Try to formulate useful heuristics for yourself. Do not reject learning or experimenting with things that seem only tangential to what you are doing.
Innovation is hard. In addition, you have to balance all this against "productivity", such as being focused on a specific problem, getting things done and finishing up.
I’m not the GP, but I feel like “unlearning” often relates to the need to feel certain.
When looking for a solution, is the path straight? Or is it curvy, full of alternatives that appear strange at first? Staying with the strangeness is part of the unlearning. :)
Take a look at these cards[0] for their fun (and creative!) prompts to look at problems (and the world?) differently. To look in different places, in different ways. To juxtapose and separate.
Take a look at some books (typically grouped under “creativity” on amazon), such as “The Artist’s Way”, or “Steal like an Artist”. “Bird by Bird” is another book I’ve enjoyed.
Some things/tricks/ideas/perspectives will click while others won’t. It’s ok to pick a path. Some tricks will work best at one time or another. That’s also fine — part of the unlearning is accepting that too.
I quite like that response! Goes right to the root of the cause :)
I can somewhat empathize (as I'm sure many others here can), as I often get similar requests from people I interact with on how programming/graphic-design is 'incomprehensible' to them and they 'wish they could do it'. I used to try to offer more practical advice — along the lines of 'Try X online coding resource and see if you like it', but have since defaulted to 'If you really want to do it, then you'll find a way of doing it (and I can help you in any way I can, learning-resource-wise).' which I of course sugar-coat to avoid coming off as condescending (as I obviously do not intend to come across as).
The sentiment that so many people seem to wish they were creative / innovative / more 'logical' is something that I grapple with quite a lot — wondering what levels of 'activation energy' some people need in order to realize they're capable of being creative, and then become self-motivated enough to continue. Like every discipline, it takes years of effort to become good at something, but hopefully only a short fraction of that to 'get into it'.
I think I'll try something similar to your line in future in order to come off as a little more inspiring...
They wish they were a creative person, they don't necessarily wish to do anything creative, it's like with instruments, I'd love to be the kind of person who plays guitar but actually sitting down with a guitar and trying to learn is painful and awkward where programming wasn't. Essentially because there were things I wanted to do that involved programming on a level I could do, writing fizzbuzz was fun, but there's nothing really fun or rewarding to me about playing a 2 chord song while I get used to my fingers hurting, I know it _could_ feel that way but it just doesn't.
I learnt to juggle really quite well in 2016, because I went to a hall once a week with some friends I really liked and practised while we talked, and dropping the balls and constantly fucking up could be fun because we were bonding over it, I don't have something like that for guitar.
One of my favorite talks is from Ken Robinson on how education kills creativity [0]. It gets relevant for this discussion from 7m30s on divergent thinking.
LOL, yes, so much fun ;) But meh, patents smachents, the important thing, in my view, is that whatever is keeping you from being creative, find a way to break out. For me, I grew up on a farm in a rural area and it ways moving to San Fran then Seattle and being exposed to all kinds of different and new things.
Funny story, my Mom (who was amazing) came to visit me and I took her to a cool food festival. She commented (and I will never forget this) "Wow there are a lot of different kinds of people here" for my Mom seeing so many people of different colors and forms was mind blowing, but bless her heart, she just sat back and took it all in.
Keep in mind, she was a second generation German immigrant. Anyhow, she saw this amazing thing and immediately understood what I was experiencing. So, for each person, I think, there is something different that will click you out of the trench you might be in and let you see something different. Not sure how to explain it any better.
Honestly I don't understand this mass hysteria about programming questions. If you go over any algorithms textbook you'll see all the algorithms that will ever be asked by any company in the world, ever. One time I was interviewing for Google, in the very last phase, and apparently this was one of the hardest question they ask. The question was just asking me to implement MaxFlow algorithm (of course not directly). I don't think there is any textbook in the world that won't go over maximum flow. Besides, usually they don't need you to solve the problem in entirety but to show them you can make some progress on the problem.
You won’t pass HC if you don’t at minimum solve the problem at all. And max flow is a nontrivial algorithm, more so than any of the ones I’ve ever known people to get.
You have to have pretty amazing natural talent to be able to get it cold without much studying.
I think it’s a shame that there doesn’t seem to be a middle ground opinion between the elitist “you mean you don’t know about Ford Fulkerson without studying?” and the anti intellectual “all coding interviews are just luck and not a good signal”...
Boom, well said. What I have seen is that most of these cries are from the majority without CS background or those that didn't take their schooling serious. I have seen enough people without CS background who don't care about programming questions because they have taken the time to study their algorithm & data structures. As much crap as the FANG get for their interviews, there's a reason they are the top companies. What they're doing is obviously working.
I wonder if you realize that in many countries, computer science was a very obscure academic path until very recently. Most schools in France wouldn't offer it until very late for example.
The older, more experienced people at these companies you cite are the ones less likely to have CS degrees, which only became a staple recently. The companies now ask for one because it's an easy mass fizzbuzz filter. It doesn't help them get the best engineers, it helps them not get the worst ones. They have the volume to make up for it.
Saying what they're doing shows ignorance as to why it's working.
Sure, there may be a correlation between a candidate's ability to answer every kind of algorithm & data structure problem across multiple respected CS textbooks and their performance at any one of these companies, but what of the many other confounding variables? What of the benefits, day-to-day perks, prestige, future open doors, workplace culture, or scale of the products they launch? Do those kinds of variables not also greatly impact the ability to attract and select top talent?
I imagine it selects for people who want the job at a FAANG company so badly that they spend a lot of time reading algo textbooks and grinding Leetcode. I guess this is also a proxy for other character traits, like grit, pattern recognition, learning from mistakes, and so on.
Nope. The main determinants of effectiveness are 2 things:
1) general intelligence
2) conscientiousness (how much effort you put into what you do)
Don't you think asking applied textbook problems is an excellent way to test for both at the same time ?
If you are smart, then what is the problem with going through an algorithm textbook, listening to Youtube explanations when you can't follow and just spending your evenings for 3 weeks or so. Frankly, if you get to chapter 5 out of 20, you've probably seen what you will be asked, so you could easily stop after one week and still easily pass the interview bar.
And if that's too much effort, why should they assume you'll put in effort once you get hired ? They'll be expecting more than one week's worth of effort from you.
Furthermore, Google actually tracks how well it's questions work, how the resulting hires perform. If there really was no correlation there, they'd have changed it (also they've documented it in Laszlo's book, well at this point books)
I bet it would measure even better if they told you a week beforehand that the question would involve some obscure, not super easy but not super difficult algorithm, but one that they can reasonably assume you've never seen before and then asked you to apply it on the interview.
conscientiousness is more complex than that. You have to understand what positive benefits manifest in what scenario for every given level of conscientiousness.
I'm very low-c and that means I have a proclivity to procrastinate and a tendency to not be able to stick with things very long term of my own volition. This doesn't make things impossible to do for me as I even managed to become fluent in Japanese which was a tortuous multi-year effort. Suffice to say higher conscientious would be nice, but it means I have a tendency to also not feel the need to follow all the rules all of the time and this results in my sometimes taking a contrarian or unorthodox approach and sometimes this has a massive and positive benefit. It also means I'm constantly forgetting to log my time sheets and take out the rubbish and things like that piss off my boss and my wife and this is generally considered a problem but my boss and my wife also love me because I do things they don't see other people doing and im considered unconventional but in a very good way.
Everything is a trade-off. It's not like all high conscientiousness people is the right mix of people to have.
They're looking for a repeatable, fair baseline cut for basic competency and motivation, and "has this person ever read a compsci textbook" is a fair proxy for that.
> This one was bad enough that Bezos left the field, yet Amazon (like every other software company) still asks these style of questions of its engineering candidates.
Maybe of their engineers and mathematicians, but as a software engineer, Amazon never asked me these types of questions. Software engineering is significantly different, however, so I wouldn't be surprised that they focused more on textbook fundamentals.
As an interviewer at Amazon, the brain teaser style questions aren't loved or taken very seriously.
There's always some of that, sure, but usually it's a problem with a bunch of ways to solve it, not just one. Some are better than others. And it's nice if you get the best one. But that's not what I personally care much about- it's how you communicate it, how your pseudocode looks, how well you've thought through the problem, how well you can explain the solution.
There's also usually interviews focused on design. The problem may be easy, but the challenge is in how you'll solve it to handle (m|b|tr)illions of inputs.
> This one was bad enough that Bezos left the field, yet Amazon (like every other software company) still asks these style of questions of its engineering candidates.
FWIW Amazon does experiment with different interview formats. They used to (still do?) interviews where you get a laptop and spend the day working on a simple problem where interviewers will ask you questions about why you did the things you did.
Not the same thing, the point is that you have had enough exposure to different things that you can solve problems. The reason software company asks those questions is that if you are familiar with your data structures (lists, stack, queues, trees) your algorithms (sorting, searching) you can pretty much give a reasonable answer to most of those questions.
> They're not brain teasers, they are real life problems that can worked through iteratively, with some hints and guidance from the interviewer.
I spilled my tea from laughing. Yeah, just the other day I was asked a real life problem of finding out in which order I would burst some balloons for maximum monetary gain.
Except that is not how it ends up. Candidates who get/take hints are largely de-ranked over those who "solved 3 questions flawlessly without any inputs within an hour" as are those candidates who bring in an approach that the interviewers did not have in their mind.
Source: Am one of those types of tech interviewers at one of those types of tech companies and also part of the HC having reviewed thousands of interview feedback!
If you're doing only technical questions, you're doing it wrong.
Spend half the time on behavioural questions, and half on tech, and suddenly you'll find it easier to justify hiring a well rounded engineer who needed a couple of hints over an unblinking automaton who can solve questions flawlessly yet would struggle with any real-life ambiguity.
You are spot on. Most of these tech rounds are a mix of coding, behavioral and design rounds (even the lunch interview matters). But when judging on coding ability, I'd be cautious of rejecting candidates with real world experience but may not be able to afford a good month of dedicated leetcode practise.
What does ethics have to do with answering interview questions? If he knows the answer from seeing the question before, that means he's a perfect candidate for the job. Are you seriously suggesting he should sabotage himself and ask for a question he doesn't know? That's beyond absurd. Frankly, it's stupid bordering on insane. What it isn't is unethical. Should the interviewee also ask for different questions on his final course exams because he knows the answers to the ones on the test from studying? What insanity!
Sorry, but unless I'm working on an actual database, I'm unlikely to need to solve btree (index/sort/resort/invert) type problems. I do have a decent understanding in principle, but generally am using a database layer that someone else wrote for these types of things.
The problem is you are going to use the hints and guidance as an excuse to cover up your biases.
Guy who looks like a close friend? Yeah I had to give him a hint, but that’s ok. Guy who’s ugly and speaks with a stutter? I had to give that guy a hint, can’t belive he couldn’t figure it out on his own.
Everyone is jealous of someone. It is amusing to compare Bezos's awe at Yasantha to that of many of his engineers towards him - or for that matter, my awe of anyone who can pass Amazon's recruitment process.
See this except from a biography of Bezos:
>... To the amazement and irritation of employees, Bezos’s criticisms are almost always on target. Bruce Jones, a former Amazon supply chain vice president, describes leading a five-engineer team figuring out ways to make the movement of workers in fulfillment centers more efficient. The group spent nine months on the task, then presented their work to Bezos. “We had beautiful documents, and everyone was really prepared,” Jones says. Bezos read the paper, said, “You’re all wrong,” stood up, and started writing on the whiteboard.
>“He had no background in control theory, no background in operating systems,” Jones says. “He only had minimum experience in the distribution centers and never spent weeks and months out on the line.” But Bezos laid out his argument on the whiteboard, and “every stinking thing he put down was correct and true,” Jones says. “It would be easier to stomach if we could prove he was wrong, but we couldn’t. That was a typical interaction with Jeff. He had this unbelievable ability to be incredibly intelligent about things he had nothing to do with, and he was totally ruthless about communicating it.”
Yasantha is an interesting fellow, with contributions in wireless networking, AI, chip design and various other fields.
In this article Yasantha describes Bezos's legendary work ethic[1].
>Yasantha described how students once dared each other to complete a computer science assignment in a single line of coding. “Finally, I gave up and did this in 10 or so lines of code,” he added, “But I remember Jeff working through all night, in pursuit of the most compact solution, and turned in a two-line solution that was probably the shortest anyone could do…”
>“It goes to prove that Jeff is tenacious, and will not give up like most of us would when presented with a challenge,” he said.
>“He sets his goals and sticks to them. I think that’s a quality that has made him who he is,” Yasantha added.
Think what it meant for someone such as him to give up on his dream to be a theoretical physicist.
Despite his tenaciousness he was forced to conclude there were levels of abstraction he could not reach through hard work.
I don't want to stroke the Bezos ego more than necessary - but he's not dumb.
Geniuses don't get rich by just being geniuses (think of Tesla holed up towards the end in a shitty hotel).
What you need is a smart person who can recognize other people can be smarter - and they need help, that you can provide.
I mean this without a trace of sarcasm - it's a bit of ego with a bit of humility (whilst ensuring you retain the stock)
I've always plugged Bezos into the Edison mold.
Reminds me of the story that Paul Allen recounted about Bill Gates going off to Harvard rather pumped up about his abilities relating to math (Gates has said he considered wanting to teach math as a profession when he was in high school) and then realizing he wasn't good enough at it to keep up with the elite math students there. That to keep up with them you'd need to be among the best in the field and even if he was good, he wasn't math-genius level good.
It's better to fail fast than to waste 5 years getting a PhD you don't really need. I've had a lot of professors who started out pursuing theoretical physics in grad school, but quickly switched to experimental physics after realizing that they were simply not the right material for theory. I was always surprised to hear it, because I regarded those professors as very bright. It is very humbling.
Tangentially, I wonder how the world would be if Jeff Bezos had become a theoretical physicist and hadn't founded Amazon. Perhaps he could've done just good-good instead of good-bad (tilting more and more to the bad side). Would someone else then have founded an Amazon-like company? That seems highly unlikely since even now, nobody else seems close to it on the combination of technology, retail, reach, growth and influence.
I would definetly like to be better at telling stories like this.
When I try to tell stories I always end up on irellevant tangents. That have almost nothing to do with the main point. Or I get stuck in some detail and forget what I was talking about.
I agree, he really sells it when he says “Yasantha” starred at the problem. Every sentence is clear and concise. It moves the story without adding anything extra. My story would still be going on — I would have hit on all of the major food groups; DMCA, Freemium model, and Publishing rights.
The key to being a good story teller, is telling the same story over and over again, refining it and removing any fluff from the story. This is what separates great comedians from just ok comedians as well. They're telling a story that has been refined in front of an audience to land just right.
He’s probably told this story 1000 times and so why you’re getting is the most refined version. I had a friend who was very very successful and he would continue to repeat a story until he nailed it and as a consequence he always came across as a genius. The genius was in the preparation.
This is off topic, but are there any physicists working today that are the modern day version of a Feynman, Einstein, Dirac, Wheeler, Bethe, Bohr, etc...?
I've been reading James Gleick's Genius and it was a question I had when reading about 23 year old Richard Feynman.
Look at prize winners - then, where relevant - their students or people they mentored. For mathematics, Abel Prize winners and for Physics the Nobel Prize winners and Sakurai Prize winners tend to be recognized as the leaders of their fields for the particular research they work on. These are names that many in a given field would recognize or at least be aware of.
he made many contributions to other parts of science (wireless, communications), though, that I doubt bezos would be able to so the point still stands.
Had Bezos continued in the theoretical physics direction, I am sure he would of been successful and productive in the field. He would of got the math. Great theoreticians have imaginations that are able to come up with ideas that have "fit". The mathematics help you get there, and help you understand the problem, but the imagination and creative thinking skills are crucial. Bezos has this ability - Amazon's success I think demonstrates this very well.
In this case, Bezos saw firsthand how a brain teaser could be answered by "seen something similar once before" much more easily than "worked on it on a team for several hours". Isn't this exactly the sort of brain teaser that we hate seeing in tech interviews?
This one was bad enough that Bezos left the field, yet Amazon (like every other software company) still asks these style of questions of its engineering candidates.