by garrettdimon on 4/24/23, 7:19 PM with 220 comments
by bennyelv on 4/25/23, 9:46 AM
There's a fundamental skill that a good programmer has to have, and that is to be able to take a novel problem that they haven't seen before and break it down to solve it in a sensible way.
There are plenty of programmers who fake their way through a career without having that skill. They just copy stuff and never really understand it. They're fine for some types of programming career, but if your business involves solving new and novel problems then you have to know which type of programmer you're hiring.
A contrived live coding exercise gives you a strong signal on this. It does have a decent chance of producing a false negative, but only a very small chance of a false positive, and that's the trade that has to be made with this kind of approach.
Is a better option to not do this kind of assessment, hire the person, find out that they can't do the hard bits and then fire them within 6 months? I'll take some convincing of that...
by macawfish on 4/25/23, 8:12 PM
Immediately after the time was up I magically started to realize how much I'd over complicated my one almost finished answer and quickly came up with much more efficient answer. Suddenly it started clicking how simple the first two problems were and how easy it would have been to crank them out if I wasn't panicking about the tech bubble collapse.
The recruiter later asked me how it went and I grumbled something about how it was a leetcode test. He said "oh well they need to make sure you have the skills for the job." At that point I was over the whole thing and honestly pretty fired up.
Over the next week I proved to myself that I do have the technical skills for that job, and that's honestly what counts.
¯ \ _ ( ツ ) _ / ¯
by vparikh on 4/25/23, 8:01 PM
- C/C++ on Win16/Win32
- Assembly language development with Z80/8051/ARM on embedded microcontrollers
- Java (core java, Servlets, J2EE)
- Ruby on Rails
- NodeJS / Javascript
- Worked with AWS tech (the full stack)
- Relational DB (MsSQL, PostGres, MySql), NoSQL db (MonoDB)
- Coded for Linux/Unix, MacOS, Windows 16/32, PalmOS, iOS
I can provide reference for each of those skillsets form my past colleagues.
You know what - I probably couldn't pass half of the insane coding puzzles these interviewers throw out. Not because I can't solve them, I just don't remember enough of the syntax or library semantics of the top of my head.
At my experience can we just assume that I am a competent coder (maybe not the top 1%, but at least in the top %20) and talk about the job and how I can contribute ? I mean its almost insulting if you ask me to make a linked list/reverse a binary tree or other such nonsense looking over my shoulder me with a time limit.
by alpinelogic on 4/25/23, 3:24 AM
by Ancapistani on 4/25/23, 7:03 PM
When I'm the applicant, I make it a point to take control of the narrative by saying something like:
> If it's alright with you, I'd like to approach this as an opportunity to expose how I approach problems in general rather than how I'd solve this specific problem. I'll speak stream-of-consciousness as I go through it so you can get an idea of how I'm thinking about it. Feel free to ask questions if you'd like; I'll rely on you to decide whether it's more important to you that I complete the task or explain my reasoning. I'm happy to switch to pseudo-code or just discuss potential approaches if we run short on time.
When I'm the interviewer, I open with pretty much the same thing. My goal is to put the applicant at ease (to the extent that I can) and make it clear what I'm trying to get out of the session:
> First, let me say that it doesn't matter to me if you complete the exercise or not. At this stage of the interview process I'm confident that you're more than capable of solving the problem, so lets use this as an opportunity to get to know each other and see if the way we think about logic is compatible. I'd love it if you could point out things you'd change, but don't worry about trying to 'finish' or end up with production-ready code. It's just a means to an end.
by uptown on 4/25/23, 4:02 PM
It’d be more akin to pair-programming, which some of the interviews I’ve conducted have evolved into, depending on the strength of the candidates.
Would that capture enough to get a sense whether this person gets the gist of the code being written such that they could replicate the same output in a less contrived scenario?
by lbriner on 4/25/23, 4:39 PM
What else are we supposed to do? Take the fact that you can talk a good game as enough of a signal to invest 10s of 1000s? Assume that everyone with 20 years experience is as good as everyone else?
The problem is that there are no reliable signals. Most Developers I have interviewed have a massively inaccurate ability to judge their own ability (in both directions). I've lost count of the number of times candidates have rpomised that they can just learn whatever they don't already know and haven't been able to do it to any degree.
Qualifications are meaningful in some contexts more than others but most people in the UK don't have comp-sci qualifications.
So yes, I will use various coding exercises because depending on the level, it shouldn't phase someone to be given something quite simple and to see how they approach it (do they write tests first? Ask some good scope questions? Explain why they've done something the way they did?)
I have failed one of these tests in the past thinking I was a good Developer (I am!) but I don't blame the test or the process, I realised that my approach was haphazard and not an objective good look to an Interviewer so it was actually helpful.
by lucisferre on 4/25/23, 3:46 PM
Whether this is actually the case comes down to the hiring manager.
If the hiring manager decides to optimize for performance in the coding interview then yes everything said here is true. They will typically hire people who can perform well and fast at simple coding test type problems above other any other desirable attributes.
If however, they simply evaluate the ability to work through (at any reasonable pace) a fairly trivial coding task, but make the hiring decision on bulk of the rest of the interview, then it shouldn't be a problem.
The problem is most hiring managers have not been selected for their ability, or even trained, in interviewing. A coding test is easy to set up (or copy from somewhere), administer and evaluate. It is often literally the least they could do.
What this post describes is simply hiring managers who lack interviewing skills. Personally, I would probably want to avoid reporting to such a person.
by kayo_20211030 on 4/24/23, 8:47 PM
by flimsypremise on 4/25/23, 5:11 PM
Of course, in order to be able to assess the answers to these questions in an effective way, you need to be able to very knowledgeable yourself, and that's the real root of the problem here.
by robomartin on 4/25/23, 7:26 PM
What's important is how people approach computational problem solving, not if they can write a solution in 1 to 45 minutes. Really, who cares?
One of my go-to examples is trying to work from home. Which is great, yet has its challenges. We have three German Shepherds. They are lovely. However, when a delivery arrives or the gardeners are out nearby, well, it's mayhem for a few minutes. I've come to understand that I should just take a coffee break when that happens. I can't even do mid-level math, much less focus on a difficult CS problem during those moments. And it takes a good 30 minutes before my head can be back on task.
Stop treating software development like performing art or athletes having to perform in the heat of a game. That is not what we do. At all.
by whoomp12342 on 4/25/23, 5:45 PM
Write a function that takes in a string and returns 1 if the amount of letters in the string is odd. It returns 0 if the amount of letters in the string is even.
If you can do this, we are good. I dont need NP hard or whiteboarding or algorithms. I can tell how good you are just by talking to you about your background.
by jamesgill on 4/25/23, 4:20 PM
Another problem is that we're prone to thinking that being able to do well on tests equates to doing well in life and work--despite a stunning lack of evidence in support it.
by nailer on 4/25/23, 4:18 PM
by RoyGBivCap on 4/25/23, 4:40 PM
It's a real skill you'll actually employ. You're coming at the code cold, which is actually a realistic scenario you'll encounter on the job. Your ability to catch bad ideas and prevent them from getting literally codified is a valuable skill. And all of that is worthless if you're in a state where you can see a mistake, but are too afraid to speak up; this gets tested too.
It might not be so great for newbies and people fresh out of college, but even they should be able to read the code and discuss it.
by JohnMakin on 4/25/23, 6:30 PM
as a devops engineer as my main job, I also try to explain to interviewers that on any given day I am reading and writing half a dozen languages of vastly different paradigms, and although I'm very proficient in many of them, I definitely need to reference things that maybe shouldn't "need" to be referenced (like confusing bash/python loop syntax is very common for me as an example). This rarely ever slows me down in reality, but will definitely cause me to fail interviews I shouldn't.
If I was an interviewer, I wouldn't care if a dev knew whatever esoteric language syntax or API calls by heart. I'd just expect them to know how to use them intelligently. The former does not always imply the latter.
by CobrastanJorji on 4/25/23, 5:53 PM
YES! Yes, exactly. A consistent and unbiased process that reliably weeds out people who is very bad at programming is incredibly useful. I'm not convinced live coding interviews are either of those things, but assuming they are, they are absolutely worth the listed downsides. Do they filter out lots of great programmers along with the bad applicants? Yes, totally, and that's a major waste. But all of the listed alternatives are less reliable, less consistent, slower, and friendlier to cheating.
I would love to eliminate live coding interviews where I work. I hate the things. But I have never encountered a mostly consistent and kinda objective solution that compares. I was hopeful that the essay was leading to a proposal for one and disappointed once again at its lack. Please, someone tell me what the giant tech companies should use instead, and I will gladly throw these "please reverse this linked list" interviews into the trash.
by sintezcs on 4/25/23, 6:36 PM
by syngrog66 on 4/25/23, 4:17 PM
and very unnatural
and insulting to an experienced person (whos been promoted, and whos survived many layoffs in the past, etc) with many accomplishments. someone who has clearly solved and shipped, repeatedly. and one who has artifacts visible out in public. and has praise testimonials from former bosses, coworkers, clients, etc
by emodendroket on 4/25/23, 4:25 PM
by Taylor_OD on 4/25/23, 4:20 PM
This type of interview related blog post seems to hit the front page every week. Almost everyone agrees that interviewing doesn't feel good and seems overly complicated/difficult/whatever.
I hate that interviewing is a skill that has to be developed, in large part, separately from other engineering skills. I also don't really enjoy SQL/Database work. But I've gained enough competence in SQL that I'll be fine in most jobs. The same is true for my interviewing skills.
by rockbruno on 4/25/23, 4:03 PM
This format is popular because it's the best time/effort trade-off for both the company and the candidate. It's massively flawed, but everything else attempted so far turned out to be even worse.
by assbuttbuttass on 4/26/23, 2:22 PM
In my experience conducting these live coding interviews, it's almost a universal rule that if the candidate starts coding immediately, they will waste a lot of time on irrelevant details, or take a long time to see that their approach can't work.
I always try to encourage the candidate to talk through their solution or draw a diagram if it helps them. Candidates who follow this advice always perform better.
by fourseventy on 4/25/23, 4:02 PM
by lasereyes136 on 4/26/23, 2:57 PM
To me this is the key to the whole problem. For many companies live coding interviews are a cargo cult approach to interviewing. Since the people that do them don't understand how to evaluate others they become a checkbox exercise that only evaluates if the candidate would tackle the problem the same way the interviewer thinks they would tackle the problem.
by beerpls on 4/25/23, 6:01 PM
I refuse to take those interviews. I had one company that promised they wouldn’t give me a live test, and when I told that to the interviewer who was trying to give me a test he said “hm, well we’re going to do it anyway”
I passed the test and was given an offer which I shot down for the company which respected my terms
In the end the other company was left high and dry when they needed the new team member
Just do it. Enough of us see this for what it is. If we just act how we think we change the industry
by Gunax on 4/25/23, 3:53 PM
The author concludes with a decent summary of the issue (ie. Is this the best method among the worst?). But doesn't actually find a better way of doing things.
And in the million forum threads on this issue, no one else has either.
by languagehacker on 4/25/23, 5:57 PM
There is a nice side effect about doing this as a coding interview; there are often obvious indicators that a candidate is a poor fit -- for instance, big knowledge gaps in the standard library of the primary coding language.
More important is how the candidate structures how they would solve the problem, and how they communicate it to the person on the other side of the discussion. Are they taking testing into account? Do they iterate rapidly, or have a monolithic solution they have a hard time conveying?
For lack of an effective whiteboarding solution with remote interviews, coding interviews are here to stay. They should be reframed, though, as focusing on collaboration -- not on being a leetcode champion.
by rolenthedeep on 4/25/23, 11:52 PM
The way we do these at my current job is extremely productive for us. We look for two things, apart from basic competency: problem solving, and asking for help.
It's structured as a two 90 minute pair programming sessions. Interviewee shares their screen, and we work through the problems together. Obviously it's pretty hands-off, but we guide and nudge where appropriate. Here and there, when they use something that relates to a deeper topic, I'll ask questions to gauge how deep their knowledge goes. Like asking if they know how C#'s foreach works under the hood. Not as a selection criteria, but simply to get a sense of how much they know.
Use of a search engine is openly encouraged. A lot of the time, we don't even care if the program actually runs. If they struggle with syntax or the correct function overload, we'll help them out after giving them a little time to find the solution.
I also throw in a problem designed to get them stuck, and ask questions I expect they can't answer. A good programmer asks for help and admits when they don't know. A bad one bullshits their way through.
We want to hire programmers who can do a real job in the real world. Implementing red/black trees on a whiteboard blindfolded isn't a job skill, it's a party trick. That's not something a programmer will ever need to do.
In the real world, real people use google and stack overflow. They don't have encyclopedic knowledge of the entire language's syntax. They ask their coworkers for help or opinions.
Our interview process is designed to show us how a person will function in a scenario as close to the job as possible. Because that's what we're hiring them for. We look for their ability to work through a problem with the resources that everyone always has. We look for how they work with others and how much they lean on coworkers.
This has worked out extremely well for us. We've hired some very talented individuals, and have totally avoided the archetypal shitty dev. The people we hire immediately mesh with the team, and learn and grow the way all programmers do.
Granted, we are a small company and we have the time to have our own programmers giving interviews. We also have a much higher need to be so selective. But every single person who has made it to the technical interview has remarked unprompted that it's the best interview they've ever had. And I mean 100%.
It's because we treat candidates the way we'd treat our own employees. They get to know what the job is like, and we get to know how they'll do the job.
by angarg12 on 4/25/23, 6:13 PM
First, how we got here. I didn't experience this first hand, so I'm just making a recount based on what I've heard and read from "old timers". My understanding is that before whiteboard, leetcode-style interviews became the norm, tech interviews were mostly unstructured and quite informal. In the really old times (pre 90s) you could get a job just by knowing how to use a computer. I believe this wasn't that unusual in the 90s and early 2000s. I still remember hearing a founder-CEO bragging that his interview process was a 30 min chat with each candidate, and if he liked them, he would hire them.
From what I could gather Microsoft is the first big company that started using these "coding challenges". Then they became wildly popular thanks to Joel Spolsky [1], the publishing of Cracking the Coding Interview [2], and Google made brain teasers world famous.
Second, why we are still here. First, I believe there is a huge cargo-cult factor. Companies want to copy big tech, and alumni from these companies go on to found their own. This kind of interview has been honed and polished over the years, landing in a local optimum. An entire industry of websites and products has been created, and there are many entrenched interests. People might hate it, but the process works well enough for tech companies that they don't need to worry too much about it. Another under-appreciated factor is scale. This kind of interview sort-of scales well, which is important when you hire at a massive scale. That's why things like "have the candidate come to work one day and pay them" won't work for companies that need to screen thousands of people a week. Lastly, a standardized and "well known" process introduces some guardrails that can avoid some obvious pitfalls. The book "Working Backwards" explain how the Bar Raiser program was created at Amazon when a bad senior leader hire used the unstructured approach to hiring to build an empire misaligned with the company. At a big enough company this is bound to happen sooner or later.
Third, where do we go from here? I find it extremely unlike existing big companies will change their methodology any time soon. It might not seem like it, but for a company like Google it would be a massive undertaking to overhaul their hiring process. It would take years to achieve the level of efficiency and effectiveness of the current system, and surely there would be tons of opposition.
I believe the only way forward is for new companies to experiment with other methods of hiring, particularly at the beginning when they are nimble and can experiment freely. As they grow they will face challenges scaling, polishing and standardizing their process. At some point they will become the next generation of Big Tech, and the cargo cult wheel might spin again.
In any event it seems we need some sort of structured approach to hiring where we assess the match between company and candidate.
[1] https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid...
by eyear on 4/25/23, 3:50 PM
by Arainach on 4/25/23, 3:48 PM
Sure, you can often tell the folks who know nothing when asking them to explain the code, but it's increasingly difficult to tell a great engineer apart from a mediocre one that's cheating, and once you start hiring cheaters the toxic effect to culture sets in fast.