from Hacker News

Testing how hard it is to cheat with ChatGPT in interviews

by michael_mroczka on 1/31/24, 5:35 PM with 474 comments

• by jhawk28 on 2/1/24, 3:30 AM

  I interviewed a number of people for a few positions and I never told them that I detected them using ChatGPT. We structured our interviews in 2 parts. The first one was finding a bug. First clue if they were using AI was that they would solve it instantly. Second part was to write something related to our work that had definitive start/end. If they were using AI, they often were able to get something out, but they had no foundation to reason about it and modify it. They would quickly become lost. We always said that they could use whatever "helps" as long as they showed what they were doing on screen. For some reason, only one person openly showed that they were using AI, but that was only because they couldn't figure out how to turn it off in the UI. We didn't disqualify anyone for using AI, we disqualified them because of their dishonesty. If you can't trust someone in an interview, how can you trust them in a remote environment?

• by babyshake on 1/31/24, 8:41 PM

  IMO asking people to not use available tools in interviews is a bad idea, unless you are trying to do a very basic check that someone knows the fundamentals.

  Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer.

  Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.

• by svilen_dobrev on 2/1/24, 9:18 AM

  yeah.. most interview procedures/ways in recent ~10+years do not check if interviewees can think. Only if they can "remember". Which is akin to education's problem of last few decades.. there are plenty of machines for remembering now. But dumb-grinding is still the expected way in schools..

  While, wait, thinking...

  So all this "cheating" is maybe a (bit delayed) response to the above trend..

• by rvz on 1/31/24, 9:30 PM

  This conclusively tells us that the Leetcode grind has been (without any dispute) been gamed to the ground and is no longer an accurate measure of exceptional performance in the role. Even the interviewers would struggle with the questions themselves.

  Why waste each other's time in the interview when I (if I was the interviewer) can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool.

  I don't need to test you if you have already made significant contributions in the open. Easy assumptions can be made with the very least:

  * Has knowledge of Git.

  * Knows how to code in X language in a large project.

  * Has done code reviews on other people's code.

  * Is able to maintain a sophisticated project with external contributors.

  Everything else beyond that is secondary or optional and it's a very efficient evaluation and hard to fake.

  When there are too many candidates in the pipeline, Leetcoding them all is a waste of everyone's time. Overall leetcode optimizes to be gamed and is now a solved problem by ChatGPT.

• by tetha on 1/31/24, 9:05 PM

  Hm, interesting. To me, team fit, curiosity and, depending on the level of seniority I'm looking for, an impression of experience are the most important things in an interview.

  The latter might look like you could fake it with ChatGPT, but it'd be hard. For example, some time ago I was interviewing an admin with a bit of a monitoring focus and.. it's hard to replicate the amount of trust I gained to the guy when he was like "Oh yeah, munin was ugly AF but it worked.. well. Now we have better tech".

  I guess that's consistent with the article?

• by SheinhardtWigCo on 2/1/24, 12:42 AM

  There's an app for that: https://github.com/leetcode-mafia/cheetah

  > Cheetah is an AI-powered macOS app designed to assist users during remote software engineering interviews by providing real-time, discreet coaching and live coding platform integration.

• by m1el on 1/31/24, 8:46 PM

  I've had a displeasure of interviewing someone who used ChatGPT in a live setting. It was pretty obvious: I ask a short question, and I say that I expect a short answer on which I will expand further. The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question.

  Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer.

  Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?

• by michael_mroczka on 1/31/24, 8:47 PM

  I'm the author of this post. Happy to answer questions if you have any. This was such a fascinating experiment!

• by edanm on 2/1/24, 8:26 AM

  I'm not sure if I missed it, but why do they call using ChatGPT "cheating"? It's only "cheating" if you are explicitly asked not to use ChatGTP. (Also, not sure if it really counts as cheating, wouldn't it be more like fraud?)

  Some interviewers wouldn't mind, or would even encourage, using all available tools to solve problems.

• by gregors on 2/1/24, 2:11 PM

  All's fair in love, war and the job market. Good for them, interviewers have been jerking people around for a long long time. Maybe they should come up with some new material or start having actual conversations.

• by nottorp on 1/31/24, 9:10 PM

  Well, i've started to use ChatGPT instead of google when looking for quickie examples for something. Mainly because of how bad google has become.

  It works fine for stuff like "give me a tutorial on how to initialize $THING and talk to it" or "how do i set $SPECIFIC_PARAMETER for $THING up".

  Where it seems to fail is when you ask "how do i set $X" and the answer is "you can't set $X from code". I got some pretty hallucinations there. At least from the free ChatGPT.

  So maybe add a trick question where the answer is "it can't be done"? If you get hallucinations back, it should be clear what is up.

  Edit: not that I'm a fan of leetcode interviews. But then to get a government job in medieval China you had to be able to write essays based on Confucius. Seems similar to me.

• by OnionBlender on 1/31/24, 9:00 PM

  I've wondered about cheating with a friend who can (out of ear shot but can hear the call) type in the question and display the result on a screen the interviewee can see. I often get stuck on leetcode problems and simple hints like "O(n), prefix sum" can make a huge difference. Especially if I haven't seen the problem before or is having a brain fart.

  I would still need to get good at leetcode, just not _as_ good.

• by tnel77 on 2/1/24, 3:36 PM

  I think the perfect interview process is:

  1) A couple basic coding questions. FizzBuzz and such. Can they actually solve something basic?

  2) Do a real code review with this person. Share your screen and let them review code. Observe what questions they ask and the comments they leave for the author.

  3) Ask some design questions. Digging in on how they would design the classes for some new product and purposely throwing a twist in there from time to time. How do they handle this new information and adapt their design? Do they take constructive criticism well?

  4) Talking to this person. Are they polite and respectful? You can help someone grow as an engineer, but good luck getting them to be a better coworker if they are rude.

• by rbut on 2/1/24, 1:16 AM

  We recently interviewed some candidates and got them to complete an initial DevSkiller test. In the interview I asked each what tools they used to complete the DevSkiller test. All admitted to using ChatGPT or Copilot to complete the test. So now DevSkiller is really just a litmus test to determine whether they can be bothered completing the test in order to get interviewed.

• by justin_oaks on 1/31/24, 11:42 PM

  > We ended up conducting 37 interviews overall, 32 of which we were able to use (we had to remove 5 because participants didn’t follow directions)

  Removing interviewees because they don't follow directions seems like a good strategy. And I mean removing them as job candidates, not removing them from the study.

  It's good to have something in the interview process used explicitly for weeding out people who don't follow directions. Something like "Email us your application, and put the word 'eggplant' somewhere in the subject line. We use it to filter out spam." And then literally delete any subjects that don't have "eggplant" in them.

• by nostromo on 1/31/24, 8:47 PM

  I'm glad ChatGPT could be the end of leetcode interviews.

  I worry though that it'll just be the end of online leetcode interviews and employers will bring people back into the office to interview.

• by User23 on 2/1/24, 12:55 AM

  Why not just let the candidate use ChatGPT or autopilot or whatever if it’s permitted at your workplace? It’s not hard to detect if they actually understand what it’s doing. In fact competence at LLM assisted coding is hire signal as far as I’m concerned.

• by andreagrandi on 2/1/24, 9:50 AM

  If you are asking questions which can be answered by Chat-GPT, maybe you are asking the wrong questions.

  GPT is a tool which can legitimately be used to do your job.

  There are so many things that GPT can't do: take decisions, find the best approach to talk with a human being, resolve conflicts between two members of the team, and last but not least explain why of a certain solution.

  Is it a coding test? Pair with the candidate. See how they think. Ask yourself: would I enjoy working with this person?

  And make your own decision.

• by bluedino on 1/31/24, 8:45 PM

  The last couple interviews I have had, warned about using ChatGPT. So it must be happening.

• by TrackerFF on 2/1/24, 2:51 PM

  My takeaway from this is:

  Companies need to start conducting interviews where tools are available.

  When everyone is cheating, no-one is cheating. Trying to customize your questions is just a race to the bottom, and will always be an arms race against the LLMs.

  So, instead, let the candidate use whatever tools they want - in the open, and rather probe them on their thought process.

• by qweqwe14 on 1/31/24, 8:58 PM

  The core problem with interviews is that it's basically impossible to tell how well someone's going to perform on the job. It's always possible to grind leetcode or whatever and make it look like you know what you're talking about, by using the model people can just skip that part entirely.

  Not to mention the fact that some interviewers feel obliged to ask useless cliche questions like "why do you think you are a good fit for this position" yada yada.

  Not going to be surprised if picking people based on random chance (if they meet basic requirements) is going to actually be better statistically than bombarding them with questions trying to determine if they are good enough. Really feels like we are trying to find a pattern in randomness at that point.

  Bottom line is that if ChatGPT is actually a problem for the interview process, then the process is just broken.

• by locallost on 1/31/24, 9:04 PM

  Personally I am waiting for deep faked video chats with chatgpt generating the answers. And maybe even questions.

• by kmoser on 2/1/24, 5:45 AM

  Instead of asking the candidate to solve coding challenges, ask them to create a coding challenge. The interviewer can then use ChatGPT to see if the coding challenge the candidate wrote was already known to ChatGPT or if it was indeed created from whole cloth by the candidate. Example:

  Interviewer: "Create a coding challenge that requires sorting a CSV file by timestamp. Require the timestamps to be in some weird, nonstandard format and describe the format. Provide a few entries in the sample data that contain a timestamp which is ambiguous."

• by bspates on 2/1/24, 1:39 AM

  The most interesting part is that the control group (no cheating) has a ~50% pass rate on random leetcode questions. Tech hiring is so arbitrary.

• by p0w3n3d on 1/31/24, 9:54 PM

  Two cameras interview. One from the laptop, another from the back of the interviewee head showing the whole screen

• by glonq on 2/1/24, 8:06 PM

  I interviewed two guys last month who were definitely using some form of real-time assistance on a remote (MS Teams) interview. Their English language skills were atrocious, but their answers were still peppered with ten-dollar words that they couldn't pronounce.

• by aryehof on 2/1/24, 5:46 AM

  I’d suggest the wrong questions are being asked. Interview for understanding not just information.

• by jawr on 1/31/24, 10:46 PM

  Is it really cheating if they’re allowed to use online tools for their day to day?

• by lkdfjlkdfjlg on 1/31/24, 9:21 PM

  I've tried a couple of time using ChatGPT on a coding assignment (because.... if I can NOT do it, better right?) and both times I got garbage and ended up doing the coding assignment myself.

• by ngneer on 1/31/24, 10:54 PM

  I am fortunate to be in a field that AI has not caught up with. I interview security researchers. Would ChatGPT spot a vulnerability in a function it has never seen before?

• by hijinks on 1/31/24, 8:50 PM

  as someone that has been remove for 10 years now and interviewed a lot of people.

  You can 100% tell when someone is reading off a screen and not looking at you during an interview via webcam

• by willcipriano on 1/31/24, 11:26 PM

  If I pass the interview with ChatGPT I don't want the job.

• by megamix on 2/1/24, 9:26 AM

  For one's own sake, use the approach that will lead to self-actualization.

• by jgilias on 2/1/24, 9:21 AM

  I really hope this ends up killing LeetCode interviews with fire.

  The negative assessment of one of the interviewers about a candidate how “he hadn’t prepared to solve even the most basic LeetCode problems” is especially telling.

  Maybe the candidate had really honed their sudoku solving skills instead.

• by mirekrusin on 2/1/24, 9:12 AM

  What's the point of all of this?

  ChatGPT (or local/hosted LLMs) should be tools available at workplace nowadays.

  Interview while using LLMs, wikipedia, google, SO, o'reilly or whatnot should be not only allowed but encouraged.

  Just have conversation/pair programming like session with gpts open and shared - just how you'd work with that person.

  That's how they'll work for/with you.

  Mission. Fucking. Accomplished. [0]

  [0] https://xkcd.com/810

• by nine_zeros on 1/31/24, 8:55 PM

  I've seen people use chrome extensions like leetbuddy.

• by andrewstuart on 1/31/24, 9:50 PM

  - Using ChatGPT is not cheating.

  - Using an IDE is not cheating.

  - Using StackOverflow is not cheating.

  - Reading the documentation is not cheating.

  I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills.

  I would also expect them to be skilled in using their choice of IDE.

  I would expect them to know how to use Google and StackOverflow for problem solving.

  I would expect programmers applying for jobs to use every tool at their disposal to get the job done.

  If you come to an interview without any AI coding skills you would certainly be marked down.

  And if I gave you some sort of skills test, then I would expect you to use all of your strongest tools to get the best result you can.

  When someone is interviewed for a job, the idea is to work out how they would go doing the job, and doing the job of programming means using AI copilots, IDEs, StackOverflow, Google, github, documentation, with the goal being to write code that builds stuff.

  Its ridiculous to demonise certain tools for what reason - prejudice? Fear? Lack of understanding?

  There's this idea that when you assess programmers in a job interview they should be assessed whilst stripped of their knowledge tools - absolute bunk. If your recruiting process trips candidates of knowledge tools then you're holding it wrong.

• by p0w3n3d on 1/31/24, 9:57 PM

  Also AI will make us dumb. Those of us who decide to use AI extensively will get lazy, and brai removes the lazy parts of knowledge as they are no longer needed. Meanwhile AI will learn from internet only based on AI generated text, which as we know, causes AI models to deteriorate. nobody will write anything. The society collapses. We admire and worship big computer and a man who can fix it. Basically a Wizard Of OZ scenario

• by pcthrowaway on 2/1/24, 5:34 AM

  For junior level candidates, I'll admit that ChatGPT might make it harder to interview.

  For senior+ candidates I honestly think the correct approach is to just lean into it though.

  Encourage them to use ChatGPT at the outset, and select questions that you've already fed to the prompt. When you ask them the question, you can show them ChatGPT's output on a screenshare. The candidate can then talk you through what they like about the answer, as well as where it falls short.

  A senior-level developer should almost always be capable of improving on any response given by ChatGPT, even when it gives a good solution.

  And if they're not able to give better output than the current AI tooling, it's a pretty good signal that you'd be better off just using LLMs yourself instead of hiring them.