by blackhole on 3/26/21, 1:38 PM with 344 comments
by kentonv on 3/26/21, 3:14 PM
I think it is remarkably effective at identifying the kinds of skills and personality traits that a software engineer actually needs to have in day-to-day work. You can find out if someone is self-directed, how fast they work, whether they produce clean designs or spaghetti code, whether they are good at cooperating or tend to go off on their own, etc. Some people will just sit and watch and do nothing unless instructed... that's bad. Some people will build stuff, but with obvious efficiency flaws and "bugs"... also bad. Some people identify what needs doing and get it done effectively but without trying to be perfect... that's good. Factorio essentially compresses real-world work patterns into a shorter time period, giving you the chance to see how someone works in the space of a couple hours. I don't know anything else that does that.
But, obviously, it's problematic. A person who has played the game before will have a big advantage. A person who doesn't play video games at all will have a big disadvantage. For these reasons, I don't think I could seriously recommend this approach be adopted more widely.
But then, I don't know of any approach to interviewing that I think is fair. Everything has problems:
* Regular interviews bias towards people that are charismatic, not effective.
* Coding interviews bias towards people who can code under pressure with someone looking over their shoulder, which is almost nothing like real-world coding.
* Puzzle-y algorithms questions identify people good at clever solutions but that's hardly what most people need to do day-to-day when coding.
* Take-home assignments bias towards people who have lots of free time, and still don't really tell you how that person works with others.
* Looking at GitHub history biases against people whose lives are too busy to code in their free time (e.g. maybe they have kids) and who weren't lucky enough to have a previous job that touched a lot of open source.
* Looking at past work experience misses brilliant junior programmers coming out of college.
* Don't even think about using ML for this, that doesn't solve biases, it just hides them in a black box.
It seems like there's no right answer here.
by odshoifsdhfs on 3/26/21, 5:33 PM
Hiring is broken because most people hiring aren't good at it and the people working in tech are pretty much idiots (I'm focusing more on SV-startup, 5hour discussion about aeropress, put people down because they chose vim or emacs - people kinds). From 'cultural' fit interviews (who the fuck cares if you are a sikh or an atheist, whiskey drinking or think alcohol is evil? You are hired to do a job, not much more). This 'i have my social life at work, so I want people I can hang out with' shit has got to end. I work remotely for 10+ years and I can see in interviews/meetings the people that seem to base their identity and life around work, and the ones that actually are well rounded people that don't care about which coffee grind size you use on your morning crap of joe.
I have been a professional boxer and dabbled with MMA/Grappling. I can say most things in the article can apply. So.. at the end of the interview just get them in gloves and duke a few rounds? I would be sued out of existence, but for some reason, between take home assignments, 10h whiteboard drilling and whatever the fuck is hip now (leetcode or what not), interviewing is broken because people are idiots. I have been interviewing people for the last 5 years. If in an one hour talk that is semi technical, you can't understand if the person knows what the fuck they are doing or not, then the problem is you, not the whiteboard or the take-home or the paid assignment. Just admit most people interviewing others for technical positions have not fucking clue or training, BUT, hey, they are smart people from Standford or MIT, so 'how hard can it be' and we get the clusterfuck that it is now
by notJim on 3/26/21, 6:47 PM
You could get a candidate who perfectly understands design patterns for programming, because they've studied programming and know how to program using programming languages and programming tools, but they don't translate that very well into the spatial environment of Factorio. Debugging and optimizing in Factorio again are conceptually similar to doing so in a programming language, but the actual techniques and patterns available to do so are very very different, and you have to learn them. Most of us do not figure these things out from scratch by ourselves.
As others have pointed out, it also is begging for a monoculture. Plenty of people who are good at programming don't give a shit about video games, and will either do poorly in this interview, or will skip it altogether. The result would be hiring one type of person, and not having any other skillsets on your team. Even as someone who would do well in this interview, I would run screaming, as I do not want to work in a monoculture.
by sangnoir on 3/26/21, 6:30 PM
I'm confounded on why the author thinks being a senior developer (say, in embedded systems) would make someone better at exploring a game UI compared to an intern-candidate who puts 20+ hours into gaming per week...
I would probably walk out of the interview if someone suggested I play factorio as part of the hiring process - I have been intentionally avoiding it - you might as well offer me a free hit of crack cocaine while you're at it.
by jontas on 3/26/21, 4:15 PM
I don't think this is a detriment to my programming ability at all--spatial relations really never enters into architecting a large system. Of course you need to think about design constraints but none of them exist on a physical plane.
Factorio, on the other hand, requires actual spatial relation ability. You need to visualize how belts intersect and how to best position things so they dont interfere with each other. This is where I struggled.
by KineticLensman on 3/26/21, 2:46 PM
"What's the takeaway from this? I don't know. We certainly can't switch to using Factorio as an interviewing method - you might as well just give a candidate a take-home assignment. At the very least, we can do better than whiteboard interviews."
So it's not a description of how to use Factorio for interviewing, beyond a simple assignment of Factorio tasks to seniority levels. It does however make an interesting comparison between Factorio and software mechanisms.
by theptip on 3/26/21, 5:42 PM
“Fizzbuzz is all we have...” is a myopic take. Interviewing is hard but not as hard as the article makes out. There is a legitimate debate between take-home and on-site interview practices, but at the end of the day you want to see a work sample that is as realistic as possible from your candidate. Companies that are good at hiring understand this. Factorio has a tenuous (at best) correlation to the work you’re asked to do.
It’s like doing puzzle questions because you “can see how they think about problems”, but interviewers are really just selecting for people that think like they do.
(And for context I like Factorio.)
by francisduvivier on 3/26/21, 2:36 PM
by wskinner on 3/26/21, 2:51 PM
by lvice on 3/26/21, 2:51 PM
While I played, I came to similar conclusions to the article. Building factories in this game is very very similar conceptually to building software. I ran into exactly the same problems that I ran while I build software like:
- If I don't plan ahead enough the scope and layout of the factory, I end up with a spaghetti mess that is very difficult to rectify (technical debt)
- If I plan too much ahead, I overwhelm myself without even starting. The amount over-engineering and over-preparation becomes counterproductive and demoralizing.
- Starting new factories is fun. Maintaining and extending a factory that is starting to show serious design issues is a chore. I always tend to want to scrap the factory and start fresh.
- While designing a factory, modularization is key. You can go with a "monolithic" factory where you provide all possible materials as input, and try to build everything as output. It is very efficient transportation wise, and can centralize all management, but it can and it will become an unmaintainable mess. You can also design factories as "microservices", where each factory is a very compact, clean and scoped. It will only produce nails, or rubber, or copper wire. When you need to increase production of that item, you just duplicate the module (horizontal scaling). It seems fantastic at first, but the issue is now transportation. Dozens of micro-factories have to communicate with themselves to combine and produce more complicated items. The physical distance makes planning transportation a logistic and construction nightmare. So you have to find the right compromise between monolithic and micro-services.
I think I agree with the article that you can extract a lot of information about how good a person can be at software development by the way he plays Factorio/Satisfactory. Not so practical though :)
by kevincox on 3/26/21, 2:47 PM
by tyingq on 3/26/21, 2:42 PM
Obviously, you still need to do interviews, but keep some perspective when deciding. Don't overweight any one thing.
by markus_zhang on 3/26/21, 3:57 PM
https://store.steampowered.com/app/1366540/Dyson_Sphere_Prog...
The leading programmer uses some sophisticated ways of optimization, including:
- DOP instead of OOP
- Leveraging GPU for parellely calculating certain stuffs
- Using a GTX 660 Ti for development
Here is a technical post but in Chinese. Lemme know if you are interested and maybe I can do some translation.
by splonk on 3/26/21, 8:11 PM
I have turned down an interview because of Diablo. I got recruited a couple times by a startup that would have been a very good fit for me. The first time it was quite small and I chatted with the CEO about possible roles and product directions and such, but I'd just quit a long term job and eventually decided that I wanted to take a break for a while.
The second time was years later after it had grown a lot and gotten tons of funding over multiple rounds. As it happens, the CTO was also a friend-of-a-friend. When the Diablo 3 beta came out, I played a bit with him and our mutual social circle, and he demonstrated enough personality traits during that brief experience that I decided I wasn't interested in socializing with him more, and just turned down the recruiter after checking to see that he was still the CTO. I would have at least discussed it further if he wasn't specifically in a leadership position.
by omginternets on 3/26/21, 2:30 PM
So really, this article is saying "live, self-directed coding is the best technical interview", which isn't very original.
by tehjoker on 3/26/21, 5:38 PM
by _e on 3/26/21, 3:10 PM
I used Zachtronic's TIS-100 [0] as a test variable and found the best person to work with. I believe it was a two way street because they saw an opportunity to work with someone who attacks problems a little differently. I am in real estate. Who would have thought knowing something about assembly would be helpful?
by S_A_P on 3/26/21, 2:39 PM
by TruthWillHurt on 3/26/21, 3:46 PM
by lmilcin on 3/26/21, 5:10 PM
More senior person is one that has more mature approach to their work. One that can get better results with less resources. Typically more senior people can also accept wider variety of tasks.
What is better results and what is less resources will depend on the project.
It has nothing to do with the knowledge or experience, in my opinion.
You can have good senior person get better results in a technology they have no previous experience with compared to a very experienced / intelligent / knowledgeable person that has track record of making bad decisions. Senior person will use their good judgement to warn the manager they are running outside of their knowledge and to know when they need to use some resources (like help) from somebody more knowledgeable.
A knowledgeable but less senior person may think they know anything but not be able to recognize they are trying to achieve something that is outside their area of expertise or not be able to recognize or accept they are bad at something.
For me senior developer is somebody I can entrust they will put honest, worthwhile attempt at making good decisions when it is called for and recognize when they need to come back for further direction.
Senior developers show ownership in that they seek to uncover problems and discuss those problems with the team, proactively and productively. Senior developers can smell problems even if they don't necessarily know the solution. They can act on the signs of the bad smell and maybe seek discussion (with architect? client? manager? team?) to figure out what is going on and what needs to be done further.
Senior developers understand there are many ways/levels to solve the problem and sometimes solution isn't more code but maybe procedural change or a shift in paradigm.
Senior developers can meaningfully help/direct more junior team members individually without taking over their projects.
Senior developers can keep working relationships even with difficult people or people they don't like.
And so on.
by miki123211 on 3/26/21, 3:42 PM
by endisneigh on 3/26/21, 2:48 PM
the issue is with technical jobs it's difficult to surmise what exactly is "representative" to begin with. I imagine the only reason candidates aren't just asked to work in the actual job for say, 8 hours instead of doing say, 7 irrelevant 45min interviews is because of intellectual property.
I wonder if Google or another large company has ever tried to re-interview all employees with more than say, 4 years of experience and correlated their interview performance with past job performance.
by yodelshady on 3/26/21, 5:01 PM
For instance, I like to group chem plants together as a block, that I can move around easily, set up remote bases, or duplicate easily. That, of course, means a common input "format". I'd love for in-game blueprints to be able to label that (or am I missing it?).
by dasil003 on 3/26/21, 7:13 PM
Casually playing Factorio with someone might give you a strong hint of someone's potential as a software engineer, but the minute you bring it anywhere near a high paying job interview process you are asking for disaster.
by lmilcin on 3/26/21, 11:31 PM
I stopped when I noticed candidates do worse at this game than actual 7 year old kids.
For example, a simple labirynth. Kids just give simple instructions (forward 3 blocks, turn right, forward 2 blocks, turn left, etc.). Candidates... try to figure out a complicated algorithm with a result that they run out of time producing nothing.
I wonder how much of this is effect of knowing you are being watched and your program analyzed and how much of this can be related to real world projects that developers just overcomplicate for no reason leading to worse results than just simple code you could throw in a fraction of the if you knew you are only judged for the effect and not for the code.
by desireco42 on 3/26/21, 2:52 PM
by mikewarot on 3/27/21, 1:20 AM
It is analogous to programming, the programming I learned in the 1980s, not modern software development.
On the plus side, once you level up a bit and get constructor robots there is a bit of copy/paste and undo capacity. You can load and save your work, and use more than 8.3 file names. 8)
On the down side, there is no source control, no commenting, very limited macro capability (blueprints). The debugging tools are limited.
On the analogy side, pollution as technical debt that causes bugs which will bite you, is spot on.
My favorite game style is to do Rocket Rush on an island... and see if I can get rid of the enemy and get self-sustaining nuclear power with Kovarex going... the it rapidly becomes boring.
by teekert on 3/26/21, 7:26 PM
by doix on 3/26/21, 2:50 PM
It would test their ability to read documentation and apply what they've read to problem. It also doesn't rely on them memorizing frameworks/API's etc. And it even in the early "levels" you can ask them to optimize for program size or cycle-count. Unfortunately no one would let me do it.
by arcturus17 on 3/26/21, 2:44 PM
by airza on 3/26/21, 3:01 PM
by ChrisMarshallNY on 3/26/21, 3:35 PM
I do that all the time. I haven't yet, today, but I'm sure I will soon. I'm working on some stuff I haven't done in a while. Googling has the extra benefit of helping me to do something that I already know how to do, but more effectively. I can always learn new stuff.
They also mention that people "don't have the time" for open source.
But I do have the time, and the result is a pretty massive portfolio.
It's been kind of amazing, watching it get ignored.
by asimpletune on 3/26/21, 11:17 PM
The trick from there is to be very well prepared yourself as an interviewer, and to make sure you’ve chosen a good problem. Good in this case can be anything, it should just be so thing that you know really well.
I like this approach a lot because if they’re great, then they should code the problem flawlessly in, say, 10-15 minutes. If it’s just an off day for them then there’s plenty of time to talk and see how they go about figuring out how to fix their first problem, ie they don’t know how to start. This also closely emulates how work is too. Most of what you have to do at work is actually quite easy if you really understand why you’re there everyday. If it’s not, then how do you go about fixing that? So this approach accounts for a lot of different styles.
Some people are just plain smart, and they’ll get it right away. Other people are very unassuming and just ask questions until they feel like they have the confidence to propose a plan.
Also you have a way of screening people who aren’t ready for the job. If they rush into a solution that’s bad and are defensive, then they’re not ready. Remember, this is an easy problem.
One thing I think is often left out when discussing interviewing is that it in fact is often a game, because work is often a game. You don’t get perfectly written asígnennos, delivered via email, where you just have to implement a function according to an interface. Instead, your job is to create business value for the company, while enjoying your life. I think a good interview format should actually lean into that. No, it’s not perfect, they don’t always perfectly test your coding skills, but real coding skills aren’t easy to test and are only part of what makes someone suitable. Ultimately, what matters is a.) can you recognize this is a game and b.) what trade offs do you make to win at the game, given your ability and character?
I think sort of easy, well chosen problems, based off personal experience are really the best for getting these answers from a candidate. Then again, it’s still not a silver bullet because it requires a really well prepared, interviewer who’s committed to giving the candidate a good experience and a fair shot at demonstrating what they can do.
I’ve had interviews where halfway I felt like the candidate was genuinely having fun, and others where we both knew this was the end for them. Like, the interview was so good at what it was designed for in either direction, yay or nay, both of us pretty much left agreeing with one another about how it went.
by kilbuz on 3/26/21, 2:32 PM
I don't see this mentioned in the article.
by commandlinefan on 3/26/21, 3:33 PM
by beaconstudios on 3/26/21, 2:53 PM
It really pisses me off that we as an industry don't broadly know our roots and the actual theory that we are putting into practice when we work. The "coding is engineering or art" debate? Easy; it's systems engineering.
The closest we get as an industry to teaching what's under the hood is to deal with computer science, which is great, but data structures and complexity heuristics are only a small piece of the puzzle.
It's no great surprise that there's such a big problem with cargo culting when most of us don't explicitly know what it is that we're doing.
by blitz_skull on 3/26/21, 3:08 PM
by ummonk on 3/26/21, 6:25 PM
by mariksolo on 3/26/21, 6:45 PM
by inopinatus on 3/26/21, 8:56 PM
Cramming these tortured metaphors into your hiring process is worse than whiteboard programming, something I didn’t believe possible until reading this steaming pile of numberwang.
by benlivengood on 3/26/21, 3:56 PM
One downside is that Factorio is almost immediately rewarding whereas programming generally takes a lot more effort for mediocre payoffs; hiring good factorio players will bias you toward people with enough focus for high reward behavior but maybe not so much for standard software engineering.
by kstenerud on 3/26/21, 2:56 PM
Any game sufficiently complicated as this will be akin to asking someone to write some example code in a language they've never used before, so you wouldn't be able to tell much anyway.
by SamBorick on 3/26/21, 3:34 PM
I also I threw together some thoughts about factorio and programming a few months ago:
https://dev.to/samborick/what-factorio-taught-me-about-work-...
by danbrooks on 3/26/21, 7:34 PM
by lukehutch on 3/26/21, 2:57 PM
by phendrenad2 on 3/27/21, 12:36 AM
by ilaksh on 3/26/21, 9:38 PM
by h2odragon on 3/26/21, 3:00 PM
Has anybody ever played less than 4hr of factorio?
by DelightOne on 3/26/21, 3:00 PM
by grogenaut on 3/26/21, 3:09 PM
by abootstrapper on 3/26/21, 8:07 PM
by christiansakai on 3/26/21, 4:47 PM
by noisy_boy on 3/26/21, 4:34 PM
by chmod775 on 3/26/21, 3:08 PM
Considering the average number of per-month applicants for software developer positions is probably less than one, you ought to have that much time.
by emrah on 3/27/21, 10:06 PM
by DesiLurker on 3/26/21, 6:50 PM
by fhifjfhjjjk on 3/26/21, 9:34 PM
by rubyist5eva on 3/26/21, 3:04 PM
by davelacy on 3/26/21, 9:18 PM
by aj7 on 3/26/21, 6:34 PM
by BoiledCabbage on 3/26/21, 2:53 PM
The author doesn't actually use factorio for interviewing, and thinks it world be a bad idea to do so.
A more accurate title would be "Comparison between Factorio and Software Engineering Concepts" - but that wouldn't have gotten it to the top of HN so fast, nor gotten so many comments so quickly.
by blacktriangle on 3/26/21, 3:00 PM
I have used and continued to use whiteboard coding in interviews. However I emphasize to the person that I am interviewing that there is no expectation that what they write on the whiteboard compile, or even be a real language. I'll usually do a quick demo and joke that my own white boarding looks like the bastard child of Python and FORTRAN. The point of the exercise is to have a design conversation and see if the person can think about algorithms or solving problems, not write valid code with a marker.
Do people still have the same level of disdain for this technique or am I misunderstanding the objections people have to whiteboard coding.
by kjjjjjjjjjjjjjj on 3/26/21, 8:07 PM
by ouid on 3/26/21, 3:40 PM
by andyxor on 3/26/21, 5:29 PM
The current situation is like every company coming up with its own version of IQ test, the randomness factor is huge. if current employees of any company were to simply retake their own tests most will fail.