Startup Series Part 1 Interviewing Engineers

There’s been a huge volume of literature written on the subject of startup methodologies and philosophies. This is a positive thing for the ecosystem - just as new software tools mean it’s simpler and cheaper to start a company, these methodologies allow first-time entrepreneurs to avoid many of the pitfalls of the past.

But I’d like to write a more practical guide - what happens once you’ve launched your MVP, iterated to a product that people actually want, and raised VC money to scale? What are the day-to-day challenges you’ll face and pitfalls you’ll encounter? Over the last 3 years at GoCardless, we’ve run into and subsequently dug ourselves out of many of these pitfalls. By making these mistakes, we’ve developed solutions that, at times, have dramatically increased our productivity.

Startup Series Part 1: Interviewing Engineers

The hiring process at technology companies has been the subject of a great number of blog posts, often written from the perspective of disgruntled candidates. They rail against what they see as capricious and ineffective techniques. This problem will probably exist as long as interviews produce unsuccessful candidates, but it is possible to design & run a process that will annoy a very minimal number of people, whilst selecting the best of the bunch to work at your company.

Once you have a good number of capable people applying to work at your company (a much harder problem!), selecting the best of them should not be a terribly difficult problem.

Over the last 3 years, we’ve seen around a thousand CVs, conducted hundreds of interviews, reviewed several dozen coding challenges and ultimately made 15 technical hires. As a result, we identified a number pitfalls in the interviewing and selection process. Usually we did this by stumbling into them head-first.

Define a process and stick to it

If you’re not absolutely clear on what each step of the interview process involves, and what you’re looking for at each stage, you’ll inevitably resort to weak, fluffy questions that don’t really reveal anything about candidate. If you throw a couple of your developers into an interview with no preparation, you’ve wasted 2 hours of engineering time at best. At worst, you’ve alienated the candidate and irritated your developers.

Instead, invest a little time defining what systems and processes you’ll use. Write down this process and share it with your team and candidates. You don’t need to invent this from scratch - choose someone else’s and adapt it to your needs.

Get an applicant-tracking system that stores CVs and the result of each interview (with detailed notes), so that you can compare candidates based on data. If you see a dozen developers in a week day, you’re unlikely to remember each of them by Friday.

Our process was pretty simple: CV screen, 15-minute phone screen, 2-hour coding exercise, 1 in-person interview to debrief on the coding exercise, 2 subsequent “final-round” interviews, with a mix of technical and “fit” questions.

Ultimately, the output of each stage should be “hire” or “no-hire”. Rack up enough “hire” votes from the different interviews, and you can make an informed decision. If you want, you can assign a more granular score, but it rarely yields a different outcome.

If you find yourself coming out of an interview and not being able to offer a hire/no-hire opinion, you’re asking the wrong questions.

Lack of preparation is unforgivable

Before the interview, write a list of good questions for each aspect of a developer’s skillset - with a backend developer, for example, you might want to focus on “web technology”, “databases”, “system design and architecture” and “algorithms”. If you have 2 in-person interviews, split the topics between them so that you don’t end up covering the same ground.

You can do the same thing for cultural & personality interviews, but you need to take care to avoid weak questions which either don’t reveal anything. Often, questions like “why do you want to work at a startup?” and “why do you want to work at this company?” are just an opportunity for the candidate to blow smoke up your arse. It’s a vanity exercise.

Instead, I like questions like “which parts of your previous job did you most enjoy? Which parts would you avoid in future?”. If you really probe for specific examples, it can tell you a lot about a person’s personality and motivations.

This list of questions should be shared with the company, and improved over time. I’d suggest it’s a key part of the company’s IP.

Relevant coding challenges are the best indicator of ability

In my experience, relevant, realistic coding challenges are by far the best indicator of ability. They should take roughly 2 hours, to be done in the candidate’s own time, and they should tell you:

- at a basic level, can the candidate structure a simple system?

- does the candidate write fluent, idiomatic code, in the context of whatever language is chosen?

- can the candidate select & use appropriate libraries?

- how much effort has gone into building the system?

For example, we sometimes ask people to write a client library for a real-world API or web-service in a language of their choice. I really like the exercise because there are such a range of solutions and opportunities for people to demonstrate exceptional ability and effort. There are also a number of great extension questions when you ask the candidate to talk you through the solution. How would you speed-up the client or the API? What are the different approaches to caching? How might you fetch a great number records simultaneously? If you were maintaining the API, how would you deal with very high volumes of requests? What are the various ways of dealing with API versioning?

We also tend to do some fairly lightweight algorithmic coding challenges in person - calculate row n of Pascal’s triangle or implement a basic sorting algorithm, for example. It’s important to provide a clear definition of the algorithm in question, allow the candidate use of a language of their choice and their own development environment, otherwise you’re not testing the right things. Interestingly, the optimal solution to many of these challenges involve recursion - a technique that seems to baffle some developers.

These shorter tests tend to reveal whether people can think under pressure, communicate clearly and debug broken code in a sensible way. For example, faced with a half-working sorting algorithm, a bad candidate will start changing variables at random. A good candidate would work methodically through the problem, ideally adding debugging lines that output the values of relevant variables at each stage. It’s worth noting that, while useful, these tests are not as good an indicator of on-the-job capability as the longer coding challenge. Many fantastic engineers don’t excel under this kind of high-pressure situation, and actually it’s not the kind of scenario they’d encounter on-the-job very often.

At the other end of the spectrum, asking someone to complete a 3 or 4-day project might be the ideal way to judge aptitude, but it just isn’t realistic in our experience.

Practical Tips

Finally, there are a number of simple practical tips that will make your life much easier:

Wherever possible, try to bunch your interviews into blocks of time. At the most extreme, I met 8 applicants for in-person interviews in a single day. It’s exhausting, but it avoids constant distraction and context-switching, and allows you compare candidates more effectively. 

Have one person in the company who is responsible for receiving & screening CVs. Do this in blocks of time, once or twice a week. Otherwise, it can get overwhelming 

Be clear what you’re looking for from CV- and phone-screens. You could be wasting a lot of time. Write a checklist after you’ve seen a dozen or two CVs. We should have rejected way more people at this stage.

Whatever the outcome, respond to candidates quickly and constructively. It makes them feel valued, and it’s simple good manners.

Finally…

Even using this process, you’ll make bad hires. It happens, and it’s not the end of the world. Fire fast and move on. Overwhelmingly, the bad hires we made were due to poor preparation, or, at times, wilfully skipping most of this process. Hiring well is one of the most important things you can do for your company. Don’t neglect it!

I’d love to hear your thoughts - please comment below, or follow the discussion over at Hacker News