How to evaluate the quality of a software engineer: 8 questions to ask

Hiring a good software developer is hard. It’s not only a challenge for the candidate, but also for experts verifying the results of the coding assignment & conducting a technical conversation. While every position differs in terms of the technological stack, the complexity of the problem & the required experience, in the following blog post, I want to share a set of universal and specialization-agnostic questions to ask both while evaluating the candidate’s solution and when discussing technology in the interview. It is a result of a few years of experience — both personal and my colleagues’ — evaluating aspiring candidates for various technical positions at SoftwareMill.

Introduction

For the last few years at SoftwareMill, besides everyday software engineering, I’ve been involved in dozens of code reviews & technical conversations with candidates of varying maturity, experience, and different software development positions they applied to. Over that time, I learned much about people’s approach to the coding assignments, their attitude towards discussing the technology with others, as well as about the other side of the coin — us, people giving the candidate the final mark.

Turns out it is not only difficult for the candidate to always use the right tools and know the correct answers, but it’s also not an obvious thing for “examiners” to know the right questions in the first place. Technology changes all the time, so one day I was asking about how the digest cycle works in Angular.js, while the other — what is an effect type in functional languages like Scala.

My point is that besides assessing the language- or framework-specific knowledge, there are some universal qualities of every software engineer and solutions they produce, which we should be looking for. Over time, I came up with a list of eight questions — four “to the code”, and four to the candidate — that I find helpful in the process. I don’t want it to be taken too strictly, though: this is a set that works for me and at least a few of my colleagues, but we also ask other questions, mostly coming up naturally in the course of the code review or conversation. What I find important for me may only partially work for my readers, and I’m very positive about the fact that one could come up with a different, though equally valuable list of what to ask about.

Evaluating the result of a coding assignment

When I verify the result of the coding assignment, I try to sense the candidate’s maturity & attitude. A good software engineer's high-quality solution can check off a few points almost immediately, while several negatives can trigger digging deeper into the author’s reasoning. Over time, I observed that in this branch of the tech industry some flaws are common despite the experience and I tend to investigate those in the first place.

Question 1: Does it work and do what it’s supposed to do?

First and foremost, I want to know whether the solution follows the specification. Does it work without errors, and could all required features be used?

This is an essential step. If the provided application doesn’t run properly, throws errors in runtime, or just doesn’t do what it was supposed to do, it’s a major red flag. There’s not much of a value in shipping a solution that either doesn’t do its job or does it unsafely. This can tell a lot about the candidate’s understanding of the task, as well as familiarity with the tooling.

I stumbled upon solutions that were incomplete. But I also stumbled upon extra features that were out of specification. While it’s fine to add a nice styling to the UI layer, spending dozens of extra hours to make the app look good on a smartwatch (when it wasn’t even asked for!) kind of misses the point of the assignment.

A good solution should express the programmer’s understanding of the challenge and care for quality. There’s hardly anything else that can say that much about the attitude towards the job they apply to.

Question 2: Is basic tooling in place?

This question is mostly about assessing whether the programmers care for their precious time. I believe that there are tools for almost every technological stack that allow to not only automate some tedious tasks (like code formatters), point out common pitfalls and defective code during development (linters, reasonable compiler settings), but also let avoid "re-inventing the wheel" (libraries and project scaffoldings). Put together, in a high-quality solution, time-consuming and repeatable steps are taken care of with dedicated tooling. In the end, the point is to make life easier for the original developer, as well as for everybody who’s about to work with the code later.

Well-adjusted tooling is an expression of the author’s self-respect for their precious time but also a sign of care for some well-known standards. Use of linters, code formatters, and recommended compiler settings is like establishing a contract, a set of guidelines the project will evolve with. It is much more ergonomic to follow the rules (and validate at least some of them automatically) than to make progress in chaos.

The use of one particular tool always gets my special attention, which leads to…

Question 3: How’s GIT?

Version control is amongst the most crucial points to check on my list. I believe that by keeping a clear history of changes, the programmer can make it much easier to grasp the solution. Putting a meaningful commit description is like leaving a note for your future self (not to mention potential teammates).

When I open a project, I always look at the GIT log. For recruitment assignments, I don’t care much about sophisticated flows, but I do look at the number of commits and their associated messages on the branch designated the main.

What is always surprising to me is that many experienced software engineers still fail at that point. I’ve seen people using commit messages along lines similar to"quick change" or "minor fixes", often consisting of whole dozens of changed files. This is simply not sustainable in the long run and there’s a high chance such style will strike back next time the developer looks into the history, searching for the source of an error. It doesn’t take much to form a habit of committing a set of discrete, related changes, doing it often and clearly describing what the change is about.

Careful attention to clear and well-described history expresses an intention to leave a good trace of every — possibly breaking — step in the project’s evolution. By leaving such a well-formed artifact on the timeline, the author gives a good sign of care for order, regularity, and respect to other — potentially future — team members.

Question 4: Can a candidate’s choices be justified?

While many coding assignments suggest the use of certain technologies, I believe a special emphasis should be put on one idea: candidates should use what they know best.

It’s good to learn new technologies, but would you start a serious project while still learning a language or framework? Yes, it is theoretically possible, at least while there’s somebody well-experienced in your team, but stepping into a new territory almost always means taking a few steps back (and even becoming a junior in that particular scope). It’s easy to spot a candidate’s first use of technology X in the coding assignment, which is often justified by:

I just wanted to use the new, shiny thing.

Over time, I learned that robust solutions are achieved with expertise in tools of choice. Switching into the yet unknown often leaves the author overwhelmed with the new architecture, quirks, and specificity of use, especially when the task is time-constrained.

The question to ask here is whether the candidate’s choice regarding the technological stack of the solution can be well justified and “defended”. Did it really make sense to use the bleeding-edge features (possibly not yet widely supported or known), or could it be solved in a more traditional, battle-tested way? Maybe there’s a lot of over-engineering? Are there errors, missing pieces, or other pitfalls that could be avoided if only the programmer didn’t have to invest the time into learning a tool previously not known?

There’s time for learning and there’s time for using what one already feels confident with. Relying on priceless experience is a sign of certainty, maturity, and respect for the challenge.

Conducting a technical interview

Code review is an essential step but it can’t tell everything about the candidate. There are a couple of questions I almost always want to ask to engage in a conversation. Again, I try to keep those maximally agnostic in relation to the technology of choice.

Question 1: Do they know what “the billion-dollar mistake” is?

But first and foremost, do you? ;-)

Null references pose a threat in almost all technological stacks. Historically, there’ve been many ideas for how to improve type safety: from language-level features & strict compiler settings to libraries promising to solve it once and for all.

Yet, I sometimes get the impression that this universal issue, spanning across a multitude of different programming languages and paradigms, is still underestimated by engineers. Blind trust in external APIs promising to return data of given shape (but throwing null instead), relying on poorly-typed libraries in dynamic or loosely-typed languages, or (the worst!) using null references deliberately (when it’s long been considered harmful for the technology of choice) — all of these raise an alarm.

I often ask candidates whether they are aware of dealing with potentially missing data. I want to know how seriously they treat the “threat” and what their take on improving runtime safety is. Whether it’s defensive programming, using language-level features like optional chaining, or resorting to the constructs taken straight from the world of functional programming is a secondary measure.

There’s a good reason why null references have been aliased the billion-dollar mistake, and I’m convinced that the awareness of the fact is a strong sign of experience.

Question 2: Can they come up with creative testing scenarios?

To be honest, I’d rather speak out loud about the approach to testing first than have a suite of tests covering every possible combination of inputs and associated outputs for every piece of the API out there.

One can write thousands of unit tests, but still miss the point of verifying the end-to-end solution. This is one reason why over time I shifted towards discussing testing scenarios a lot. I’d rather hear ideas about how to plan and write test cases for core functionalities than see walls of code that look like generated by a machine in some automatic process.

It’s not an argument against thorough unit tests. It’s about making reasonable decisions and taking the perspective of the end-user. Again, frameworks and tools are mostly less important than good ideas.

Writing good tests is a difficult art and I think that creative thinking and taking responsibility for provided solutions are essential, while engineers should also be able to express the familiarity and understanding of the problem they are trying to solve.

Question 3: Can they talk in an engaging way about technical challenges?

One of the most valuable traits of a good software engineer is the ability to talk about technology in an engaging way. Openness to exchanging arguments (both for and against) makes a good expert, well aware of the advantages and drawbacks of a discussed approach.

I can’t set any strict rules about the topic of conversation. Ask about how they found a solution to an interesting problem or performance issue. Maybe they’ve just finished a major technological migration? Are they exploring new territories out of their main area of expertise?

Talking about technology interestingly, with a dose of criticism, is a sign of maturity. I feel confident partnering up with people having opinions and looking to broaden their knowledge by exchanging ideas with others. It’s always good to work with somebody you could learn from. Speaking of that…

Question 4: What learning sources could they recommend?

This question kind of comes naturally from the previous one. Being a great software developer means you constantly have to learn. While there are hundreds of sources out there, there’s only a fraction I could really recommend with a clear conscience.

What are the ones they can suggest?

Aspiring for an expert’s position means — among many other things — being able to share your knowledge, as well as pointing others in the right direction when it comes to learning. It’s a priceless trait I value highly with my teammates!

Summary

The art of reviewing somebody else’s code, as well as coming up with the right questions to get to know the candidate well, is difficult. The ones I described in this article have been giving me the most thorough insights so far.

However, I’d like to emphasize again: this list should not be treated as a strict handbook — instead, I hope to inspire my readers next time they ask themselves: How do I spot a good software engineer?

Learn more about how we hire software engineers in this article by Tomasz, our VP of Engineering.