Interview Process: Code Challenge

I've been tasked with hiring several Rust engineers and I'm looking for a way to go alleviate some of the typical challenges of hiring. I know there will still have to be the get to know you phone call and obligatory resume review by the non-technical folks.

However, for the technical side of things I would like to try something different. I've been thinking about what I would like regarding a technical interview and what would make me want to work for that company. I would like some opinions on my proposed "code challenge". Is it a terrible idea? Good idea? I know some will probably hate it, but why?

Trying something that I hope will help solve the hiring problem for a small company that can't afford to make mistakes in hiring engineers.

1 Like

Starting with the positive: it does seem like a fair effort at finding something relevant to measure.

Personally, however, I see two flaws. First, if I am a regular Rust developer and I have several of my own Rust repos, I don't see why you'd need further evidence that I can work on such projects. The "pick two" seems a bit of an arbitrary time-sink. Consider that your applicant may have no idea how hard or easy any of these issues are, or how much time and expertise it takes to solve them. Answering those kinds of questions is yet more work. And then what if they spend all that time on an issue they can't resolve? You asked for resolved issues.

Second, the idea that this is fair because I will get something of value out of it is questionable. If it were valuable for me to contribute to these repos, why wouldn't I already be doing it? It still comes off like you're looking for free work, because (A) as far as I know, you're running your business on this code, and (B) you're asking people to do real work here and not paying them for it. "Exposure" is not a respectable payment for real work.

For the most part, if you're going to be looking at someone's contributions, you can just ask for a link to their account and see what they're doing. If you mention that you have a very strong preference for people who have made contributions like this in the past, and simply ask them to link to those contributions and/or talk about them, it will probably come off way better.

Anyway, that's just my personal opinion.

4 Likes

Thank you for the well reasoned response. It is difficult not to come across as looking for "free work", however, as much as I would like to be able to do the trial work period for everyone it's just not economically feasible. However, without the person working in a larger project how do I determine how well they would work within an already established code base? I asked them to commit to 2 different projects because I wanted to see how they would approach each one. It would give me as the evaluator a chance to see them work as they move among groups.

I do believe that it would be of value to the community and possibly that some people would receive value from such a task. Personally, I would, but I do understand that others will feel as you do.

I will:

  • update the text to say pull request, instead of resolved issue.
  • remove the "something of value to you" idea from the coding challenge

Regarding the "I'm already a Rust developer" flaw. A large number of the people whom I've asked for their GitHub / GitLab / SourceHut / etc. do not have code which I can review. I don't want to disqualify people because they don't have open source repositories. This is my attempt at resolving that. I think that part should be made more clear. If you have a bunch of code that I can review, then this is something that you would likely never see.

I've also specifically called out the items which are Marker Trax dependencies. I would like this to be as transparent as possible.

How will you rank candidates?

Let's say:

  • Candidate A makes a pull request which fixes issue A in repo A
  • Candidate B makes a pull request which fixes issue B in repo B
  • Candidate C makes a pull request which fixes issue C in repo C

You can only hire one, so how would you choose?

That's not relevant, they're not solving the hiring problem algorithmically. The people who make the decisions will use whatever criteria suits their business, and that doesn't require any advance knowledge of how to rank people.

3 Likes

That will have to be a decision which involves the other parts of the process. If they all are successful in solving issues then that's a great thing. Honestly, I would pick the one whom I feel like we would work best together. It's a small team and I believe in that context it's important because they're "no where to run" so to speak.

Also, given that they've all shown aptitude I would certainly do what I can to market them to others seeking engineers.

Let me ask you the same question, what is your criteria?

Thank you for making me think about this.

I don't know, to be honest.

I've worked at a couple companies that gave real-world coding exercises to interview candidates as part of the interview process and the only way we've figured out to make the process fair for all candidates is to create a private custom project with the same bugs in it just for the interviews and then we ask every candidate to fix the same bugs.

2 Likes

I've done the same before. Just seems so wasteful, mostly of the applicant's time. My initial thought with something like this is that time would be put to good use, for a good cause. I suppose an offer of both would make sense.

This sounds like it could be quite a good compromise! From a candidate's perspective, it disavows the notion that you're trying to get them to do free work for you, while still allowing them to give themselves some value if they chose to (and if they value that kind of contribution history).

I would prefer this to the usual coding problems (which I don't dislike). I would only have them choose one issue though. You're already asking them to a do a moderate amount of work for that one issue, and this is just determining whether or not they're worth talking to. Unless you're going to hire them on the spot after completing two issues, you're asking too much.

I really like this idea though. I think it's better for all parties and it also helps out the community.

Thank you all for your input. This is a difficult thing to get right. I've given a lot of thought to the compromise of giving the candidate a choice of a private, staged project and the open source contribution model. I don't think that it's the right compromise.

The work relationship is like any other relationship, either party can choose to continue, or choose to sever the relationship. Also, as with any relationship openness and honesty the principles that will make it strong. With those as my guiding principles I've re-written the challenge. I hope it reflects more to the principles of honesty, openness and freedom to choose.

I respect the thought behind getting candidates to fix a couple issues in open-source projects and really like the premise. It promotes contributing to the community and lets you see how they would behave in the real world, outside of a contrived interview where you see the candidate do 20 or 30 minutes of problems on paper or a whiteboard.

In the past one of my exercises was to give the candidate a massively simplified scenario from our codebase, in this case it was moving something along a 1D axis while monitoring a couple digital inputs - you could implement it in about 100 lines of Rust. This helped weed out a lot of low hanging fruit, but because you aren't doing things in the real world, you don't really get to see how well they can actually write code or understand/create layers of abstractions.

I think interviewers have been struggling with this problem forever, though... It's really hard to predict how they competent someone and how well they'll fit in without actually employing them for a couple weeks and continually reviewing their progress.

I feel like letting them pick random issues from random projects has a couple flaws, though:

  • Not all issues are born equal - so how do you quantify and compare the quality of the PR? You could spend 20 hours tracking down a really subtle soundness bug and only end up adding 1 or 2 lines of code, but at the same time someone could spend an hour or two implementing something relatively straightforward and easy and make a 500 line PR.
  • Not all projects are born equal - by their very nature, open-source projects tend to evolve at their own pace. By making PRs to projects outside of your organisation, you lose a lot of control over the review and third parties (and personalities!) now become entangled in your interview process. For example, a maintainer not replying for several weeks because they are busy will reflect poorly on the candidate because they never got the chance to go back and forth with the maintainer and do a proper review.
  • How much work are you expecting them to do? Candidates probably won't want to dedicate more than 3 or 4 hours to preparing for a job interview, but to do the task justice and create a PR that's indicative of the developer's quality you'd probably need 10+ (e.g. 3-4 hours learning the project structure and identifying the issue, 1 or 2 hours actually fixing it, 2 or 3 hours going back and forth with the maintainer so the PR is at a point where it's ready to merge). The time would be spread out, but that's still a lot of hours and unpaid work.

This is an interesting question. If your organisation values contributing to open-source a lot and the candidate has only ever written code inside their company, does that align with your values?

I second this sentiment.

For example, I already maintain several popular Rust crates, am a prolific member of the user forums, and I frequently write in-depth articles featuring Rust on my blog several times a month... with a wealth of information already freely available, what benefit would 1 or 2 extra PR links give?

2 Likes