Is there a market for Rust code reviews?

I hang around the #code-review channel of the Rust community discord and the Rust User Forums quite a bit, and you often see extremely skilled Rustaceans responding to requests for code review or providing feedback.

Newbies often use code reviews to learn about the language and improve their skills, but you don't really hear existing projects or companies making use of independent code reviews.

For people using Rust professionally, would you consider hiring 3rd parties to do code reviews/audits of parts of your codebase? And if not, why?

I feel like this could be quite beneficial for both parties... Companies can identify bugs/soundness issues and start paying off technical debt without diverting resources (i.e. permanent staff) away from existing projects, while also providing developers with a side hustle that makes use of their existing skill and experience.

Technologies like the crev project also provide a framework for carrying out code review and building a web of trust, which could be quite useful here.

1 Like

I have no idea, but the topic is interesting!

I can attempt to give you a perspective from the other side of the question: I work in crypto as a security engineer. Many companies in the field are using Rust for building blockchain-related software and to utilize its WebAssembly tool stack. Due to the nature of the industry, with money typically always on the line, many of the reviews that we perform at my company take place in private. There is absolutely no shortage of professional Rust users who take advantage of code reviews and security assessments.

1 Like

yeah I was thinking that if this happened, a lot of it would need to be done behind closed doors and with NDAs. It's not often that a company will want to publish their IP to GitHub.

How would you normally go about something like that, though? Would you hire a 3rd party contractor explicitly to audit code, or does it just happen as part of contracting out certain work?

That's correct, typically there's a written contract which outlines the terms of the engagement. Some companies share their code openly, some use private repositories that they share with the company, others will send an archive of the source code in question, there's multiple ways to go about it. I believe that audits are delegated to third party companies in order to keep things honest, at least in this situation. It's a bit more difficult to take someone's word on a particular piece of code just because they wrote the lines and they couldn't find anything wrong with their system. There's also companies who don't particularly have dedicated engineers, so they contract the work out to a separate party and then might want some assurance on the work that was done.

1 Like

I guess it says a lot about the different industries, but it sounds like you've had a different experience to the people on the Rust subreddit.

The vast majority of people who answered my question on Reddit were saying they wouldn't solicit 3rd party code reviews, preferring to keep everything internal or saying it wasn't a good way to spend resources.

The paid third-party code reviews I've been involved with have mostly been security/process audits for compliance with specific laws or standards. Usually the code review is just one part of such an audit. If you want to find work doing Rust code reviews, you could try to connect with a company that does such audits and that has occasional need for Rust expertise which they can't fill internally. (Or you could broaden your own skills to be able to do more complete audits beyond reviewing Rust code.)

It definitely depends on the individuals and the use cases of the language. A lot of the people who might take the time to respond to posts like these are most likely people who prefer to use the language. I've been using Rust as one of my main languages since 2013, so I have a fair amount of experience with it. I've seen plenty of capable people who work on open source projects over the years as well. Outside of the group of enthusiasts, any time I would bring up Rust to friends or other people from a project, it was typically met with ridicule and opposition. Over the last two years however, I've had people in the field start to ask me questions about it, and have started to see it pick up more in industry use. I've already had to look at a number of different projects utilizing thousands of lines. To wrap up my observations: I believe there is a definite need for Rust expertise, code reviews and security assessments in the professional space, but not so much with open source and indie-level projects. The community has done a fantastic job of being welcoming to newcomers and providing introductory material.

1 Like

Oh, yes -- definitely.

Security audits by external consultants make a lot of sense to me: fresh eyes can find fresh bugs. It's notoriously hard to find (security) bugs in code you wrote yourself since you already did a lot of work to convince yourself that it's safe and correct.

I've seen startups bring in consultants a few times. In one case, the company decided to switch to Go, and they hired external consultants which helped write the foundation for the new code base. The consultants didn't do code reviews as such — instead they were there to get the existing developers up to speed faster than they would have been able to alone. In another case, consultants were brought in to improve the story for building Haskell with Nix on Windows. This required specialized knowledge in an isolated corner of the development life cycle.

However, I don't think most companies would want day-to-day code reviews to be done by an external company. Some reasons for this:

  • Good reviews require a lot of context. External reviewers can comment on style issues, but it's harder to discuss and improve design decisions as an external reviewer. This makes the reviews shallower than reviews by good in-house developers.

  • Building up a deep knowledge of the code base should be a top priority for a development team. This helps with maintenance — in the first company above, we did have certain parts of the code that nobody liked to touch because it was "weird" and nobody understood it. This was a corner of the code written by consultants who no longer worked with us.

So in my opinion, external consultants only make in the short term: perhaps time-to-market is so important that the company prioritizes speed over building up in-house knowledge. Or perhaps the problem that need solving is something outside of the core business, such as building Haskell on Windows.

3 Likes

The main concerns I'd have about such a service are that first, code reviews are very much subjective, even with good coding standards in place, different developers can often have different opinions and in many cases, neither of them are 'wrong'.
Second, just as with 'penetration tests', people pay for a service because they want a desired outcome, or the perception of a desired outcome either for themselves or their downstream customers. Will the service provider feel obliged to always have suggestions and improvements? What happens when most reviews pass without the need for comment or alterations? Does the service then become unnecessary but then the dev knowing their code won't be reviewed, starts to relax, this making the need necessary again?
With pen testing, I've seen some really bad things get through, but, of course, the customer wants to 'pass' the test and show that their product is 'secure', so in this case the service provider feels pressured to 'pass' more sites than they fail, otherwise customers just go to others for an easier pass, claiming that the tests are too strict. Except those rare customers who have the tests because they genuinely want to improve.
So, with the code reviews, could good reviews with good feedback also be contrary to the customer's desire to just have a 'confirmation' that their code is good, rather than a genuine desire to learn and improve.
Phew, sorry, crazy long reply I know, just my opinion on the ways this could go :slight_smile:

5 Likes