General advice on the use of nightly

I am embarking on a new embedded Rust project. I intend to use the espressif ESP32-C3 microcontroller (includes a RISC-V core with RV32IMC ISA). The guys from Ferrous System have developed a training based on it.

I was positively surprised that there is support for std using espressif ESP-IDF (Tier 3 ); however it requires Rust nightly. Stable Rust can be used with no_std for RV32IMC (Tier 2).

I am asking for general advice on the use (or not use) of nightly, under the following plan (feel free to also give constructive criticism on this plan):

  1. Although I intend to release the code as FOSS in github, I do not intend to publish it in crates.io (at least not initially). The binary would be deployed on the embedded target; the main development will be done by a small team behind closed doors.
  2. The project would be split into two types of modules: std for anything where using ESP-IDF (requires nightly, remember) may make our life easier (e.g., bluetooth communication) and no_std for everything else like the business logic and simple device drivers (i2c sensors, etc.)
  3. If we use nightly, we will stick to a specific version during the whole development (possibly several years)
  4. This is not a toy project but something that will be deployed on commercial embedded devices (although not safety critical)

My general questions are:

  • is the use of nightly discouraged for such a project (non-safety critical embedded for commercial applications with multi-year product life-cycle)
  • what are the main cons (besides unstable Rust API) of using nightly, for example Updating “nightly” + “clippy” is a nightmare.
  • The way I understand it, if I stick to a specific nightly version, many problems with unstable API could be avoided. What are the disadvantages of using a nightly that is several years old (besides bugs not being fixed)? Say, I start the project today with nightly-2024-11-21, and after years of development we ship devices on 2027 (code compiled using the same nightly version from 2024).
  • are there best practices specifically when mixing no_std with stable for the main development, and std with nightly for specific functionality (isolated on a module or crate)?
  • any other hints or general advice is welcome

Note that there's a risk here in that you won't get any security fixes.

I'm not sure what Ferrous offers, but for the commercial offering it might be worth getting a toolchain from them with some support commitment -- and maybe consider updating not every 6 weeks, but maybe every 6 months or so.

2 Likes

I think @scottmcm's suggestion to upgrade periodically is important to reduce reliance on unstable APIs that are removed or changed. Doing the changes incrementally seems less risky to me. Plus I would want to leverage new features as they are added.

3 Likes

If you only use unstable features to build the standard library for this target and not in the code you write, it should be pretty painless to regularly update nightly. Every once in a while you might hit a bug, but in that case you can report the bug upstream (if it hasn't been already) and rollback to an older nightly. This also has the advantage (mostly for rustc, but also somewhat for yourself) that you provide some extra testing to rustc to prevent those bugs from hitting stable. Most of the time rustc bugs show up as crashes, not miscompilations, so you can be reasonably sure that if you hit them, you will know that you know about the bug before it lands in production, though it does occasionally happen that there are silent miscompilations. Especially shortly after we update LLVM.

2 Likes

My general advice would be to avoid nightly in production code. You can mitigate some of the issues by pinning your compiler, but if you care about stability your likely do that for a specific stable rust version too.

My pain point of nightly was the usage of too many nightly only features. The good news is if you're disciplined you can just not depend on any unstable features; then switching toolchains should be easier.

1-2 times we'd have builds break due to compiler bugs when updating the nightly version, but we either updated crates or the compiler ( and downgraded the compiler in the mean time).

These were very rare, but we decided we didn't want to spend time figuring out unstable compiler bugs.

That said, not being able to use std would also be quite limiting. I'm not in the embedded context though.

1 Like

The main issue with long-term compiler pinning isn't just that you'll be missing out on updates from the compiler or standard library, but you'll probably also be missing out on updates to third-party crates, since those crates are likely to start using features that require new compiler versions (eg. the new trait solver or borrow checker, once those are stabilized)

The other issue: trying to mix stable and nightly compilers isn't a great idea, and its only really possible if you shove everything through a C ABI (since the Rust ABI is not stable across compiler versions)

Larger projects that use nightly like clippy and rust4linux have the advantage that, if there's a feature they use heavily, it is unlikely to be removed without coordination from the rust team, but smaller projects, especially those not published on crates.io, are unlikely to be considered as a factor when removing unstable features.

3 Likes

I use Rust nightly on Raspberry Pi. I didn't notice any glitches and yes, I use Rust features I can't use with the stable. I observed couple times Rust didn't delete couple temp files after a compilation, but it's a very minor. I didn't update nightly because it works as expected for me.

1 Like

Thank you all. Maybe my original post wasn't clear enough, but @bjorn3 summarized my intention:

If you only use unstable features to build the standard library for this target and not in the code you write, it should be pretty painless to regularly update nightly.

The feeling I get from the answers is that in my case it may be okay to use nightly, provided I am able to update it from time to time. That may lead to bugs in some cases.

I personally can live without std in the embedded code I write. However, ESP-IDF offers very attractive high-level hardware abstractions that would speed up the development, and thus the need of nightly.

In the end, what I am trying to estimate is the tradeoff between:

  • use Rust stable, which means only no_std code, which means we have to write up-front a lot of boiler plate code. In the long run, problems are very rare (or easy to fix) if we update compiler or other dependencies.
  • use ESP-IDF, which provides many high-level abstractions and std (nice), but with nightly (potentially not so nice in the long term). This would lead to faster development, but updating compiler/dependencies may lead to problems which may slow down the development in the long run. This is what @richardscollin was referring to:

These were very rare, but we decided we didn't want to spend time figuring out unstable compiler bugs.

I would be very thankful if anyone can provide further insights (from experience) on how to better "guesstimate" these two tradeoffs:

  1. Rust stable: higher upfront coding effort (only no_std), but possibly less effort on the long term due to smooth updates
  2. Rust nightly: higher development speed (std and ESP-IDF), but possibly more effort on the long term due to failed updates

Thanks a lot! Cheers.

I want to bring up a point that is often missed on this discussions around nightly vs stable vs other options to use nightly features: If you decide to use nightly, and pin against the specific version that was used to cut a new beta (and eventually the new stable), you need to be aware that there might be bugs that are fixed on either beta (or worse, stable) that will never be backported to the nightly you pinned to, while newer nightlies will have the fix. This is not necessarily a big problem in practice, particularly if you are not updating nightly too often and testing thoroughly when doing so, but you want to be aware of it.

4 Likes

The Rust ecosystem has quite a different philosophy of how to offer stability — instead of not changing anything and keeping old bugs, it rolls forward fixing bugs as soon as possible. I know this is very different how serious development is traditionally done, and may seem shockingly irresponsible, but it does work in Rust's case.

The compiler is quite stable and reliable, so updates of the Rust compiler are generally low risk. In my experience over the last 10 years, I don't recall any Rust release that would be dangerously miscompiling code. I needed to delay updating or skip a Rust release about once per 20 releases, and these were mostly due to performance regressions. The compiler sometimes adds accidentally quadratic code, and compilation times on certain code patterns can shoot up from seconds to minutes (and these problems of course get fixed within a release or two). The second type of breakage that happens sometimes is Rust breaking some popular crates.io crate, usually by removing or changing a nightly feature, or new standard library additions making type inference fail (type inference never picks wrong types, but may give up when it finds multiple possibilities when there previously was only one). When it affects your code, it's typically fixable in minutes. When it affects dependencies, it depends how fast they release, but in Rust/Cargo it's easy to fork or vendor deps and patch yourself if necessary.

If you need to freeze the version of the compiler, I suggest freezing it as close to the final release as possible, rather than at the start of the development.

If you stay 2-3 releases behind the latest Rust version, you will still be able to use the crates-io ecosystem easily, and there will be enough time for others to find any issues with the latest Rust, and you'll be able to skip over any releases if they turn out to be problematic (which should be very rare).

Most crates support up to 1-year old compilers. If you have compilers older than that, you'll usually be on your own, without support from Rust, and without support from your dependencies.

5 Likes

As for nightlies themselves, they're usually okay for day-to-day programming, but they have some bad days (if the compiler is in the middle of a big refactoring or enabling a new feature, there can be regressions in nightlies).

There is an open secret environmental variable [1] that enables nightly-only features in the stable releases of the compiler. This is probably the best option if you need to freeze on properly tested reliable version of the compiler, but still access some not-yet-stabilized feature.


  1. RUSTC_BOOTSTRAP=1 ↩︎

3 Likes

Thanks, awesome statistics!

I’m still unsure what the best way forward is in my case. I’ll personally prefer the Rust stable way, but I’ll give nightly a shot, if only to acquire some first-hand experience. It might be not as bad as I imagine it.

Thanks again everyone!