Checking *which* compile-time errors happen?

[rauschma]

When including TypeScript code examples in documentation, I found two features useful (supported by a checker I wrote):

#1: checking inferred types

// %inferred-type: number
let num = 123;

#2: checking type errors

// @ts-expect-error: Object is possibly 'null'. (2531)
null.myProperty

As a result, readers see what is going on and the checker ensures that there are no typos. I first extract the example code and then check it.

Is there a way to perform similar checks in Rust code?

[Heliozoa]

What you're describing are called doc tests in Rust and you can read about them here: Documentation tests - The rustdoc book

For the first one, you can just annotate the type:

/// ```
/// let num: i32 = 123;
/// ```

For the second one, you can start the code block with compile_fail to create a doc test that passes if the example doesn't compile. There's no good way to specify which error it should result in AFAIK.

/// ```compile_fail
/// let num: i32 = "hello";
/// ```

[Cerber-Ursi]

@Heliozoa

There's no good way to specify which error it should result in AFAIK.

And if it's desirable, one might want to reach out for trybuild and create a full-blown compilation UI test.

[CAD97]

On nightly, you can actually test which error code is emitted in a compile_fail test. On stable, this just silently doesn't do anything. IIRC it's doing ```compile_fail,E0123, but I basically never do this, so I'm mostly just guessing.

Testing the exact form of an error is fundamentally fragile, though. The compiler is constantly tweaking and improving diagnostics, so if you're testing what the error message is, updates to the compiler may require re-"blessing" your tests. Error codes are a bit more stable, but it's still explicitly considered unstable what specific errors an invalid code snippet causes.

If you understand the limitations, then trybuild is the state of the art for snapshot testing of compiler diagnostics. It doesn't do anything to directly^[1] help with testing the examples in your documentation, but it does allow monitoring error message quality.

---

1. You can get close if you include_str! the test file and/or error message snapshot file into the documentation, but this means the entire file has to be included, and you can't trim the example. Better IME to just put regular comments begging people to keep the doc example in sync with the tested example if you want to be selective in what you're including in the docs in any way. ↩︎

[ds84182]

I've also run into this, slightly. I think it would be best if you could specify something sorta like a "diff" in the rustdoc source for a compile_fail doctest.

/// ```compile_fail
/// - let num: i32 = "hello";
/// + let num: i32 = 123;
/// ```

Right now, compile_fail is insensitive to code changes. After refactoring a crate, any existing compile_fail tests will continue to fail, but for a different reason than intended.