running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests arif
running 1 test
test src/lib.rs - foo (line 1) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s
Which is wrong IMO. the test should panic.
I believe this should panic, and my realworld doc ests just aren't able to find any code that I put behind a #[cfg(doctest)] gate.
allows you to have certain items only appear when it's collecting doctests, so you can utilize doctest functionality without forcing the test to appear in docs
So maybe it's only there when "collecting" the tests and not when compiling them? This seems like a missing feature because my usecase seems very common)
Can you explain why would it be very common? Doc tests exist to verify precisely the functionality which end-user would use. Why should they deal with separate version of the crate?
This doesn't sound very logical: why would you provide someone with example which wouldn't work as described in his own code? How is that ever useful?
It's there when you run doc tests and not there when the actual crate is built.
The text looks clear enough to me: you can not do some things in regular tests but only in doc tests (compilation errors, mostly).
But if you are creating a doctests then appropriate part would be added to both documentation and your crate, too.
Solution (a tiny bit hacky): mark an item with #[cfg(doctest)] and then it's interpreted when doctest are run, but not included neither in the documentation nor in actual code of your crate.
So in many of my crates I very commonly provide structs, or helper functions only available in tests that reduce boilerplate, mock, or provide dummy values etc. I can't get access to any of this in my doctests. I have to write WET code if I want them in my doctests.
why would you provide someone with example which wouldn't work as described in his own code?
I'm not providing him an example, i'm writing a doctest to prove a claim about my code's behaviour.
Solution (a tiny bit hacky): mark an item with #[cfg(doctest)] and then it's interpreted when doctest are run
I don't understand... isn't my post expressing the very fact that I can not do this?
You still haven't explained why would you want to access internal parts of your crate in doctests.
If you worry that someone else would start relying on that machinery then you can document is as unstable and say that you don't see it as breakage of symver to change API for that debug feature.
You still haven't explained why that boilerplate is not part of your regular public API. Or debug API.
Presumably if these helper structs help to test public API then they would be useful for someone else, too — and they wouldn't depend on #[test] flag.
Or if they are parts of internal machinery of your crate — then why are these doctests and not regular test?
But why you are using a doctest for that?
Of course you can! But these “doctests”, like any other doctests, would verify public API of your crate.
I'm not saying that the need which you describe doesn't ever happen, I'm just surprised that it's characterised as “very common” while for me it looks as if it should be very uncommon.
Documentation tests are documentation first. Their primary function in existing is as a documentation example.
They're compiled and executed to ensure that they're not an inaccurate example, but this is to test the example, not the code used by the example.
The way of testing your code's behavior is #[test].
That said, it's not impossible that you could want access to mocks or similar from tested documentation examples — the purpose of mocks is to replace dependencies on the environment, and tests generally can't (or at least don't want to) rely on the environment. But generally the accepted solution is just to make the dummy/mock provider public; your downstream users probably also want to be able to write tests independent of the environment, so you should give them the tools to do so, if you have them.
That doesn't work well if you're using a mocking system that depends on #[cfg(test)] and disables mocking support outside of tests, but if the interface trait is always available, it's generally a good idea.
Unfortunately you can not do some things there that are possible with doctests. You can not verify compileability of certain [purposefully incorrect] code.
But again: I would expect that even if compileability of some code snippets may be important for you public API it's very hard for me to imagine why it would be very common desire to test internal mocks for compileability.
This may happen, sure, but very common? That's what surprised me.
The way of testing your code's behavior is #[test].
The proximity of a test to the code in question is valuable to me. I see no better way to prove the claims of a function's expected behaviour than right there in the documentation where it makes such claim.