We've been working in a workspace for several years on prerelease versions that used path dependencies. Now that we're releasing 1.0.0 for most crates, I'm working on changing those into version dependencies. This works for the most part - and we tried this before as a test, but our repo dependencies got more complicated since I now realize - but I run into problems where:
A --(path)--> B (B unpublished)
B --(dev-dependency version) --> A
And now any command that previously worked with -p A fails with "which version?"
What we'd like is for anything with a dependency on A to be a versioned dependency such that they can ship independently, but keep A ready for the next release. Anything that requires changes in A we haven't shipped yet can't ship then, so it keeps the dependency tree clean since crates.io won't let us publish a crate with an unresolved dependency.
I've tried a number of things including strict dependencies vs. dev-dependencies adherence on who can use a version (via workspace) vs. path, but that hasn't worked. In fact, it's made the multi-version problem worse because the more version dependencies there are, the more A@1.0.0 and A@1.1.0 there are (and for any crate).
Is there any good way split dependencies in such a way that [dependencies] use version while [dev-dependencies] use path? When I tried that, cargo errs that a dependency must have a single source regardless of target.
Yes, it is a known limitation that you can't version+path (regular sources) and path-only (cycle breaking sources) in a single workspace dependency. This is one of the reasons I've felt workspace dependencies aren't mature enough for cargo to push users to use them.
Thanks for the confirmation. Moving tests is one of our least-worst ideas. For cross-crate samples I can make "look-alikes" in a module within examples/ to sort of "hide" the issue and break the cross-crate dependency.
Another not-terrible idea (compared to some other ideas we've come up with) I'm toying with is either actually publishing our test crates with a README that basically says, "not supported" (hopefully the eternal prerelease version would help) but a dependency override - to make it look like they're on crates.io - might actually work. I'll try that out tomorrow, too.
Another possibility is to leave the real crate name for samples and tests but add a versioned dependency e.g. A_next to other crates and just make them deal with the different name. Samples are what people/LLMs will look at so I think it's better to make sure those mimic real-world examples.
All in all, we've been pretty happy with the workspaces. The main reason I wanted to use them was for centralized dependency management - not just of 3P crates but our own as well. In general, we want all service crates to take a version dependency on what is released and only if some service crate needs a change to a dependency does it take a path dependency. If we ship an update to a dependency, we want to make sure all the other crates ship their next version with that as the min. We prototyped that early, but I think that all worked before we introduced cross-target circular dependencies.
Moving tests to a separate crate, if you can do it without complicating things further, also reduces the total amount of code that needs to be built. This is because when you have tests inside a library crate, the entire library crate is built as a test binary (rustc --test), including all the code that isn't the test, but when you have a separate test crate (whether it’s in the same package or not), and have the library marked lib.test = false, the library sources don’t get be built once normally and once as a test binary; instead, the separate test binary crate depends on the normal library crate.
Could you clarify? Maybe I'm reading too much into crate vs. package, or maybe this is a concept I've not come across before that makes this a slam dunk. The only reason I'm reluctant to tests to a separate crate is the ramification to our DX and build pipelines where, logically, if you want to test changes in B you just run cargo test -p B. If I move tests to B_tests then you have to remember tests are in a separate crate. If there's some "magic" way to have the former run tests in the latter, that sounds like a win.
Yes, you have identified one of the ways in which this complicates things further: that there is no automatic way to say “run the tests that are for this package” if some of the tests are in another package. This isn’t a problem if the tests are in a different crate but the same package, i.e. in a test target[1], but of course that doesn’t help your dependencies situation. I just thought I’d mention an additional benefit that you get along with the complication, if your tests are currently inside the library crate source.
Ah, I think I get what you mean. I know tests (and examples) are compiled into separate crates, but I was erroneously considering everything the "crate". You're calling all crates - lib, tests, examples, etc. - the "package". Gotcha (or correct me if I misinterpretted your meaning).
It's actually only the integration tests that are the problem anyway. All lib tests are just unit tests and never use other dependent crates - only dependencies as needed. Those build fine. It's only integration tests i.e. crates where this problem comes up. Still, if I move the tests and examples to another crate - package - that would allow me to effectively have different sources for the same dependency. I'll give that a shot as well.
Thinking about this more last night, I realized something along these lines should fix the build issue with integration tests and examples, but won't prevent partners from having to pass a version in the package reference for any crate in the workspace known by different versions e.g., cargo test -p B@1.1.0.
That alone isn't terrible, I think, but it does make the DX a little worse but the pipeline task/tooling harder because we have to harvest the latest version.
Some package managers like npm or homebrew let you use monikers like latest to refer to the latest version. cargo already knows this (at some point) because it writes,
error: specificationm `azure_core` is ambiguous
help: re-run this command with one of the following specifications
A@1.0.0
A@1.1.0
(I see the typo was recently fixed - was just about to file an issue and submit a PR)
Would you be open to supporting a latest moniker? It seems some semver compat is already there e.g., I can run cargo test -p A@1 or even cargo test -p A@1.1 to select the right version.
In addition to supporting a latest moniker, would you be open to exploring (I can write up an RFC or whatever if so) allowing separate sources for [dependencies] vs [dev-dependencies]? I assume [build-dependencies] would otherwise be treated as [dependencies] in this regard. Default behavior wouldn't change, but since bin/lib targets compile to separate crates than tests/examples/benches anyway, it seems it would be technically possible to allow for separate dependency sources instead of inferring the current crate version as the source of truth.
In one case where we wanted to enable - not simply require - a feature for tests in a crate, we found that this worked already:
[package]
name = "foo"
version = "0.2.0"
[features]
default = []
test = []
[dev-dependencies]
foo = { path = ".", features = ["test"] }
While my idea probably couldn't work for a self-reference, referencing a separate crate by path - when a path exists - in [dev-dependencies] in lieu of whatever [dependencies] has if different could work because we're building a different crate for tests/examples/benches anyway.
Our main way to improve that workflow, as you already noted, is that we allow partial versions in package specification IDs.
So if I'm understanding this correctly, if you have a dependencies and a dev-dependencies on the same dependency name (I'm assuming it wouldn't be package name or some other characteristic) and you select a dev-target, override the dependencies entry for normal targets (lib, bins). The current behavior is a "different source paths" error (even with build-dependencies).
and it acts as if you are patching clap when building a dev-dependency.
Personally, I would be hesitant about such an idea. I think it could be a source of user confusion. Initially, one thought was a concern over dev-dependencies affecting non-dev targets but we do have that today with features. Maybe that isn't a great showcase though as it causes a lot of user confusion. features ends up I think affecting most examples, both as a "yes, we already do something like this" along with a "but it annoys people". For example, there are build time aspects that shared code has to be rebuilt. For code under test, you are further diverging from testing what will be run in production.
There are also [patch]. If you go forward in exploring this idea, you would need to make the pitch for [patch] wouldn't be sufficient (along with why this workflow is important and why it needs to be solved in this way).
My simple example I've been using is over-simplifying, but it still comes down to the same problem: we have/want version dependencies for bin/lib targets but want path dependencies for test/example/bench targets (in some cases). There are actually 3 to 4 crates in this cycle that end up causing a version mismatch for e.g., trait implementation. Same trait, just different versions because the examples infer the current crate version while that references a crate that ends up having a version dependency on the same crate.
My thinking was that consumers seem primarily concerned with [build-dependencies] and [dependencies]. I know there's been discussion about running tests for dependencies but ours already wouldn't work because we have test/example dependencies on paths to crates in our repo we don't intend to publish e.g., an async test runner that routes traffic through a proxy so we can record and play back tests - something all our language SDKs share. Very bespoke and not something we want to support outside our repo.
So if [dev-dependencies] are more for the crate developers themselves and compile to separate crates, allowing for a different source - a path, for example - would be possible. I haven't delved into the code yet for this - and certainly you'd know it best - but I wanted to gauge if there was at least enough interest into it. My hope was that if a different sourced dependency was found in [dev-dependencies] that could take precedent rather than erring. I imagine it's not quite so simple (it rarely is), but hoping that might be a start.
I'm still trying a few options based on our discussions. If I can break the cycle there's less reason we might need differently-sourced dependencies, but the multi-version package refs are still going to be a problem. Most devs will probably just run cargo build or cargo test while inside crates' directories so that shouldn't be an issue, but all our pipelines do refer to packages by name often and will need a version - almost always the latest, hence the desire to be able to use a moniker like latest and not have to harvest the latest version, which gets especially hard when we need to refer to multiple packages at once.
Alternatively, what about allowing --manifest-path to be specified more than once? cargo already knows when that manifest is part of a workspace, so is there any reason why it's fundamentally different than multiple --package parameters?
Here's a concrete example after I redeclared or redefined a few types to avoid a cross-target cross-version cyclic dependency:
Remove path from shipping dependencies. This provides central management of both 1P and 3P dependencies such that dependent crates can ship new versions independently and, when they do, they get the latest core crates from under sdk/core. We do something similar with other languages in whatever way is idiomatic to the language build toolchain.
So that's where azure_core@1.0.0 and azure_core@1.1.0-beta.1 come from. Minimally redeclaring or redefining types we used in examples, tests, and doc tests in azure_core at least broke the pseudo-cyclic dependency that created ambiguity at least, and I'm seemingly able to limit the impact of distinct versions to the core crates so downstream service crates owned by partners shouldn't be impacted making their DX better.
Already I'm seeing a few failures in our pipeline that always build or test azure_core and its dependencies to make sure the core stack is always working. I'll resolve those as appropriate, but this is where having a version moniker like latest scoped to the latest semver cargo knows about would help. I should be able to switch to --manifest-path in most cases, but I don't believe all where we need to build multiple packages or specifically need to test azure_core when it's updated. If we could pass <crate>@latest in any case - which would always refer to the vNext version in source - it would certainly simplify things.
At least in this use case, I'd expect latest to just refer to the latest semver that cargo already knows about after resolving all crates within the workspace.
I also want to clarify that I appreciate that even if any proposal is entertained, this wouldn’t happen overnight. I’ll have to resolve these issues somehow regardless and possibly with some hacky workarounds like harvesting the latest version in a wrapper script. I’ll try to avoid that when I can, but it’s one possibility we considered. I wanted to explore ways I could maybe make this better/easier in the future for anyone.