In light of the arrayref incident, it's probably a good idea to try to think about dependency security differently.
It's often complained that this is due to missing feature in cargo, for example, crate namespaces. But we actually kind of already have everything we need: custom registry. And a design based on that already works mostly hassle free.
The core idea is: use custom registries to represent trust boundaries.
The default crates-io registry is disabled. And everything works on custom registries.
One curated stdlib custom registry: commonly used stuff like serde, tokio, etc.
Special purpose custom registries, or per-team registries.
This design already works and do not have ergonomics issue:
# <project>/.cargo/config.toml
# This gets loaded automatically by cargo.
# Disable crates-io
[source.crates-io]
replace-with = "crates-io-disabled"
# vendor/crates-io-disabled is an empty folder
[source.crates-io-disabled]
directory = "vendor/crates-io-disabled"
# Define all registries. This is the trust boundary.
[registries.stdlib]
index = "sparse+https://somewhere-stdlib.org/cargo/index/"
[registries.a-large-web-framework]
index = "sparse+https://somewhere-web-framework.org/cargo/index/"
[registries.other-downstream]
index = "..."
[registries.another-dev-user-trust]
index = "..."
In Cargo.toml dependency, it becomes:
# Cargo.toml file dependencies section.
tokio = { version = "1.0", registry = "stdlib" }
something-else = { version = "x.x", registry = "other-downstream" }
Cargo.lock file already properly resolves to the full registry url. So this actually should already work (at most small custom registry implementation changes that do not require RFCs).
The main advantages of this are:
Like go.mod, it forces users to consider the entire transitive trust graph -- all registries (as trust boundaries) must be defined. No surprise new trust boundary.
We get Rust's own stdlib -- a vetted, commonly used package set as a custom registry. Maintainable, as this can be kept intentionally small and well audited.
It's the namespace system many people wanted -- each team is its own custom registry, decentralized. No need for any unnecessary coordination.
The transition to custom registry is also doable:
A curated stdlib. And that only needs figuring out the core package set + republishing to custom registry.
The system then becomes immediately usable for those who want to practice minimal dependency philosophy.
Other teams then join by hosting their own custom registries.
I don't see how a custom registry per se prevents compromised 3rd party packages. Does it make a difference from where I pull the compromised library? If my team needs arrayref and puts it in the custom registry, Cargo will pull it, extract it and run the malicious build script on my computer, same as if I had pulled it from crates.io directly. I'm no expert, but to me trust boundaries and manual curation don't sound like very effective[1] measurements against the growing issue of software supply chain attacks.
Because it would not have been possible to put in the malicious arrayref in the first place. Think about publishing to stdlib as submitting a new PR, etc.
Multiple custom registries are what allow each registry to be curated. And then you can define your own trust boundaries on top.
Of course, the stdlib's team credential can also be stolen, but both npm and Rust showed that this is not what commonly happens. What commonly happens in supply chain attack is an individual who is usually independent/outside but maintains an important yet small crate, and that gets compromised.
And that's why security boundary is important and how the stdlib design reduces the number of security boudnaries needed -- arrayref update would have been a normal PR before its publishing.
Previously there's another security boundary just with the independent developer. Now that is removed. One less.
Common Rust projects have 300+ 400+ deps and a large number of them are cases like this. And a curated core set reduces the trust boundary by a lot.
I have idea to put fast security AI in between the crate upload, like Google's Magika that protect Gmail and Drive from mallicious file, but in this case is code instead of file
So the idea is
First, extract code different of the old and new version, kind like how Github can differ new and old code, but only take the new code
Send it to the AI to be analyzed
If the analysis result is it is mallicious code, reject the upload
If there is safe code that is rejected (false positive), send review request, show the list of review request in crate.io website in different page url, the whole community can contribute to the review request, to be approved requiring at least 5 acceptance of review + 1 final acceptance from the security team (so the team will look up fewer lists, because it is filtered out by the community first). If it is accepted by 1 security team it doesn't need the 5 community acceptance
I'm not convinced. Are you saying you want to personally vet every single version of all your 300+ 400+ (transitive) dependencies, to put them in your "trusted" custom registry? I can't see how this would prevent vulnerabilities in a meaningful way (I personally would probably miss 100% of any potential vulnerabilities). The merits and shortcomings of a "web-of-trust" approach to ecosystem curation are also discussed in the topic I linked above (in the footnote; posting it again for better visibility):
Are you using "trust boundaries" and "security boundaries" interchangeably or are these two different concepts?
The elegance of the custom registry design is that you do not need to wait for acceptance of your proposal. We can have multiple stdlib registries, "competing":
One is nixpkgs style manual PR and merge.
Another auto AI review like you envisioned.
Some other types with more rigid, smaller core.
Then people get to choose their trust boundary. At least in my mental model, the security risks would be much more understandable -- instead of tap into a global crates-io where there may or may not be malicious packages, I choose a registry that I can place higher trusts.
Yes I'm using those two concepts interchangably. Non-native speaker, but to put it concretely:
You check one registry, give it a number -- risk/percentage this registry may get hacked.
Each registry has one number.
The reason this is a lot more manageable is because there are simply a lot less numbers. Think single digit. Right now, an ordinary Rust project has 300+ 400+ dependencies all tangled together. So each time when you add a dep, you must analyze it separately and completely, 300+ 400+ times, every single user. Or we just blindly trust all crates on crates.io and do nothing, which is what majority of users do.
Yes. This is what right now everyone has to do anyway if they ever want to gain any notion of security. Or we wait a negative from security researchers, but that's fragile.
This is entirely doable -- we're not checking whether the code is absolutely secure. We're only checking whether the code is malicious. Nixpkgs manage 100k+ packages this way. Julia package manager is actually also designed this way (PR based, but last time I used it was 5+ years ago, so take this with a grain of salt). And they manage 10k+ packages.
You can think of it as a more organized cargo vet, with proper granularity -- place the trust boundary on a curation / team, instead of individual crates. Instead of 300+ 400+, each user then only needs to check 5-10. The magnitude reduced number is why this works.
At first blush, this sounds harmful because custom registries would be employed only by more serious organisations, which yes makes them secure. Yet, those major organisations would be the ones who do scanning too, no?
If those organisations scan slower because they all adopt custom registries, then they increase how many random developers get compromised, and this would make credential stealers overall more effective. It's possible organisations who scan would not scan slower because of custom registries, but.. enshitification usually wins.
We need the organisations who scan crates to establish a default non-zero value for cargo's min-push-age. If they say 1 day then great, make that the default. If they say 1 week, not so great but still make that the default. Anyways min-push-age would provide a similar defence for everyone, not just the major organisations. We just need the scanning organisations to control the default value, since they de facto control the correct value.
You could override min-push-age of course. In fact, you'd typically do so for your own dependencies, so that's really what would make namespaces relevant here: min-push-age = 0 in the same namespace or repo. min-push-age has some non-zero default otherwise.
Everyone cares about themselves before they can accommodate security of others. As I searched Github, Azure already has something like "approved" crate registry. So this is probably already happening.
Thing is, if custom registries are more secure, people will use it. They won't first care about the effects on others. Therefore the correct move is to migrate everyone to more secure construct. You can't force people to stay insecure "for the benefits of the community".
They organize themselves. Someone, or some teams, start a new stdlib registry. They then curate it. Review PRs, etc. When there are multiple "competing" curations, people will be able to choose who they trust.
I'm honestly thinking about starting a pilot project and see how it goes. The current crates-io-importing-300-plus-packages-for-a-small-scripting-program situation is clearly nonsustainable, and I believe practicing a minimal dependency philosophy may be the only way to keep oneself safe in the Rust ecosystem.
Nixpkgs, Julia, and I recently learned Nim all do curation and they are fine. Nixpkgs is probably the prime example, and considering the fact that they have the largest package count among Linux distributions, it's a great result.
This seems like an ecosystem discussion by virtue of being on this forum.
cargo's min-push-age is a better similar solution for the ecosystem. It defaults to zero now, which seems like a problem. I'm saying someone must establish some default non-zero value.
If min-push-age is set correctly, then custom registries provide essentially nothing of value, because realistically registries would not do additional testing beyond the scanners.
If you've some mid sized organization, then you're probably not going to run your own scanners, so custom regestries wind up mostly being just a sloppier min-push-age.
This has nothing to do with the compiler team if you read the full discussions.
I think you've established exactly why custom registries work. You can then do additional testing if you want. That's what curation means.
min-push-age is definitely good to employ when it becomes possible (ironically, at this very moment, custom registries are stable while min-push-age is not). But it also has the same "good-for-me-is-bad-for-the-ecosystem" problem. If the default is 24 hours, security sensitive people like us will set it to 7 days. If default becomes 7 days, people will set it to 30 days.
There's no reason why you "probably not going to run your own scanners". If you're a small/mid sized organization, you can even still do manual human review if you practice minimal dependency philosophy. That philosophy, but not any tools, is the vital reason this strategy works. Custom registries are just still the best way to have full control over your dependency count -- a pull model, rather than auto-push-after-age.
The results are quite encouraging. The current registry has ~750 crates. We did some further analysis:
Adding things that I commonly develop (async, backend, database, crypto), the crate count plateau at just ~1000 crates.
Adding "crates of significance", that is, commonly used crates in development categories I can think of (testing/benching, TUI, GUI, wasm, etc), the crate count plateau at around 1500-2000 crates.
Each genuine new category adds new crate counts, but otherwise, for a specific category, the crate count always plateau.
This approach of a curated stdlib custom registry is totally doable.