I am a beginner. I would like to ask:
The third-party dependencies of Rust seem to be placed under the project folder created by cargo. Then, when projects A and B both need to use a third-party package, normally it needs to be installed twice under projects A and B. However, for packages like polars, the installation size is very large (about 3G). If it is installed for each project, it will take up too much disk space. In practice, how should this problem be solved? For example, can projects A and B share the same multiple third-party packages? I think Python should adopt this model, but I don't know how Rust users solve this problem.
Cargo does not have a good solution to this.
The closest you can do is to set CARGO_TARGET_DIR env variable to a common directory shared by multiple projects. This will mostly share dependencies, but when projects use slightly different dependency versions or feature flags or Cargo config, it will cause unnecessary rebuilds. Also projects using the same names for binaries will overwrite each other's built files.
You can use a Cargo workspace to build multiple crates together. This will share their common dependencies much better.
You might also reduce disk space usage by hardlinking built results across different target/ folders. I wrote dupe-krill for that.
2 Likes
So is it possible to maintain a tool like anaconda as python in rust?
No, it wouldn't do anything in your case. Sizes of packages are tiny compared to sizes of their compiled artifacts, so alternative distribution of packages would not make a difference.
Size of the polars package and its immediate dependencies is pretty small — under 1MB, and this data is already perfectly shared across all projects built by Cargo.
The massive "installation" size is the size of intermediate artifacts compiled specifically for a project using the dependency, and this is not shared across projects, and these files are not even compatible across computers. These multi-GB files mostly work only for one project on one computer.
Most of the excessive size of these files comes from debug information. You can cut most of it out by adding this to Cargo.toml:
[profile.dev]
debug = 1 # less precise locations
[profile.dev.package."*"]
debug = false # no debug symbols for deps
opt-level = 2 # this makes their code smaller too
3 Likes
I have tried your code, and the size of project become more large 
.
Why the intermediate artifacts are so large? Then if I have multiple projects that need to use polars, wouldn't that take up a lot of hard disk space?
You may need to run cargo clean first, because otherwise you get both old large files and new smaller files together.
1 Like
Buy more storage.
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.