I plan to migrate an existing closed source project to rust. For that I'm looking for a way, to share compiled libraries with other rust users.
As far as I understand, the dependencies in cargo are all from crates.io and are open source.
What is the right way to share my own library with other rust users without crates.io? As well as how do I share them as compiled version to protect my knowledge? (e.g. *.so or *.dll)
There's no support in Cargo for compiled Rust libraries. You will need to make a C-compatible shared library with the C ABI, and then write a sys crate that links it into Rust projects.
There's no support for exposing Rust-native types in shared libraries. There's a crate that helps wrap native-Rust types into a stable format: abi_stable — Rust library // Lib.rs
Ok, I see the slice and it make sense to me.
Thanks for the link to the sys crate, which I quickly checked out. There is a Note:
What about closed-source libraries?
Unfortunately, Cargo crates (and crates.io) are not suited for distribution of binaries. Use another package manager (e.g. apt/RPM, chocolatey) to distribute a pre-compiled shared library (
cdylib) and in the sys crate expect it to be pre-installed.
But it will be still valid to download the binaries in the sys crate build script and than link to the downloaded library. Right?
It's possible, but your users may not appreciate a download that can't be managed (vendored, sent through a proxy, cached and used offline…). It's surprising if nothing else.
As @kornel's article itself says:
Avoid downloading anything. There are packaging and deployment tools that require builds to run off-line in isolated containers. If you want to build the library from source, bundle the source in the Cargo crate.
However, if you aren't specifically looking to publish to
crates.io, then nothing is stopping you from including a compiled library in a Git repo that is used as a dependency.
The primary issue is that Rust has no defined stable Rust-native ABI. There is some work on stable Rust ABI, but it still requires a lot of work. Thus far only the variations of C ABI are stable, so that's what you have to use if you want to distribute precompiled artifacts. That is unfortunate, of course, since you lose quite a lot of ergonomics and safety on the FFI boundary.
Arguably one does lose any safety in calling a Rust library through the C FFI boundary. After all the library is closed source so one cannot know what dangers hide in there. The whole thing is necessarily unsafe.
This approach looks a bit better. But the ugly thing is, that it requires to include binaries in a git repository which is as well not that common and what git wasn't made for.
But what about an additional key for cargo like
https where an URL to a binary file (zip archive) could be provided?
This is a clear declaration of the external dependency. Is there any way to declare such request for cargo?
Unfortunately not. Cargo will not download anything other than the usual package tarballs, which are meant for source code.
You'd have to add an HTTPS library as a build-time dependency, but that's relatively heavy and burdensome (e.g. may need openssl headers that are difficult to install in some scenarios).
Is sharing the source code with your users (but no one else) acceptable? If so, you could set up your own private registry, which requires authentication, and have your users download your code as a dependency from there.
Closed source often doesn't strictly mean "the source must never be visible to anyone" but rather "the source is ours and we choose whom to share it with"
Thanks for the explanation. That's one of the reason why I think a https source in the Cargo dependency list could handle it the right way. Cargo has already support for git which also supports the https interface. So Cargo already depends and use all the ssl stuff.
I'm not sure where is the right place to discuss such idea?
IIRC the git support just uses the system git CLI? (Or am I thinking of npm?)
Cargo can use either built-in git (
gix) or CLI.
The internals forum: https://internals.rust-lang.org/
However, I suspect that "can you just download a blob over https?" may not be good enough for Cargo. The design would probably need to ensure the blobs have some integrity check, and when such crate is used on crates.io, that it doesn't depend on any external URL that could become unavailable or track users.
When binaries are used from a https source in other tools, a hash is often used to validate the file. Such hash could also be part of the dependency declaration in cargo and should be checked after downloading the file.
Additionally the possibility of an unavailable source or the tracking of users is already true for the git source. I'm not sure how crates.io handle this, but it seams to be possible.
As a matter of policy, crates-io does not allow any crates that use git dependencies.
That seems very limiting. I can't find it stated anywhere on
crates.io. Do you know where it is documented, or is it just something you find out when you try to publish?
I am glad such limitations are in place. When it comes to downloading and running code on my machine I would like at least some accountability/traceability even if it is a little inconvenient.
I think the most straightforward “documentation” of this is that a normal
cargo publish invocation won’t let you publish a crate with git dependencies in the first place. (I haven’t tested the behavior. I believe, git dependencies are stripped, and if there isn’t also a version number for the crates.io variant of the same dependency, your crate might fail to compile and thus not publish, or there might be a custom error message. Edit: Looks like custom error message it is.)
There's a note about it here (but should probably be more notes elsewhere).