How can a closed-source crate be used by others?

help
1

Today I experiment with building a simple "hello world" binary using two crates.

The main crate has a single source file main.rs, which is listed as follows:

extern crate pr;
 
fn main() {
     pr::pr();
}

The main crate calls the "pr" crate, which is a library crate. The pr crate also has a single source file lib.rs:

pub fn pr() {
    println!("Hello, world!");
}

The two crates live in their respective directories. Now I use cargo to build pr. It is successful.

In cargo.toml of the main crate, there is the following segment:

[dependencies]
pr = "0.1.0"

Therefore I have declared the fact that main crate uses the pr crate both in the source file and in the cargo.toml. But when I use cargo to build the binary, I see the following error message:

Updating registry `https://github.com/rust-lang/crates.io-index`

warning: spurious network error (2 tries remaining): unknown certificate check f
ailure; class=Net (12); code=Certificate (-17)
warning: spurious network error (1 tries remaining): unknown certificate check f
ailure; class=Net (12); code=Certificate (-17)
error: failed to load source for a dependency on pr

Caused by:
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
failed to fetch https://github.com/rust-lang/crates.io-index

Caused by:
unknown certificate check failure; class=Net (12); code=Certificate (-17)

So it seems that cargo only supports using crates published in crates.io?

If I don't want to publish pr in crates.io, then how can we build the binary?

If this can be done directly using rustc but not cargo, then how to write the command line? Furthemore, if pr is a closed-source crate, i.e. I only have pr.rlib, but not the rs file, then how can I use pr?

1 Like
2

Yes. Primary Cargo supports crates that are published in crates.io. You can check this post

3

You can use it with a path dependency.

[dependencies]
pr = { path = "path/to/pr" }

path/to/pr can be an absolute path, or relative to this crate's directory.

There are also git dependencies, for e.g. a private repository:

[dependencies]
foo = { rev = "4fed247a", git = "https://github.com/ExpHP/foo" }
bar = { tag = "v0.1.2", git = "https://github.com/ExpHP/bar" }

(EDIT: originally, I wrote commit = "4fed247a". The correct name for this field is rev.)

You won't be able to publish a crate on crates.io with a closed-source dependency. but you can build it, run it, and do everything else just fine.

5 Likes
4

Thank you! The error message is gone after I do what you suggested. But no exe is generated, I am finding out why.

5

Try cargo run. If you're looking for the binary, it's generated somewhere in target/debug or target/release. (on Unix, at least, but I'm sure the same must be true for Windows).

A binary can also be installed as a CLI command using cargo install.

6

The problem that no exe was generated was because I used link intercepter. Now I have removed the intercepter, and exe is generated.

If I use cargo, the source file of pr crate must be present. I now remain to find out how to build the exe without the pr source file directly using rustc.

7

You can't build anything without source using rustc, because by definition that's a source-to-binary compiler.

However, you can link an already-compiled libraries with Rust programs without needing to give the source of the library. Look for guides on Cargo build scripts and *-sys crates.

You can make a Rust library usable by others by giving it a C API (#[no_mangle] extern "C") and compiling it as cdylib type. Native Rust libraries don't have stable interface (ABI), so without a C-compatible interface (which is stable in Rust) they won't be usable.

So for closed-source Rust libraries the path is: Rust project with closed source → binary C-compatible library → Rust project using it.

You can't publish binary libraries on crates.io directly, but you can publish a minimal open-source Rust wrapper that loads a library from some other source. That's what *-sys crates do.

A completely different option is to give the source, but not publicly. Cargo can use private git repositories.

12 Likes
8

Thank you! I am curious that no book has discussed such issues, but they are very important to earners, I think.

Further questions: Will there be performance penalty (even the slightest) by using a C-wrapper? And is it true that closed-source crate can't be statically linked into the executable? I think it's not true, because on my computer there is no source code of the rust standard library, but I can use them, only correctly. But then where does rustc find the calling information (function name, return type and argument types etc) of the standard library functions, previously provided by header files as in C/C++?

9

It can be statically linked too. There's no extra cost in calling C functions.

For external libraries Rust needs equivalent of C headers. In Rust they're called bindings. See bindgen tool.

1 Like
10

Using a C wrapper means that you can't use #[inline].

It's not true

They're shipped with the compiler, and always have the same calling convention.

The Rust-specific rlib metadata header. On my Windows machine, they're in C:\Program Files\Rust stable MSVC 1.24\lib\rustlib\x86_64-pc-windows-msvc\lib\libstd-8f33fb13bb6c9735.rlib. Notice that they're binaries, that the format isn't stable, and that some of the libraries have accompanying .dll.rlib files allowing them to be dynamically linked.

That random hex number at the end changes every time a new compiler is released. Again, your dynamically-loadable rlibs are tied to your version of rustc.

1 Like
11

By the way, the rlib metadata is are also generated when you do cargo builds for normal libraries. Like .\target\debug\deps\libhtml5ever-980ab38a9dfb57c9.rlib

12

Thank you, notriddle!

I am still curious about one more thing.

In the quote you said function prototypes can be encoded in a binary manner in the rlib, as the standard libraries are treated this way, why is this approach not used in user-written crates?

I still miss the C/C++ model of compilation, where each source file can be independently compiled. Now since the text include files can now be replaced with binary header in the rlib, just simply based on the "extern crate XXX" statement in the rs files, we can extract the prototype information of the called functions from the built libXXX.rlib, and compile our local source files independently.

13

From your wording, and because you seem to be concerned about closed source, I'm not sure if you are talking about legal or technical "can't link"

Notriddle is correct from the technical perspective; rustc can statically link closed-source rustfiles just as good as open-source rust files.

Legally speaking, it depends on the license used by the closed source. Licenses can impose conditions.

For example, the GPL says "anything statically linked to GPL code automatically also becomes GPL licensed".
However, the MPL (favoured in the rust world) says that any form of linking is OK, only derivates of the source file must remain MPL.

You will have to look at your own closed license to see what you can and cannot do.

14

User-written crates aren't distributed as rlibs because the rlib format is not stable. An rlib from rustc 1.0 won't work in rustc 1.24.

Cargo actually does build rlib files when you compile your dependencies. They're just distributed as source code since the source code format is stable.

Stability doesn't matter for the standard library since that thing is already tied to a specific version of the rust compiler.

15

How do other modern programming languages deal with this issue? For example, Swift, Go, Java, Kotlin, C#? They all have got rid of header files. Do they all need source files of dependencies to compile?

16

Java/Kotlin/Scala/etc use the JVM bytecode and C#/F#/etc uses the .NET bytecode. This is the (VM) compiled form of the code (that gets JIT'd/interpreted at runtime). Note though that this does not close off the source in anything but a purely trivial manner, as the bytecode can be decompiled back to the host language pretty trivially. And because of the powerful reflection runtime provided, it needs to be that way.

I'm not sure if Swift handles pre-compiled libraries. The first-party support for using external packages was weak when I tried Swift out and the third-party solution that seemed to be the de-facto standard used source distribution. Go uses GitHub sources for libraries.

17

Generally it's solved by having a file format that has combination of compiled code and equivalent of header file merged together in one file.

That's more or less what Rust has with rlib, too. The difference is that other languages have chosen to call their format stable and try not to break it. Rust's own rlib format is still evolving.

1 Like
18

How about writing an API wrapper in Rust? I.e. like have these modules:

  • crate A: closed source
  • crate B: just an API wrapper for A

So you could compile A and B and ship B's source code plus both binaries. This way you would not need to get unsafe by using a C API.

I have not tested this, so I am not sure it'll work.