Compiling a Rust library as a dynamic library, then using it from Rust projects

help
1

Hi. So this is a problem I've come about: I want to use libservo to create multiple Rust applications. But I don't want to statically link libservo into each binary - the reason is size. Each binary would be 70MB big and the code would be duplicated. I plan to develop a few applications, that will all use this as the layout and rendering backend. So I want to put it in a shared library.

So far I've setup an empty launchpad project (https://launchpad.net/~sharazam/+archive/ubuntu/libservo-embed-experimental), of course I am aware that the API of servo is not very stable, but this is just a proof of concept. Now, Rust does not have a stable ABI yet (https://github.com/rust-lang/rfcs/issues/600). However, I'm wondering how I should go about this without a stable ABI.

The only way would be to use the C ABI. See, I could mark everything as #[repr(C)], but servo is a big project, I can't do this manually. I've looked at cheddar, but that requires you to mark types as #[repr(C)] and limits the use of generics. The second step would be to create some sort of -sys crate (with rust-bindgen) that links back from libservo.so to Rust, so that I can use the functions. But this is just duplicated and unnecessary work, so I'm wondering if there's a better way

In the end, I want people to be able to install libservo as a library. Whether it's an .rlib or a .dylib or a .so, I don't care, as long as my Rust code can talk to the library. I don't want to statically link this, because of the size.

Even if Rust does not have a stable ABI, it should be somewhat compatible between minor releases, shouldn't it? So if I say which compiler the application has to be compiled with, it shouldn't be much of a problem. Could I publish .rlib files? And if yes, how do I make cargo aware of the fact that the library is now managed outside of cargo (i.e. that cargo shouldn't try to get it from crates.io, but should rather look in /usr/lib)? I DON'T WANT C CODE TO CALL MY LIBRARY, JUST RUST CODE. Just saying this because pretty much all tutorials out there are about calling C code from Rust or calling Rust code from C - not calling Rust code from Rust.

3 Likes
2

Yes, this is the only way to be sure.

There is absolutely no guarantee that this is true.

3

Thanks for the answers.

I think it's OK for now if the ABI breaks between releases. Since the builds are reproducible, it only takes 20 minutes to update them for a new compiler / toolchain, so it's not a big deal. I only have to say which compiler this is built for.

However, let's say I've generated the .so file. How would I go and tell cargo what the function definitions are? C libraries usually have a libsomething-sys crate, which provides these bindings. However, since I originally created the library from Rust, there should be some way that Rust knows about these things, so in theory I could say:

extern "Rust" crate servo;

... but I am not sure how the rust compiler would know the function definitions of the .so file, so that the linker can resolve them. Or should I publish .rlib or .dylib files? I could use this:

#[link(kind = "dylib", name = "servo")]
extern {
    fn some_function() -> ();
}

... but then I'd have to re-declare all of servos function to be able to dynamically load them. Is there a way to extract this information automatically?

4

It'd be just extern crate servo.

I believe that if you want only this to be dynamically linked, the way you'd do it is through Page Moved, that is,

cargo:rustc-link-lib=servo
cargo:rustc-link-search=crate=/path/to/where/servo/is

but I haven't tried this myself!

5

Looks like cbindgen is more powerful and better maintained than cheddar. Just FYI

6

@steveklabnik this somehow doesn't work - libservo doesn't link everything statically, even when I tell it to:

error[E0460]: found possibly newer version of crate `env_logger` which `servo` depends on
 --> src/main.rs:1:1
  |
1 | extern crate servo;
  | ^^^^^^^^^^^^^^^^^^^
  |
  = note: perhaps that crate needs to be recompiled?
  = note: crate `env_logger` path #1: /home/felix/.rustup/toolchains/nightly-2017-09-17-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libenv_logger-fe0ada1fd17ca881.rlib
  = note: crate `servo` path #1: /home/felix/Desktop/libservo-embed-experimental-0.0.1/libservo-embed-experimental.so

error: aborting due to previous error

error: Could not compile `linkservodynamicallytest`.

Even when I recompile the library, the error doesn't go away and I don't know what to do. How do I tell it "hey, take env_logger and include it in libservo"? Cargo downloaded and found env_logger, so I'm kind of stumped what this error means.

7

This is BS. I've now spent several hours trying to fix this and I don't get anywhere.

I cloned servo, went into components/servo and changed the crate-type to "dylib". That's all I did in regards to servo. Then I copied the resulting .so file to my desktop, just to see if I could use it as a standalone library. TLDR: I can't.

I created a new project with just an empty library with "extern crate servo" at the start. For reproducabilitys sake, I've put the repo here: https://github.com/fschutt/servodynamictest

First, I nuked the rustup cache, which is the only way that I got rid of the error message I showed earlier - because rust doesn't "get" that I want to use the servo version in the libservo.so file. Here is a "fix", but I can't figure out what this should mean? What is stage-0-tools? Is this specific to the rust compiler?? Apparently, it has to do something with this Fixme, but I am not sure what I should do about this - recompile the compiler? So i remove the rustup cache, because it's the only thing I can do right now:

rm -rf /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/*

That gets rid of the duplicated crate error. But now, rust can't find the standard library:

$ cargo build --release --verbose   
Compiling servodynamictest v0.1.0 (file:///home/felix/Development/servodynamictest)
     Running `/home/felix/Development/servodynamictest/target/release/build/servodynamictest-2444ab718ef9b803/build-script-build`
     Running `rustc --crate-name servodynamictest src/lib.rs --crate-type lib --emit=dep-info,link -C opt-level=3 -C metadata=7a4ef19a9cf4a27a -C extra-filename=-7a4ef19a9cf4a27a --out-dir /home/felix/Development/servodynamictest/target/release/deps -L dependency=/home/felix/Development/servodynamictest/target/release/deps -L crate=/home/felix/Schreibtisch -l dylib=servo`
error[E0463]: can't find crate for `std`

error: aborting due to previous error

error: Could not compile `servodynamictest`.

Caused by:
  process didn't exit successfully: `rustc --crate-name servodynamictest src/lib.rs --crate-type lib --emit=dep-info,link -C opt-level=3 -C metadata=7a4ef19a9cf4a27a -C extra-filename=-7a4ef19a9cf4a27a --out-dir /home/felix/Development/servodynamictest/target/release/deps -L dependency=/home/felix/Development/servodynamictest/target/release/deps -L crate=/home/felix/Schreibtisch -l dylib=servo` (exit code: 101)

I found this (outdated) article: Where Is The Std Crate Located In A Rust Installation | newspaint

Using rustc --print sysroot and then

find /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu -name 'libstd-*'

... I was able to print the paths to where the standard library is installed:

/home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/libstd-b54a6d28690b7929.so

Alright, I put this directory in the RUSTFLAGS before invoking cargo:

$ RUSTFLAGS="-L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib" cargo build --verbose --release

But now rust complains that it "can't find core, which std depends on":

   Compiling servodynamictest v0.1.0 (file:///home/felix/Development/servodynamictest)
     Running `rustc --crate-name build_script_build build.rs --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=2444ab718ef9b803 -C extra-filename=-2444ab718ef9b803 --out-dir /home/felix/Development/servodynamictest/target/release/build/servodynamictest-2444ab718ef9b803 -L dependency=/home/felix/Development/servodynamictest/target/release/deps -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib`
error[E0463]: can't find crate for `core` which `std` depends on

error: aborting due to previous error

error: Could not compile `servodynamictest`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build build.rs --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=2444ab718ef9b803 -C extra-filename=-2444ab718ef9b803 --out-dir /home/felix/Development/servodynamictest/target/release/build/servodynamictest-2444ab718ef9b803 -L dependency=/home/felix/Development/servodynamictest/target/release/deps -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib` (exit code: 101)

Why, just why? All I could find regarding this topic was this (outdated) stackoverflow discussion: rust - Cannot link against core library when cross compiling - Stack Overflow

But the solution proposed there doesn't work:

$ rustup target add x86_64-unknown-linux-gnu
error: component 'rust-std' for target 'x86_64-unknown-linux-gnu' is required for toolchain 'nightly-2017-10-01-x86_64-unknown-linux-gnu' and cannot be re-added

At this point I have no idea what this means. libcore is already installed, ffs!.

So I try locating libcore manually, again, via the find command

find /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu -name 'libcore*'

In the article linked above, the author gets two libraries listed: A .rlib file and the directory with the sources. This was after I executed rustup component add rust-src. But I only get one:

`/home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore`

So what do I do now? I go in this directory and try to build it manually via cargo build --release. Since I didn't switch the toolchain, this should work, right? Wrong! I add the /target/release to the -L flag and get this:

$ RUSTFLAGS="-L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/target/release" cargo build --verbose --release
   Compiling servodynamictest v0.1.0 (file:///home/felix/Development/servodynamictest)
     Running `rustc --crate-name build_script_build build.rs --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=2444ab718ef9b803 -C extra-filename=-2444ab718ef9b803 --out-dir /home/felix/Development/servodynamictest/target/release/build/servodynamictest-2444ab718ef9b803 -L dependency=/home/felix/Development/servodynamictest/target/release/deps -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/target/release`
error[E0460]: found possibly newer version of crate `core` which `std` depends on
  |
  = note: perhaps that crate needs to be recompiled?
  = note: crate `core` path #1: /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/target/release/libcore.rlib
  = note: crate `std` path #1: /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/libstd-b54a6d28690b7929.so

error: aborting due to previous error

error: Could not compile `servodynamictest`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build build.rs --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=2444ab718ef9b803 -C extra-filename=-2444ab718ef9b803 --out-dir /home/felix/Development/servodynamictest/target/release/build/servodynamictest-2444ab718ef9b803 -L dependency=/home/felix/Development/servodynamictest/target/release/deps -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib -L /home/felix/.rustup/toolchains/nightly-2017-10-01-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/target/release` (exit code: 101)

How? I just recompiled it?! Yes, you can build dynamic libraries with Rust, but what sense does this make if you can't even use them! Statically linking everything is not a valid solution. Sorry for my tone.

4 Likes
8

@sharazam, I'm sorry to see you had such problems with this. I got curious about using shared libraries today and started to do a little research. I see it's been a few years, but just wondering if you've tried the libloading crate?