Dynamic linking of SharedLibs with Cargo

emoon · 2016-02-25T05:21:38.479Z

Hi,

One thing I do with one of my projects is to have pretty much most code in shared libraries (as plugins) this makes the architecture cleaner and code be reloaded while not closing down the application. One issue right now is that a shared lib (Mac OS) tested is 1.5 MB with this code

#[no_mangle]
pub fn shared_fun() -> i32 {
    42
}

And if you have 10 of them it’s 15 MB which is pretty much 15 copies of std lib in this case. If I compile it manually with rustc -C prefer-dynamic it ends up being 8.4k which is much nicer in size.

Now I wonder if there is a way to tell Cargo to prefer dynamic linking over static? Also when I try to load the shared lib with that has been compiled prefer dynamic my program exists with signal 11. I would assume it means that it can’t find the dynamic version of std lib or do I need to manually load it myself?

Cheers!

oli_obk · 2016-02-25T08:34:10.513Z

You can at least decide to make your crate a dynamic linkable crate. I’m not sure what effects that has though.

emoon · 2016-02-25T08:52:07.430Z

oli_obk:

You can at least decide to make your crate a dynamic linkable crate. I’m not sure what effects that has though.

I already do that and that works fine. The issue is that std lib gets linked in for each dynamic libray and if you have a bunch of them this size adds up quite fast. So size might not be a super big issue today but it still seems a bit unnecessary to duplicate the stdlib code so many times when it’s not needed.

Tabor · 2016-02-26T02:40:47.992Z

The fact that this isn’t supported by cargo yet really baffles me. I guess if we want this feature we might have to implement it ourselves and hope that they take our changes. It hasn’t blocked me yet so I have been working on other things.

emoon · 2016-02-26T06:58:46.360Z

I could likely do a work-around by using a script which does rustc -C prefer-dynamic and the have Cargo call that instead of regular rustc but it would be nicer to have this supported directly in Cargo imo.

sjmackenzie · 2016-02-27T07:52:00.097Z

Yes indeed, we’ve also chosen to go the path of shared lib as plugin approach. Size does matter, having cargo be aware of this helps greatly.

Side note, do any of your plugins pull in OS level deps like openssl? Do you run out of initial-exec TLS variables? If so, how do you deal with it? (please answer specifically on this topic Initial-exec TLS variables )

emoon · 2016-02-27T08:54:28.625Z

Hi,

No, I don’t have any TLS that I currently use. That being said I do run into this issue https://github.com/rust-lang/rust/issues/28794 on Mac. Also I get signal 11 when trying to load a shared library when using std library as shared. Have you run into the same issue? I assume that something can’t be loaded correctly or such.

stevedonovan · 2017-05-31T11:30:51.280Z

Setting RUSTFLAGS='-C prefer-dynamic' works nicely.

emoon · 2017-05-31T13:17:13.757Z

This has now already been implemented with https://doc.rust-lang.org/reference/linkage.html and cdylib that can be set in Cargo.toml

stevedonovan · 2017-05-31T13:35:05.274Z

So the difference is that cdylib is a plain shared library, without the Rust metadata of dylib? For plugins, dylib is desirable. And I can’t yet find a Cargo incantation other than RUSTFLAGS to force these dylibs to be linked dynamically with the stdlib.

emoon · 2017-05-31T14:22:43.867Z

no, cdylib will not statically link in the standard library while the dylib will. So it depends on your use case. If you want a fully standalone dylib you want to link with dylib but if you are fine with including the standard lib files you can use cdylib

In my case I will have several plugins which in the case of dylib would include standard lib in each lib making them quite big so I rather have cdylib and just include the standard lib dylibs.