What is the smallest unit for static linking?

1

In C/C++, the smallest unit for static linking is obj. If your program invokes a library written in C, your executable normally does not contain all the code of the library, but the objs directly or indirectly used by your program. An obj is either entirely linked or not a single byte linked into the executable, i.e. obj is the smallest linking unit. Am I right?

But what is the case for rust? It seems to me roughly c/c++ library is equivalent to rust crate, and c/c++ obj is equivalent to rust module. But I already see a big difference, c/c++ emits independent obj files, but rust does not emit independent module files. And, is it true that module is the smallest unit for static linking? Furthermore, in rust a module may have submodules while c/c++ does not have sub-objs, which further complicates the issue for rust. If module is indeed the smallest unit, then there are still several possibilities:

  1. The highest level module (direct child of crate) is the smallest unit.
  2. The lowest level module is the smallest unit.
  3. The part of a module except all its submodules is the smallest unit.

Which of the three possibilities is true? or I am totally wrong?

1 Like
2

Typically Rust's base compilation unit is the crate, not the module, this is very different to C/C++ where every *.c file is its own compilation unit. I believe this was done because it removes the need for forward declarations, allows you to have cyclic dependencies between modules, and also lends itself to better optimisations because LLVM has more context to work with.

For performance reasons (incremental compilation, etc) the Rust compiler may break this up into multiple codegen units which LLVM can then compile in parallel, but that's purely an implementation detail.

5 Likes
3

Then this feature of rust is what I don't like. The executable then tends to be bloated because of this. Will this get a chance to be changed in the future?

4

A Rust .rlib crate is more like a C .a archive. The linker is responsible for discarding unused sections. Could you be more specific about which specific symbols you're seeing in a compiled executable that shouldn't be there?

1 Like
5

You seem to mean that the section is the smallest unit for linking. That is a very good news for me, as the section is even smaller than a module.

6

I really what the rust core team to address this issue. Unnecessarily huge executable size is unacceptable to many programming purists, and purists are the most likely users of rust. This is a big shame! See also the following page:

https://lifthrasiir.github.io/rustlog/why-is-a-rust-executable-large.html

7

If I were to statically link a similar C++ program I'd actually end up with an executable with a similar size, so I'm not exactly sure what your point is...

A section isn't really a unit for linking, trimming unused sections is the machine code equivalent of the dead code elimination that normally happens.

I've never found executable size to be much of an issue. At work we distribute my Rust code as a DLL alongside our main product and despite the project being fairly decently sized (we've written tens of thousands of lines, pulling in hundreds of thousands via crates.io) the DLL is still only 1-2MB when compiled in release mode.

3 Likes
8

The size of executable is less wasteful if you make fuller use of a crate. So this is really not a big issue for most commercial products. But for classroom exercises such as the hello world program this really is an issue, and I think purists (including me) will be unhappy with it.

9

Most of the overhead from hello world is because you pull in all the standard library stuff for string formatting and working with io. That's not really avoidable without jumping through a lot of hoops.

If you really care about making a small executable you can always tell rust to link dynamically where possible. The downside to this is you need to have the Rust standard library on your system in order to run the program. C and C++ can do this just fine because libc is installed everywhere, however Rust doesn't have this luxury and if we didn't statically link (causing larger binaries) people would complain that it generates "broken" executables... You just can't win :disappointed:

If you were to ask rustc to prefer dynamic linking then the executable size is comparable to the equivalent C program.

$ cat main.rs
fn main() {
    println!("Hello World!");
}
$ cat main.c 
#include <stdio.h>

int main(int argc, char *argv[])
{
    printf("Hello World!");
    return 0;
}
$ rustc main.rs -O -o main_release                  
$ rustc main.rs -O -o main_dynamic -C prefer-dynamic
$ gcc main.c -o main_c
$ du -h main_*
12K	main_c
16K	main_dynamic
5.0M	main_release
7 Likes
10

Sadly, Rust is optimised for real world use-cases. We support trimming down your executable to your hearts intent, but the default is reasonable for general programs.

Also, the number of purists is in general quite low, it would be harmful to tune the default settings towards them.

For example, Rust - by default - uses an allocator that is purist-unfriendly. jemalloc.

We are happy to support these usecases, though, Rust is even used in the demo scene.

I really what the rust core team to address this issue. Unnecessarily huge executable size is unacceptable to many programming purists, and purists are the most likely users of rust.

I'm sorry, but we hate gut feeling, so we measure. Your statement doesn't match numbers at all. State of Rust Survey 2016 | Rust Blog

Scroll down to "what programming language are you most comfortable with". If you do a tally, the number of C and C++ programmers isn't even half of the community and purists are a subgroup of them.

12 Likes
11

Dear Skade, thank you for your information! Could you please tell me why the crate must be the smallest linking unit? There are three possibilities:

  1. Using C-style obj as smallest linking unit in rust is in principle not in contradiction with other performance considerations, but because the rust team is too busy doing other more important things. This issue will be addressed if those more important things have been fixed.

  2. Using C-style obj as smallest linking unit in rust is in principle not in contradiction with other performance considerations, but it is very difficult to find a perfect solution so that both purists and practitioners can be happy, and the rust team will therefore not seek that kind of a solution in the foreseeable future.

  3. It is theoretically impossible to have C-style obj as smallest linking unit, while not hurting performance in other aspects.

Which of the above three is correct? Thank you in advance!

12

I think your dot points are probably too absolute and the real answer will probably be somewhere in the middle, or maybe something completely different. Instead, like all engineering tasks it's all about trade offs and finding something which suits your needs.

Using the crate as the base compilation unit probably provides a happy medium between not having enough context (C-style one object per source file, requires headers and other ugliness) and lumping a crate with all its dependencies into one massive object file (bloated binaries, massive compilation times etc).

Incremental compilation also changes everything because the the compiler can dynamically choose to break a crate up into an an arbitrary number of smaller object files to make builds more parallel. With incremental compilation the idea is if only one module (or function, or whatever) changes we only need to recompile that bit of code. I would definitely recommend you read the RFC for more!

Given that, stating that you must either take a one-object-per-source-file or a one-object-per-crate approach is probably too narrow minded.

So I guess you could say:

  • There's nothing stopping Rust from using lots of small object files
  • The rust developers are actively working on this (incremental compilation, etc)
  • Solutions exist

I should also mention that in real life you never actually deal with individual object files anyway. rustc and cargo exist to automate the build process and figure out the best way to do things.

6 Likes
13

Have you tried using LTO? This will mitigate the fact that the compilation unit is an entire crate because it allows the linker to strip out unused stuff at link time.

If you use Xargo you can even perform LTO on std because Xargo uses the source for std when it performs a build of your project!

5 Likes
14

Thank you Michael! Your answer is really one I have been looking for.

Rust is the final stop of the evolution of computer languages. It is still young, still on its way toward perfection, and we need patience.

15

Thank you John! I am now looking into Xargo. LTO may be a promising solution.

16

You can also specifiy opt-level = 'z' when using a nightly compiler to instruct rustc to optimize for size instead of performance if binary size is of more concern.

Finally, if you are compiling on Linux or Mac, you may want to remove jemalloc (~200-300K).

1 Like
17

Isn't that the missing bullet holes problem? The low number of "purist" programmers responding to the survey may as well be exactly the symptom of turning them off from Rust.

18

The continued references to "purity" in this thread are disconcerting. It is completely undefined, and liable to be redefined during any given conversation to serve the user's biases.

I would encourage people to use more precise language when talking about what they're looking for. "purity" only seems to serve the purpose of making one group of people feel superior to others.

3 Likes
19

I agree. In addition to being generally vague/problematic, "pure" is a word which I most frequently hear from functional programmers with a totally different meaning, so the way it is used here is especially confusing.

1 Like
20

No, sorry. The missing bullet hole problem was a nice metaphor applied to a very specific problem in that blog post, but you can't just throw it at any statistic.

First, as others say, "engine" is a very defined term, "embedded" isn't quite, but has enough boundaries. As other mentioned here, "purist" is a fuzzy term that matches many descriptions. Also, the point of me invoking this statistic is also that Rust has a lot of draw outside of its classic field. Second, "embedded" is a much more important field than "purism" is. Third, Rust supports some notion "purism" (through a couple of flags), while the blog post was about Rust having problems in the embedded space that make it not usable to some users.

Yes, I liked appreciated the blog post, I'd hate if I see it used to undermine any statistic.

1 Like