Linker args no longer being forwarded? (Cortex-M)

Did something recently change in the forwarding of linker command line flags to a custom linker? My embedded projects are no longer building with the latest nightly.

(CC: @japaric)

For example, my minimal embedded Rust example now fails with a link error (just rustup update and follow the instructions in the readme to repro):

error: linking with `arm-none-eabi-gcc` failed: exit code: 1
  |
  = note: "arm-none-eabi-gcc" "-L" "/home/cbiffle/.multirust/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7em-none-eabi/lib" "/home/cbiffle/proj/rs/minimal-embedded-rust/target/thumbv7em-none-eabi/release/emb1.0.o" "-o" "/home/cbiffle/proj/rs/minimal-embedded-rust/target/thumbv7em-none-eabi/release/emb1" "-Wl,--gc-sections" "-nodefaultlibs" "-L" "/home/cbiffle/proj/rs/minimal-embedded-rust/target/thumbv7em-none-eabi/release/deps" "-L" "/home/cbiffle/.multirust/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7em-none-eabi/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/cbiffle/proj/rs/minimal-embedded-rust/target/thumbv7em-none-eabi/release/deps/libcore-882297ed0c39d543.rlib"
  = note: /opt/arm-toolchain-4.8/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/lib/crt0.o: In function `_start':
/tmp/gae/gcc-arm-embedded/src/newlib-nano-2.1/libgloss/arm/crt0.S:264: undefined reference to `memset'
/tmp/gae/gcc-arm-embedded/src/newlib-nano-2.1/libgloss/arm/crt0.S:416: undefined reference to `__libc_init_array'
/tmp/gae/gcc-arm-embedded/src/newlib-nano-2.1/libgloss/arm/crt0.S:420: undefined reference to `main'
/tmp/gae/gcc-arm-embedded/src/newlib-nano-2.1/libgloss/arm/crt0.S:422: undefined reference to `exit'
collect2: error: ld returned 1 exit status

That linker command line is wrong; it's missing -mcpu=cortex-m4 -mthumb -Tlayout.ld, which are all given in the target definition.

I noticed that a thumbv7em-none-eabi target hit Rust master, so I've tried several different ways of piping these arguments through that don't rely on the target definition file, including (in .cargo/config):

[target.thumbv7em-none-eabi]
rustflags = [
    "-C",
    "link-arg=-Tlayout.ld",
    "-C",
    "link-arg=-mcpu=cortex-m4",
]

but they also don't make it to the linker commandline.

Help!

If your custom target has the same name as a builtit-in target then rustc will use the built-in target definition and ignore your .json file.

target.thumbv7em-none-eabi.rustflags should work though, but it requires Xargo >=0.1.12 iirc. build.rustflags should work with older Xargo versions.

FWIW, teensy3-rs also run into this and switched from their custom target (+ .json) to the built-in target + .cargo/config. cf. jamesmunns/teensy3-rs-demo#4

Interesting. Are we treating target additions as breaking changes, then?

That change actually appears to be incorrect: it is important for C linkers to receive things like -lm after the object files that use them, but that change eliminates the distinction between pre- and post-link flags.

Interesting. Are we treating target additions as breaking changes, then?

Perhaps. In any case, target specification files are not stable -- they have been broken at least twice before by nightly changes.

it is important for C linkers to receive things like -lm after the object files that use them

Linkers are fun. Do you have more info on why this is needed?

We could add two new rustc flags: -C post-link-arg and -C pre-link-args. My impression is that the tools team in OK-ish with adding rustc flags that match the fields of target specifications.

cc @alexcrichton

Once you start dealing with order of linker args is when my advice would be to stop using the compiler and emit a staticlib instead (and then pass the args yourself). I'd personally prefer to avoid a whole litany of ways to pass linker arguments to the compiler.

Yup. Short version: the linker maintains a table of unresolved symbols. As each object is processed, it removes any symbols resolved by that object, and adds any newly unresolved symbols found in that object. When linking a binary, it requires that table to be empty when all objects have been processed, or it complains of undefined symbols.

So in general, objects must appear on the linker command line as a topological ordering over the dependency graph -- uses first, then definitions. To do otherwise risks confusing link failures.

I've described the process in more detail in the C Production Model reference page for Cobble.

Alright -- but I disagree. :slight_smile: (We met in person on Wednesday, and I said some of what I've written below, so forgive me for repeating.)

We are very close to supporting most embedded use cases using purely Cargo/Xargo. The target post-link-args field as currently implemented by the flexible target specification covers most cases. It just doesn't appear to be possible to override it through Cargo right now.

I believe that, as long as Cargo's build process involves a C-style linker, it is unwise to pretend that the C linker's model is simpler than it is -- which is effectively what the link-args model does today. I have spent a fair amount of work nailing down its semantics in Cobble (particularly here) and would be happy to work on extending Cargo to support them.

As a workaround, we can copy the target spec out of librustc_build into our projects, give it a name that is unlikely to ever be broken by a rustc update (e.g. including the project name in the target name), and alter the post-link-args. This is sufficient for link environment alterations performed by the top-level target.

1 Like

Nice!

It's definitely true yeah that there's a cliff between generating a staticlib and not typically when working with linkers. Especially if the delta is small it's unfortunate to fall off the cliff so easily!

If it's alright with you, I'd like to dig in a bit and see what's going on here. Where are the memset, __libc_init_array, etc, symbols defined? Or I guess put another way, what's the linker argument that needs to be passed to fix this link problem?

Down we go! :slight_smile:

The undefined symbols you're seeing are from the inappropriate inclusion of startfiles, specifically crt0.o.

  • memset is used to clear bss and would normally come from some combination of -lgcc, -lgloss, and -lc, depending on the platform.
  • __libc_init_array is normally exposed by the linker script.

Passing through -nostartfiles would "fix" this, but it isn't actually necessary, because...

...the fundamental problem is the first line, which reads (in part):

fp16.0.o uses VFP register arguments, fp16 does not

This is a sign that different parts of the application have been compiled targeting different ARM processor variants. Specifically, rustc is using the thumbv7em-none-eabihf target, which specifies the Thumb v2 instruction set, with DSP extensions and floating point, microcontroller subset, hard float ABI.

But when gcc is invoked to act as linker, no information about the triple is passed. ARM processors in particular have enough variants that it's critical to pass information about the processor to the linker:

  1. It uses it to fill in special ELF sections used to detect inappropriate mixing of architectures, and
  2. It controls GCC's multilib behavior, which will choose different versions of crt0.o and libc compiled for the appropriate instruction set variant.

Basically: the thumbv7em-none-eabi target now built into rustc is slightly wrong, in that it won't work correctly unless the application adds link-args. At minimum we need to select the right instruction set (!):

  • -mthumb

Then we must control multilib lookup for -lm and friends:

  • -mcpu=cortex-m4 (or m3 or m7 or...)
  • -mfloat-abi=hard
  • -mfpu=fpv4-sp-d16

...because the toolchain has not one libc, but many:

./lib/thumb/libc.a
./lib/armv7-ar/thumb/libc.a
./lib/armv7-ar/thumb/softfp/libc.a
./lib/armv7-ar/thumb/fpu/libc.a
./lib/armv7-m/libc.a
./lib/libc.a                 <-- currently getting this
./lib/armv6-m/libc.a
./lib/fpu/libc.a
./lib/armv7e-m/libc.a
./lib/armv7e-m/softfp/libc.a
./lib/armv7e-m/fpu/libc.a     <-- we want this

Note that the information GCC wants to receive is modeled differently from the way LLVM represents it. In particular, the switches for specifying ARM architecture variants are less flexible, and are mostly controlled by -mcpu. To make the rustc Cortex-M targets useful out-of-the-box without writing .cargo/config in every project, we might need to separate them into per-CPU variants that contain the right GCC link flags. I'm happy to take a crack at this if nobody objects.

Anyway.

We can fix this problem (needing to specify the architecture and variant) using linker flags that are not position-sensitive. They conventionally go first, i.e.:

ld $(PRE_ARGS) $(OBJECTS)

But they can also technically go after; see below.


The code @japaric linked was also adding arguments to model universal dependencies on C libraries. This is not unusual; on ARM, you'll almost certainly need -lgcc. The gcc linker frontend normally stuffs it in for you, unless you use -nodefaultlibs (which rustc appears to be adding to the flags by default) or -nostdlib.

You may need -lm depending on how Rust compiles operations like sin, or if you use any fp64, since the variant in question only supports single-precision floating point.

That code is also using linker object groups. This is an emergency feature for handling cyclic dependencies between objects, which are normally processed in the linear order I described in my last post. It's most commonly deployed for handling the (somewhat necessary) interdependence of -lgcc, -lc, and -lm (and in this case Newlib's -lnosys).

For correctness, all of these -lfoo flags, including the group, need to appear last in the list of objects, i.e. in this position:

ld $(PRE_ARGS) $(OBJECTS) $(DEPENDENCIES)

Now, as it turns out, rustc is adding flags provided by -C link-arg=foo on the end of the linker command line. With GNU ld, I believe that almost all the relevant switches are insensitive to their precise location on the command line, so it should be legit to do this:

ld $(OBJECTS) $(PRE_ARGS) $(DEPENDENCIES)

The main exception is -U. It is occasionally useful to add -U foo to the linker command line, which explicitly adds an undefined symbol to the symbol table (at the point in argument processing where the flag appears). By putting this on the command line before the linked objects, you ensure that definitions of the named symbols are collected from objects (which otherwise isn't guaranteed to happen, particularly in the presence of --gc-sections).

So that's the main case where using GNU ld really needs a "pre" link arg, in my experience. There are some other obscure cases.

I have mostly had to do this in cases where a library deliberately leaves an undefined 'hook' symbol to be filled in by the application, as in some of FreeRTOS's error handlers. I usually encounter this when I'm cramming pre-existing software into my build system; you can probably get pretty far while ignoring this use case.

Finally, there are cases where it is useful to emit flags alongside objects in the middle section of the linker command line. You've already seen object groups used in teensy3-rs-demo; --whole-archive is another useful case that I discuss here.

Cobble models these two concepts separately using the link_flags key (switches that go to the left of the objects on the command line) and the link_srcs key (objects -- and flags! -- that are collected across the whole graph and topologically sorted). Any library target can emit both link_flags and link_srcs into the environment of any user, so they are accumulated into the top-level (typically executable) target.

3 Likes

Thanks for the detailed explanation! Definitely sounds like the compiler should pass -mcpu and such flags by default for various targets for multilib-powered gcc "linkers" we're using, and feel free to either open an issue or send a PR for that!

In terms of where -lgcc and friends come into play, we currently prefer for crates to declare these dependencies so they can all get sorted in the DAG. The compiler already does a topological sort of crate dependencies as well as their native dependencies for when calling the linker (exactly for the reasons you've specified), so as long as a crate links to the gcc native library it'll get thrown into the correct spot (in theory). In that sense is it possible for a crate to be compiled with -lgcc?