Rust-lld, how to troubleshoot 'section' ... 'will not fit in region' errors?

Hello

I have been struggling to find information on how to troubleshoot rust-ldd errors with a message like this:

  = note: rust-lld: error: section '.text' will not fit in region 'FLASH': overflowed by 123696 bytes
          rust-lld: error: section '.rodata' will not fit in region 'FLASH': overflowed by 125724 bytes
          rust-lld: error: section '.data' will not fit in region 'FLASH': overflowed by 125780 bytes
          rust-lld: error: section '.gnu.sgstubs' will not fit in region 'FLASH': overflowed by 125792 bytes

I tried using:

cargo objdump -- -D > /tmp/disassembly 

but this fails with the same error as above, and I am not sure if this would lead me to understand what the issue might be.

Below is the description of the specific issue.

(On AHB SMRAM2, STM32H7) Using a raw pointer to an array of raw pointers I define a structure like this:

[*mut &str, ... , *mut &str], &str, .... &str
 |                            |
 0x30004000 (AHBSRAM2)        0x30004800 (First addr. after pointers to messages)

Concrete:

    // Pointer to array containg poiners to messages.
    let pointers_msg = AHB_SRAM2_FIRST as *mut [*mut &str; N_MSG_MAX];
    // Pointer to default message after the space intended for pointers_msg.
    let msg_default = unsafe { pointers_msg.offset(1) }
        as *mut [u8; MSG_DEFAULT.len()]
        as *mut &str;
    // Writing default message into its place. 
    unsafe { core::ptr::write_volatile(msg_default, MSG_DEFAULT) };
    // Setting all pointers to the location of the default message.
    unsafe { core::ptr::write_volatile(pointers_msg, [msg_default; N_MSG_MAX]) };

All positions in the array now point to the default message. Now I copy a new message to memory, placed after the default message, and point to it from the first position (0) of the pointer array (pointers_msg).

    // ERROR
    let msg_error = unsafe { msg_default.offset(1) }
        as *mut [u8; ERROR.len()]
        as *mut &str;
    unsafe { core::ptr::write_volatile(msg_error, ERROR) };
    unsafe { core::ptr::write_volatile( &mut (*pointers_msg)[0], msg_error); }

In the same init() function, i test if I can access the messages:

defmt::println!("{}\n{}\n{}\n{}\n{}",
        unsafe { pointers_msg.offset(1) },
        unsafe { *(*pointers_msg)[0] },
        unsafe { *(*pointers_msg)[1] },
        unsafe { *(*pointers_msg)[2] },
        unsafe { *(*pointers_msg)[3] },
        unsafe { *(*pointers_msg)[4] },
    );

This prints as expect:

0x30004800
Error
Denied
BankConfig can only be created once.
Maximum number of banks reached.
Bank overlaps with previous one.

A function to access a string using a matching pointers_msg. This way, I can access any message as long as i know what position in the array points to it (would improove this with an enum). If the initalisation was not run, I do that now:

pub fn get_msg(n: usize) -> &'static str {
    if unsafe { (&raw mut INIT).read() } {
        // Pointer to array containg poiners to messages.
        let pointers_msg = AHB_SRAM2_FIRST as *mut [*mut &str; N_MSG_MAX];
        unsafe { *(*pointers_msg)[n] }
    } else {
        init();
        get_msg(n)
    }
}

And it works:

defmt::println!("get_msg(4): {}", msg::get_msg(4));
get_msg(4): Bank overlaps with previous one.

Or does it? Calling the above function from a struct method, i get the rust-lld error shown on top of this page. Here is a bit of context:

pub fn add_bank(&mut self, addr: u32, size: u32)
            -> Result<(), crate::err::E<()>>
    {
        defmt::println!("{}", crate::msg::get_msg(4))
...

I am stuck here.
Things i tried:

  • cargo clean.
  • Commenting the "-C", "linker=flip-link", line in .cargo/config.toml.
  • Trying opt-level = 's' instead of opt-level = 3 in the dev profile in Cargo.toml.
  • Having get_msg() returning the pointer rather than &str (Getting a bit desperate).
  • Removing the INIT flag handling in the init() and/or get_msg() functions, which had some success, but I can't life without the INIT check and seemed too random to be a reliable "solution".
  • Searching online. I thought this could be an issue with how I use raw pointers, but other than better ways to access mutable statics, nothing meaningful to me came up. Same when searching for the error message directly. I have the impression that what I am seeing is not a common occurrence.

can you post your memory.x? What is the value of N_MSG_MAX?

it uncommon the .text section would overflow the FLASH region, unless you (accidentally) created a monomorphization explosion. it's possible, but rare in embedded code.

I think it's more likely that you either made a mistake in the memory.x, or you are using a wrong memory.x for the target chip.

so my first suggestion is to check the content of the memory.x file, especially if you created it yourself or edited manually.

if the memory.x is copied or generated from a template, then it's probably correct, in which case you should check for excessive monomorphization, e.g. check which generic types are used in many places, each instantiated with different type arguments.

you mentioned the add_bank() method, how's the type defined? specifically, does it have generic type parameters?

if you can't find the culprit by looking at the code, you can try cargo bloat to see which functions are taking too much space.

By the way, if you want to get it to compile, to then objdump, but not to flash, you can just temporarily increase the size of the section, debug the problem and shrink back.

I use DTCM0 and DTCM1 combined for RAM. The FLASH is indeed only 64 KiB. In its working state, the program uses about 32 KiB.

MEMORY
{
    FLASH : ORIGIN = 0x08000000, LENGTH =   64K /* BANK_1 */
    /*RAM   : ORIGIN = 0x24000000, LENGTH =  456K *//* SRAM1 + SRAM2 + SRAM3 + SRAM4 */
    /*ITCM  : ORIGIN = 0x00000000, LENGTH =   64K */ /* Instruction Tightly Coupled Memory */
    RAM : ORIGIN = 0x20000000, LENGTH = 64K /* Data Tightly Coupled Memory 0 allocation and 1 allocation as system RAM. */
    /*DTCM0 : ORIGIN = 0x20000000, LENGTH = 32K */ /* Data Tightly Coupled Memory 0 allocation. */
    /*DTCM1 : ORIGIN = 0x20008000, LENGTH = 32K*/ /* Data Tightly Coupled Memory 0 allocation. */
}

const N_MSG_MAX: usize = 512;

My understanding is that this should not influence the usage in FLASH, as it is "just" a pointer? I did try with values of 20 and less, to see if I made some kind of mistake, no change. I did not touch this value during most trials. I am happy to look more into this, though if you have proposals.

I just read the Wikipedia article of what monomorphization is. If my understanding is correct, this would occur if I have generic type parameters. I checked for this, including the method from where I call get_msg() and if at all, I use primitive types. Very interesting to learn about this.

I was conserned about this too, but believe to have double checked everything. What wonders me is the step jump form using about 32 KiB (halv of 64 KiB) to being 123 KiB over (effectively tripple 64 KiB), by solely calling get_msg() from a struct method rather than main. Absolutely no other change.

I did indeed edit memory.x manually. The memory layout can be found on page 151 here: https://www.st.com/resource/en/reference_manual/rm0477-stm32h7rx7sx-armbased-32bit-mcus-stmicroelectronics.pdf Here is a screenshot and highlight of the relevant parts if someone wishes to have a look themselves (I have also marked the start address of AHB - SRAM2, while I was at it):

I would be happy to share all source files. I am an EE and have just started a rewrite of this hobby project, so no harm done sharing it.

I hope this is sufficient code to answer this:
The final struct and its builder:

pub struct BankConfig {
    transfer_states: [TransferState; MAX_BANKS],
    last_index: usize,
}

pub struct BankConfigBuilder {
    bc: BankConfig,
}

The specific method I am calling. I would like to replace the text strings with calls to crate::msg::get_msg(N) (I surrounded the "offending" line with comments containing !!!):

impl BankConfigBuilder {
...
    pub fn add_bank(&mut self, addr: u32, size: u32)
            -> Result<(), crate::err::E<()>>
    {
        // !!! Commenting this line and the overflow error is gone.
        defmt::println!("{}", crate::msg::get_msg(4));
        // !!!

        // Check if maxium number of banks reached.
        if self.bc.last_index == MAX_BANKS - 1 {
            return Err(
                crate::err::E::Denied(
                    "Maximum number of banks reached.",
                    ()
                )
            )
        }
        if self.bc.last_index > 0 {
            // Check if previous bank overlaps with current bank.
            if
                    self.bc.transfer_states[self.bc.last_index - 1].target
                    +
                    self.bc.transfer_states[self.bc.last_index - 1].target_size
                    >=
                    addr
            {
                return Err(
                    crate::err::E::Denied(
                        //crate::msg::get_msg(4),
                        "Bank overlaps with previous one.",
                        ()
                    )
                )
            }
            if self.bc.transfer_states[self.bc.last_index - 1].target > addr {
                return Err(
                    crate::err::E::Denied(
                        "Banks shall be added by increasing order of their \
                        address.",
                        ()
                    )
                )
            }
        } 
        // Record data.
        self.bc.transfer_states[self.bc.last_index].target = addr;
        self.bc.transfer_states[self.bc.last_index].target_size = size;
        // Increment bank counter.
        self.bc.last_index += 1;
        Ok(())
    }
...

cargo bloat fails to run when (the the FLASH overflow error) when calling crate::msg::get_msg(N) from the BankConfigBuilder struct. I commented the call there and uncommeted the call in in main, so that get_msg() is called but the code compiles:

    Finished `dev` profile [optimized + debuginfo] target(s) in 0.02s
    Analyzing target/thumbv7em-none-eabihf/debug/nucleo-h7s3l8__stm32h7s3l8h6

File  .text    Size                        Crate Name
0.2%  66.7% 17.8KiB nucleo_h7s3l8__stm32h7s3l8h6 nucleo_h7s3l8__stm32h7s3l8h6::init_bank_config
0.0%   5.8%  1.6KiB nucleo_h7s3l8__stm32h7s3l8h6 nucleo_h7s3l8__stm32h7s3l8h6::init
0.0%   4.6%  1.2KiB                          std core::str::count::do_count_chars
0.0%   3.1%    834B nucleo_h7s3l8__stm32h7s3l8h6 nucleo_h7s3l8__stm32h7s3l8h6::msg::init
0.0%   2.7%    730B                          std compiler_builtins::mem::memcpy
0.0%   2.3%    614B                          std <core::fmt::Formatter>::pad_integral
0.0%   1.7%    466B                          std <core::fmt::Formatter>::pad
0.0%   1.6%    448B                    defmt_rtt <defmt_rtt::Logger as defmt::traits::Logger>::write
0.0%   1.0%    268B                          std core::fmt::write
0.0%   1.0%    266B                    defmt_rtt <defmt_rtt::Logger as defmt::traits::Logger>::release
0.0%   0.9%    234B                          std <u32 as core::fmt::Display>::fmt
0.0%   0.8%    210B                    defmt_rtt defmt_rtt::channel::Channel::write_impl
0.0%   0.6%    172B                          std <core::panic::panic_info::PanicInfo as core::fmt::Display>::fmt
0.0%   0.6%    156B                    [Unknown] __aeabi_memcpy4
0.0%   0.5%    148B                    defmt_rtt _defmt_acquire
0.0%   0.5%    142B                          std <u32 as core::fmt::LowerHex>::fmt
0.0%   0.5%    138B                          std core::slice::index::slice_index_fail
0.0%   0.5%    138B                          std core::fmt::Write::write_char
0.0%   0.4%    122B                    defmt_rtt defmt_rtt::channel::Channel::blocking_write
0.0%   0.3%     88B                  panic_probe __rustc::rust_begin_unwind
0.0%   3.7%    998B                              And 46 smaller methods. Use -n N to show more.
0.3% 100.0% 26.6KiB                              .text section size, the file size is 8.0MiB

init_bank_config() creates a new BankConfigBuilder, which holds a BankConfig, which holds transfer_states. MAX_BANKS in the transfer_states array was set 1024. For fun I tried to change it to 10, uncommented the get_msg() call in pub fn add_bank(&mut self, addr: u32, size: u32) and it worked.

pub struct BankConfig {
    transfer_states: [TransferState; MAX_BANKS],
    last_index: usize,
}

That is a great leap forward. I do however not understand how the call to get_msg() is related to the size of transfer_states. This is how ´TransferState´ looks like:

#[derive(Clone, Copy)]
struct TransferState {
    source: u32,
    buffer: u32,
    target: u32,
    target_size: u32,
    target_remain: u32,
}

I do not see how this could grow form 1024 * 5 * 4 = 20 KiB to anything bigger by calling a function which does not operate on this sturct, or for that matter anything other than AHB SRAM2. If a kind soul wishes to look at init_bank_config(), its here below. The parts where I added (DUMMY, testing) in the comment, are meant to cause one of the error messages be triggered:

fn init_bank_config() -> Option<memory::BankConfig> {
    match memory::BankConfigBuilder::new() {
        Ok(mut bcb) => {
            let mut bcb_option = Some(bcb);
            loop {
                loop {
                    // bcb_option = None; // Can be triggered by up-link com to
                                          // abort.
                    //defmt::println!("Adding bank entry.");
                    if let Some(ref mut bcb) = bcb_option {
                        // Adding bank information (DUMMY, testing).
                        match bcb.add_bank(0x10000, 0xffff) {
                            Ok(_) => defmt::println!("Ok."),
                            Err(err) => defmt::println!("{}: {}",
                                err.kind(), err.msg()),
                        }
                        // Adding bank information (DUMMY, testing).
                        match bcb.add_bank(0x0, 0xffff) {
                            Ok(_) => defmt::println!("Ok."),
                            Err(err) => defmt::println!("{}: {}",
                                err.kind(), err.msg()),
                        }
                    }
                    break; // Getting out of this loop would be triggered by 
                           // up-link com.
                }
                if let Some(bcb) = bcb_option.take() {
                    // Trying to retrive bank configuration.
                    match bcb.get() {
                        Ok(bc) => return Some(bc), // Valid bank configuration.
                        Err(err) => {
                            defmt::println!("{}: {}", err.kind(), err.msg());
                            if let Some(bcb) = err.get_type() {
                                bcb_option = Some(bcb); // Able to continue.
                            } else {
                                bcb_option = None; // Unable to continue.
                            }
                        },
                    }
                } else {
                    defmt::println!("Aborted.");
                    return None;
                }
            }
        },
        Err(err) => defmt::println!("{}: {}", err.kind(), err.msg()),
    }
    None
}

The BankConfigBuilder can only be instantiated ones. When building BankConfig, BankConfigBuilder is consumed:

    pub fn get(self)
            -> Result<BankConfig, crate::err::E<BankConfigBuilder>>
    {
        unsafe { INIT = true; }
        if self.bc.last_index == 0 {
            Err(
                crate::err::E::Denied(
                    "At least one target bank needs to be defined.",
                    self
                )
            )
        } else {
            Ok(self.bc)
        }
    }

The get_msg() call in add_bank() completely breaks my understanding of what is going on.

That is a great idea!

My understanding is that this should not influence the usage in FLASH, as it is "just" a pointer?

You are constructing a value:

 // Setting all pointers to the location of the default message.
unsafe { core::ptr::write_volatile(pointers_msg, [msg_default; N_MSG_MAX]) };

That value could be stored on the stack, or if it doesn't depend on any runtime inputs, the compiler could store it in the executable as an optimization.

But it's unlikely that is is responsible for your size problem.

A single function compiled to 17 kilobytes of code -- ouch!

Also try tagging init() and/or get_msg() with #[inline(never)].

that's correct. monomophization only happens when you use generic type parameters. rust uses generic lifetime parameters for typecheck, but they don't get monomorphized for codegen.

also, I forgot another potential reason of code bloat: excessive inlining. but this typically should not be the major cause when you use opt-level=s. but if in doubt, try add #[inline(never)] to the largest functions, as others suggested.

this could be a side effect that when you called get_msg() from one method, add_bank() in this case, it prevents some other potentially bloated code from being deadcode-eliminated. just a guess.

you may also try to increase the FLASH region temporarily instead to get a more accurate report.

but even with the report so far, it look like a single function init_bank_config() is taking up 2/3 of code size, which is rather odd, so better to look into this function first.

this might prove my above guess about why calling get_msg() from the get_bank() method (as opposed to from main()) triggers the linker error, since add_bank() not only calls get_msg(), it also calls init(), which I assume is related to init_bank_config(). for one, this keeps init() from being stripped if it were otherwise not called. and what's more, if for some reason, init() (or init_bank_config()) got inlined into add_bank(), the code size could double or triple, matching your observation.

I tried to mark init() and/or get_msg() with #[inline(never)], no effect. Then I marked all BankConfigBuilder methods #[inline(never)]. Not only did the problem disappear, it now only programs about 14 KiB, down form around 30 KiB:

      Erasing ✔ 100% [####################]  16.00 KiB @ 396.19 KiB/s (took 0s)
  Programming ✔ 100% [####################]  14.00 KiB @  91.20 KiB/s (took 0s)

I am still pusled by how calling get_msg() makes a minor issue so much worse, then i read on:

Thank you for sharing these thoughts. I think they explain things nicely, especially:

I use opt-level=3. As the methods I now marked with #[inline(never)] are for initial setup only and will never be the bottle neck, I am happy that they are not inlined (I shall consider this actively in the future).

Thank you everybody for helping out. After failing to find anything online for hours and hours, my hopes where somewhat low this could be solved. Tremendously appreciate your time and genuine effort.


CORRECTION:

codegen-units only affects codgen performance, but does NOT affect the sections' arrangement in objects: rustc already put each function into a separate section.

see follow up posts below.


in addition to opt-level, other compiler flags might also affect the code side (in subtle ways), but it also depends heavily on the use case.

for example. you can tinker with lto and codegen-units to see how they change the generated binary size. this could make a huge difference in certain scenarios, e.g. when there are a lot of machine generated code like derived trait impls in the same module but only a fraction of them are actually used.

the main reason is, the linker's reachability analysis can only "throw away" code by sections, not even per-function or per-module! this is very limited compared to the frontend optimizer, which can eliminate dead code in very fine granularity: e.g. it can omit code for a single expression based on data flow analysis.

lto enables the linker to do finer-grained data flow analysis in IR, and the codegen-units can split the generated code of a single crate into multiple smaller object files, but the sections in the objects remains the same. both lto can help the linker remove more "dead" code in some cases.

More codegen units doesn't help the linker remove dead code at all. Dead code is removed on a section by section basis with --gc-sections. Rustc already puts every function and static in a separate section. If anything more codegen units increases duplication of #[inline] and generic functions across codegen units. With LTO LLVM may merge them together again, or it may not.

thansk for the correction.

I didn't know rustc put functions into separate sections. this is news to me. I saw one object and just assumed there's one section in it.