My toolchain is generating bad ELFs, need to track down the culprit

I have an app for the Arduino (AVR architecture). It uses FFI to call existing libraries (neopixel, arducam, ethernet). I used to be able to build working apps with it, but recently I upgraded the software on my development machine, and now my apps do not work.

I used a docker container to build the same source code, and the ELF it generates works as expected.

If I use avr-objdump on the two ELFs, there are differences, but I do not have the experience to guess which part of my toolchain is causing the differences, or which exact differences are important.

I suspect it has something to do with my call to the arduino C library raw::init() method. Leaving out that call allows the app to work when compiled by both machines.

avr-objdump -d target/avr-atmega328p/debug/mini-cam-2.elf > /var/tmp/y.asm

$ diff -uw /tmp/{x,y}.asm    
--- /tmp/x.asm  2022-10-31 14:20:20.216485232 -0400
+++ /tmp/y.asm  2022-10-31 14:20:34.847532412 -0400
@@ -44,7 +44,7 @@
        11 e0           ldi     r17, 0x01
        a0 e0           ldi     r26, 0x00
        b1 e0           ldi     r27, 0x01
-       ea e0           ldi     r30, 0x0A
+       e6 e0           ldi     r30, 0x06
        f3 e0           ldi     r31, 0x03
        02 c0           rjmp    .+4      
        05 90           lpm     r0, Z+
@@ -66,8 +66,8 @@
        ae 34           cpi     r26, 0x4E
        b2 07           cpc     r27, r18
        e1 f7           brne    .-8      
-       0e 94 63 00     call    0xc6
-       0c 94 83 01     jmp     0x306
+       0e 94 6b 00     call    0xd6
+       0c 94 81 01     jmp     0x302
 
 000000a2 <__bad_interrupt>:
        0c 94 00 00     jmp     0
@@ -88,11 +88,19 @@
        a1 f7           brne    .-24     
        08 95           ret
 
-000000c6 <main>:
-       0e 94 67 00     call    0xce
-       0e 94 7e 01     call    0x2fc
+000000c6 <core::panicking::panic_fmt>:
+       0e 94 ec 00     call    0x1d8
+       0e 94 7c 01     call    0x2f8
 
-000000ce <mini_cam_2::__avr_device_rt_main>:
+000000ce <core::panicking::panic>:
+       0e 94 63 00     call    0xc6
+       0e 94 7c 01     call    0x2f8
+
+000000d6 <main>:
+       0e 94 6f 00     call    0xde
+       0e 94 7c 01     call    0x2f8
+
+000000de <mini_cam_2::__avr_device_rt_main>:
        8f 92           push    r8
        9f 92           push    r9
        af 92           push    r10
@@ -103,7 +111,7 @@
        ff 92           push    r15
        0f 93           push    r16
        1f 93           push    r17
-       0e 94 3d 01     call    0x27a
+       0e 94 3b 01     call    0x276
        9f b7           in      r25, 0x3f
        f8 94           cli
        80 91 44 01     lds     r24, 0x0144
@@ -204,25 +212,16 @@
        91 e0           ldi     r25, 0x01
        6b e2           ldi     r22, 0x2B
        70 e0           ldi     r23, 0x00
-       0e 94 e8 00     call    0x1d0
-       0e 94 7e 01     call    0x2fc
-
-000001c8 <core::panicking::panic_fmt>:
-       0e 94 ec 00     call    0x1d8
-       0e 94 7e 01     call    0x2fc
-
-000001d0 <core::panicking::panic>:
-       0e 94 e4 00     call    0x1c8
-       0e 94 7e 01     call    0x2fc
+       0e 94 67 00     call    0xce
+       0e 94 7c 01     call    0x2f8
 
 000001d8 <rust_begin_unwind>:
        ff cf           rjmp    .-2      
 
 000001da <__vector_16>:
        1f 92           push    r1
-       0f 92           push    r0
-       0f b6           in      r0, 0x3f
-       0f 92           push    r0
+       1f b6           in      r1, 0x3f
+       1f 92           push    r1
        11 24           eor     r1, r1
        2f 93           push    r18
        3f 93           push    r19
@@ -234,29 +233,39 @@
        df 93           push    r29
        cd b7           in      r28, 0x3d
        de b7           in      r29, 0x3e
+
+000001f6 <.LM1>:
        80 91 46 01     lds     r24, 0x0146
        90 91 47 01     lds     r25, 0x0147
        a0 91 48 01     lds     r26, 0x0148
        b0 91 49 01     lds     r27, 0x0149
+
+00000206 <.LM2>:
        30 91 45 01     lds     r19, 0x0145
+
+0000020a <.LM3>:
        23 e0           ldi     r18, 0x03
        23 0f           add     r18, r19
+
+0000020e <.LM4>:
        2d 37           cpi     r18, 0x7D
-       20 f4           brcc    .+8      
+       60 f5           brcc    .+88     
+
+00000212 <.LM5>:
        01 96           adiw    r24, 0x01
        a1 1d           adc     r26, r1
        b1 1d           adc     r27, r1
-       05 c0           rjmp    .+10     
-       26 e8           ldi     r18, 0x86
-       23 0f           add     r18, r19
-       02 96           adiw    r24, 0x02
-       a1 1d           adc     r26, r1
-       b1 1d           adc     r27, r1
+
+00000218 <.L3>:
        20 93 45 01     sts     0x0145, r18
+
+0000021c <.LM7>:
        80 93 46 01     sts     0x0146, r24
        90 93 47 01     sts     0x0147, r25
        a0 93 48 01     sts     0x0148, r26
        b0 93 49 01     sts     0x0149, r27
+
+0000022c <.LM8>:
        80 91 4a 01     lds     r24, 0x014A
        90 91 4b 01     lds     r25, 0x014B
        a0 91 4c 01     lds     r26, 0x014C
@@ -268,6 +277,8 @@
        90 93 4b 01     sts     0x014B, r25
        a0 93 4c 01     sts     0x014C, r26
        b0 93 4d 01     sts     0x014D, r27
+
+00000252 <.LM9>:
        df 91           pop     r29
        cf 91           pop     r28
        bf 91           pop     r27
@@ -276,86 +287,131 @@
        8f 91           pop     r24
        3f 91           pop     r19
        2f 91           pop     r18
-       0f 90           pop     r0
-       0f be           out     0x3f, r0
-       0f 90           pop     r0
+       1f 90           pop     r1
+       1f be           out     0x3f, r1
        1f 90           pop     r1
        18 95           reti
 
-0000027a <init>:
+0000026a <.L2>:
+       26 e8           ldi     r18, 0x86
+       23 0f           add     r18, r19
+
+0000026e <.LM11>:
+       02 96           adiw    r24, 0x02
+       a1 1d           adc     r26, r1
+       b1 1d           adc     r27, r1
+       d1 cf           rjmp    .-94     
+
+00000276 <init>:
        cf 93           push    r28
        df 93           push    r29
        cd b7           in      r28, 0x3d
        de b7           in      r29, 0x3e
+
+0000027e <.LM49>:
        78 94           sei
+
+00000280 <.LM50>:
        84 b5           in      r24, 0x24
        82 60           ori     r24, 0x02
        84 bd           out     0x24, r24
+
+00000286 <.LM51>:
        84 b5           in      r24, 0x24
        81 60           ori     r24, 0x01
        84 bd           out     0x24, r24
+
+0000028c <.LM52>:
        85 b5           in      r24, 0x25
        82 60           ori     r24, 0x02
        85 bd           out     0x25, r24
+
+00000292 <.LM53>:
        85 b5           in      r24, 0x25
        81 60           ori     r24, 0x01
        85 bd           out     0x25, r24
+
+00000298 <.LM54>:
        ee e6           ldi     r30, 0x6E
        f0 e0           ldi     r31, 0x00
        80 81           ld      r24, Z
        81 60           ori     r24, 0x01
        80 83           st      Z, r24
+
+000002a2 <.LM55>:
        e1 e8           ldi     r30, 0x81
        f0 e0           ldi     r31, 0x00
        10 82           st      Z, r1
+
+000002a8 <.LM56>:
        80 81           ld      r24, Z
        82 60           ori     r24, 0x02
        80 83           st      Z, r24
+
+000002ae <.LM57>:
        80 81           ld      r24, Z
        81 60           ori     r24, 0x01
        80 83           st      Z, r24
+
+000002b4 <.LM58>:
        e0 e8           ldi     r30, 0x80
        f0 e0           ldi     r31, 0x00
        80 81           ld      r24, Z
        81 60           ori     r24, 0x01
        80 83           st      Z, r24
+
+000002be <.LM59>:
        e1 eb           ldi     r30, 0xB1
        f0 e0           ldi     r31, 0x00
        80 81           ld      r24, Z
        84 60           ori     r24, 0x04
        80 83           st      Z, r24
+
+000002c8 <.LM60>:
        e0 eb           ldi     r30, 0xB0
        f0 e0           ldi     r31, 0x00
        80 81           ld      r24, Z
        81 60           ori     r24, 0x01
        80 83           st      Z, r24
+
+000002d2 <.LM61>:
        ea e7           ldi     r30, 0x7A
        f0 e0           ldi     r31, 0x00
        80 81           ld      r24, Z
        84 60           ori     r24, 0x04
        80 83           st      Z, r24
+
+000002dc <.LM62>:
        80 81           ld      r24, Z
        82 60           ori     r24, 0x02
        80 83           st      Z, r24
+
+000002e2 <.LM63>:
        80 81           ld      r24, Z
        81 60           ori     r24, 0x01
        80 83           st      Z, r24
+
+000002e8 <.LM64>:
        80 81           ld      r24, Z
        80 68           ori     r24, 0x80
        80 83           st      Z, r24
+
+000002ee <.LM65>:
        10 92 c1 00     sts     0x00C1, r1
+
+000002f2 <.LM66>:
        df 91           pop     r29
        cf 91           pop     r28
        08 95           ret
 
-000002fc <abort>:
+000002f8 <abort>:
        81 e0           ldi     r24, 0x01
        90 e0           ldi     r25, 0x00
        f8 94           cli
-       0c 94 83 01     jmp     0x306
+       0c 94 81 01     jmp     0x302
 
-00000306 <_exit>:
+00000302 <_exit>:
        f8 94           cli
 
-00000308 <__stop_program>:
+00000304 <__stop_program>:
        ff cf           rjmp    .-2    

The src/main.rs is

#![no_std]
#![no_main]
#![feature(const_option)]

use arduino_hal::{default_serial, delay_ms};
use panic_halt as _;
use ufmt::uwriteln;

#[arduino_hal::entry]
fn main() -> ! {
    rust_arduino_runtime::arduino_main_init();

    //

    let dp = arduino_hal::Peripherals::take().unwrap();

    if true {
        // neopixel needs the clock divisor set to something other than no_clock().  Normally the arduino init() method does that (and a lot of other stuff)
        let tc0 = dp.TC0;
        tc0.tccr0b.write(|w| w.cs0().prescale_64())
    }

    /*if true {
        link_bug_kludge();
    }*/

    let pins = arduino_hal::pins!(dp);

    let mut serial = default_serial!(dp, pins, 115200);

    delay_ms(1000);

    // let _ = uwriteln!(&mut serial, "{}", progmem_display!("hello from Arduino"));
    let _ = uwriteln!(&mut serial, "hello from Arduino");

    loop {
        let _ = uwriteln!(&mut serial, "tick");
        delay_ms(1000);
    }

}

Cargo.toml is

name = "mini-cam-2"
version = "0.1.0"
authors = ["Robert Forsman <git@thoth.purplefrog.com>"]
edition = "2018"
resolver = "2"
[features]
default = ["atmega328p"]
#default = ["atmega2560"]
atmega2560 = [
#    "arduino-spi/atmega2560",
#    "rust-arduino-helpers/atmega2560",
#    "arducam/arduino-mega2560",
    "arduino-hal/arduino-mega2560",
#    "serial-println/atmega2560",
    "atmega-hal/atmega2560"
]
atmega328p = [
#    "arduino-spi/atmega328p",
#    "rust-arduino-helpers/atmega328p",
#    "arducam/arduino-uno",
    "arduino-hal/arduino-uno",
#    "serial-println/atmega328p",
    "atmega-hal/atmega328p"
]


[dependencies]
panic-halt = "0.2.0"
ufmt = "0.1.2"
embedded-hal = "*"

[dependencies.rust-arduino-runtime]
git="https://github.com/mutantbob/rust-arduino-helpers.git"
branch="2022-mar-kludge"
#path = "../../rust-arduino-helpers/rust-arduino-runtime"

[dependencies.avr-hal-generic]
git = "https://github.com/rahix/avr-hal"
rev = "d0d2f243bd3e10b49f6a78d44839a6caa8be7d43"
#path="../../../vendor/avr-hal/avr-hal-generic"

[dependencies.arduino-hal]
git = "https://github.com/rahix/avr-hal"
rev = "d0d2f243bd3e10b49f6a78d44839a6caa8be7d43"
#path="../../../vendor/avr-hal/arduino-hal"

[dependencies.atmega-hal]
git = "https://github.com/rahix/avr-hal"
rev = "d0d2f243bd3e10b49f6a78d44839a6caa8be7d43"
#path="../../../vendor/avr-hal/mcu/atmega-hal"

# Configure the build for minimal size - AVRs have very little program memory
[profile.dev]
panic = "abort"
lto = true
opt-level = "s"

[profile.release]
panic = "abort"
codegen-units = 1
debug = true
lto = true
opt-level = "z"

[patch.crates-io]
## Fixes formatting of u32, however, it causes a link error due to u32-div
## Which in turn is fixed (in config.toml) by:
##     build-std-features = ["compiler-builtins-mangled-names"]
#ufmt = { git = "https://github.com/mrk-its/ufmt.git", branch="ptr_width_16_fix" }
cty = { git="https://github.com/mutantbob/cty.git", branch="2022-Feb-avr" }

If the answer is not immediately apparent I could turn it into a mini github repo with the supporting AVR JSON target specifications and stuff.

Can you be more specific about what "my apps do not work" looks like? Do you mean the binary is malformed and you can't flash it to the device, or is it continuously rebooting, or maybe it's unresponsive because the code is stuck in some fault handler?

If possible, you could attach a debugger. I'm pretty sure Arduinos expose a JTAG interface you can use to figure out what is going on.

I do not see any output from the serial port (using the arduino's serial monitor). A working app displays

hello from Arduino
tick
tick
tick

I am not familiar with JTAG.