Booting `Linux` with a bootloader written in Rust

I've been working on a rust-based bootloader for the raspberry pi 4 .

I'm not sure what I'm missing as I'm not well-versed with Linux but I assume folks who are, may be able to provide some pointers. So, here what I've done so far - follow the Booting AArch64 Linux guide and from what I was able to gather, there are 3 parts to booting Linux -

  • device tree modification:
    • Linux uses a device tree for hardware discovery and we use this to pass boot parameters (i.e. kernel cmd line parameters).
    • We also use the DT to provide basic information about the address and size of the physical memory.
    • So, rather than write my own device tree for rpi4, I decompiled an existing one, edited it with the required info and recompiled it. Its in the dts folder
  • pre-boot checks: The kernel expects a known HW state. So, we'll need to do a few additional things like
    • jump to a lower privilege level (Linux calls this EL1 or exception level 1). By default, the raspberry pi starts executing at EL2 but I've transitioned into the more appropriate EL1.
    • disable the memory management unit MMU. (we never turn it on, so good here).
    • disable interrupts (we never enable interrupts i.e. GICv2 or the global interrupt controller is never turned on, so good again)
  • jump to kernel instruction:
    • After we're done with all of the above, we load the kernel and dtb into RAM and simply jump to the kernel's entry point, passing the device tree as an argument.
    • I've 2 impls for this. Both of them don’t work. Although, the generated assembly (below) checks out.
0000000000089f90 <boot_to_kernel>:
   89f90:       f81f0ffe        str     x30, [sp, #-16]!
   89f94:       aa0003e8        mov     x8, x0
   89f98:       aa0103e0        mov     x0, x1
   89f9c:       aa1f03e1        mov     x1, xzr
   89fa0:       aa1f03e2        mov     x2, xzr
   89fa4:       aa1f03e3        mov     x3, xzr
   89fa8:       d63f0100        blr     x8
   89fac:       d503205f        wfe
   89fb0:       17ffffff        b       89fac <boot_to_kernel+0x1c>

My code manages to make it to the final jump after loading the kernel and dtb but after this point nothing happens i.e. I see no serial output or any indication that the kernel has booted (or failed to boot).

Full output:

[    0.000027] EMMC: reset card.
[    0.000154] Divisor = 63, Freq Set = 396825
[    0.407636] CSD Contents : 00 40 0e 00 32 5b 59 00 00ee 9d 7f 80 0a 40 00
[    0.411465] cemmc_structure=1, spec_vers=0, taac=0x0E, nsac=0x00, tran_speed=0x32,ccc=0x05B5, read_bl_len=0x09, read_bl_partial=0b, write_blk_misalign=0b,read_blk_misalign=0b, dsr_imp=0b, sector_size =0x7F, erase_blk_en=1b
[    0.431096] CSD 2.0: ver2_c_size = 0xEFFD, card capacity: 32026132480 bytes or 32.03GiB
[    0.439001] wp_grp_size=0x0000000b, wp_grp_enable=0b, default_ecc=00b, r2w_factor=010b, write_bl_len=0x09, write_bl_partial=0b, file_format_grp=0, copy=0b, perm_write_protect=0b, tmp_write_protect=0b, file_format=0b ecc=00b
[    0.458726] Divisor = 1, Freq Set = 25000000
[    0.463437] EMMC: Bus width set to 4
[    0.466463] EMMC: SD Card Type 2 HC, 30542Mb, mfr_id: 27, 'SM:EB2MW', r3.0, mfr_date: 8/2017, serial: 0xc8e6576d, RCA: 0x59b4
[    0.477748] EMMC2 driver initialized…

[    0.481484] rpi4 version 0.1.0
[    0.484437] Booting on: Raspberry Pi 4
[    0.488086] Current privilege level: EL1
[    0.491908] Exception handling state:
[    0.495469]       Debug:  Masked
[    0.498597]       SError: Masked
[    0.501724]       IRQ:    Masked
[    0.504852]       FIQ:    Masked
[    0.507980] Architectural timer resolution: 18 ns
[    0.512584] Drivers loaded:
[    0.515277]       1. BCM GPIO
[    0.518144]       2. BCM PL011 UART
[    0.521532] Chars written: 1426
[W   0.524575] wait duration smaller than architecturally supported, skipping
[    0.531349] create new emmc-fat controller...
[    1.309897]  Listing root directory:

[    1.311514]          Found: DirEntry { name: ShortFileName("boot"), mtime: Timestamp(2021-05-07 16:06:08), ctime: Timestamp(2021-05-07 16:06:08), attributes: FV, cluster: Cluster(0), size: 0, entry_block: BlockIdx(16290), entry_offset: 0 }
[    1.331106]          Found: DirEntry { name: ShortFileName("BCM271~1.DTB"), mtime: Timestamp(2022-01-08 18:31:30), ctime: Timestamp(2022-01-08 18:35:56), attributes: FA, cluster: Cluster(65538), size: 25713, entry_block: BlockIdx(16290), entry_offset: 96 }
[    1.353084]          Found: DirEntry { name: ShortFileName("KERNEL8.IMG"), mtime: Timestamp(2022-01-08 18:39:00), ctime: Timestamp(2022-01-08 18:39:36), attributes: FA, cluster: Cluster(154740), size: 131192, entry_block: BlockIdx(16290), entry_offset: 128 }
[    1.375238]          Found: DirEntry { name: ShortFileName("VMLINUZ"), mtime: Timestamp(2022-01-08 18:37:12), ctime: Timestamp(2022-01-08 18:38:16), attributes: FA, cluster: Cluster(97567), size: 29272576, entry_block: BlockIdx(16290), entry_offset: 160 }
[    1.401246]          Found: DirEntry { name: ShortFileName("CONFIG.TXT"), mtime: Timestamp(2022-01-04 11:46:38), ctime: Timestamp(2021-05-07 15:07:00), attributes: FA, cluster: Cluster(1017), size: 1846, entry_block: BlockIdx(17103), entry_offset: 352 }
[    1.420111]          Found: DirEntry { name: ShortFileName("FIXUP4.DAT"), mtime: Timestamp(2021-04-30 14:01:38), ctime: Timestamp(2021-05-07 15:07:00), attributes: FA, cluster: Cluster(1036), size: 5446, entry_block: BlockIdx(17103), entry_offset: 480 }
[    1.444596]          Found: DirEntry { name: ShortFileName("START4.ELF"), mtime: Timestamp(2021-04-30 14:01:38), ctime: Timestamp(2021-05-07 15:07:02), attributes: FA, cluster: Cluster(59166), size: 2228768, entry_block: BlockIdx(54532), entry_offset: 224 }
[    1.465552]          Found: DirEntry { name: ShortFileName("SYSTEM~1"), mtime: Timestamp(2021-09-21 13:57:30), ctime: Timestamp(2021-09-21 13:57:28), attributes: DHS, cluster: Cluster(97564), size: 0, entry_block: BlockIdx(105365), entry_offset: 192 }
[    1.486331]  Get handle to `dtb` file in root_dir...
[    1.491266]  load `dtb` into RAM...
[    1.579706]          loaded dtb: 25713 bytes, starting at addr: 0x3bc33e8
[    1.582924] EOF
[    1.584576]  Get handle to `kernel` file in root_dir...
[    1.590823]  load `kernel` into RAM...
[   94.896272]          loaded kernel: 29272576 bytes, starting at addr: 0x3bcfbe8
[   94.900010] first 40 kernel bytes: [77, 90, 64, 250, 255, 63, 79, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 201, 1, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[   94.914952] EOF

So, I'm kind out of ideas (until I find a HW debugger) but if anyone has any pointers on what I might be missing here, that would be much appreciated.

It works now. Linux has a 2MB base-address alignment requirement. So, instead of relying on the stack for allocation, I edited the linker-script to add custom sections, one for each the dtb and kernel

In case anyone runs into a similar issue, here a link to the thread discussing this.