Well, it only took me 3 years but i think i finally found the bug. The aarch64-unknown-none target has a feature called +strict-align. After enabling it, everything seems to work as it should. I really hope nobody else has to go through the pain of debugging this. I am beyond happy that it's finally working, thank you! ^^