PKG_CONFIG_ALLOW_CROSS=1 breaks things, because pkg-config does not support cross-compilation. pkg-config is a simple tool that doesn't understand that concept at all, which is why it gets disabled to avoid causing problems.
It's theoretically possible to make pkg-config not break cross-compilation, but it takes a significant amount of work. It's not a matter of setting a variable, but you need to actually install arm-compatible libraries, get arm-compatible .pc files for them, and then override pkg-config directories (setting so-called sysroot) to make it see only the arm-specific versions.
Try to build everything static/vendored instead.