Cross compiling a package that depends on shared libraries

I'm trying to cross-compile the opencv package from my Linux PC to a Raspberry PI Zero 2W. I've set up a cross compiling environment, which was working until I added this dependency. It relies on a library called libopencv-dev, which is properly installed in both machines. But my cross-compilation won't work due to linking issues, and I don't know why.

If I run cargo build --package opencv --target armv7-unknown-linux-gnueabihf, it runs fine. However, when building the whole app with cargo build --target armv7-unknown-linux-gnueabihf, it will print thousands of lines of error, ending with:

  = note: /usr/lib/x86_64-linux-gnu/libopencv_stitching.so: file not recognized: file format not recognized
          collect2: error: ld returned 1 exit status
          

error: could not compile `camera` (bin "camera") due to 1 previous error

/usr/lib/x86_64-linux-gnu/libopencv_stitching.so is the path of a file from libopencv of my machine. In the PI, it is located under /lib/arm-linux-gnueabihf; there are several libopencv shared object files there. The package documentation mentions the following:

  • build OpenCV manually and set up the following environment variables prior to building the project with opencv crate:
    • PKG_CONFIG_PATH for the location of *.pc files or OpenCV_DIR for the location of *.cmake files
    • LD_LIBRARY_PATH for where to look for the installed *.so files during runtime

I don't quite understand PKG_CONFIG_PATH, but I made sure to set LD_LIBRARY_PATH as /lib/arm-linux-gnueabihf before compiling the package. Still, the linking error persists. Is the package trying to make a static link? Could I turn that off?

I am aware that there are a few guides out there that do mention cross compiling this specific package to the Raspberry PI, but they all use Docker for that. I have absolutely no knowledge of Docker, and I really wished I could avoid using it just for achieving cross-compiling. I feel that what I'm trying to do is not that complex...