How do I set system library root for cross compilation?

I want to build https://github.com/kubos/kubos for Rock Pi in a x86 machine. This depends on several system libraries, which are libsqlite3, libssl and libpthread.
To do this, I installed aarch64-linux-gnu- cross compiler on my computer, and set the .cargo/config target so that the linker points to aarch64-linux-gnu-gcc. To provide aarch64 system libraries for my x86, I copied /usr and /lib folder from my Rock Pi into folder cross_compile/rockpi that is in my x86 computer.

How do I set Cargo to compile using system libraries in cross_compile/rockpi instead of my x86 system library?