Undefined symbol when using a `proc_macro` having a dependency that compiles a C file

rgiot · September 27, 2020, 6:08am

Hello,

I have a crate cpclib_asm that relies on some functions provided by a C file. The executable generated by this crate is properly loaded by my system without issues, and to compile other crates that depend on this one is not problematic neither.

However, there is a link issue when using this crate in a proc_macro (that worked properly before I have added this C code dependency).
I obtain such kind of error

$cargo test
....
error: /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_macros-33b7b1bb80af8bf9.so: undefined symbol: apultra_read_gamma2
 --> cpclib/tests/macros.rs:3:5
  |
3 | use cpclib_macros::*;
  |     ^^^^^^^^^^^^^

...

I guess it is because the compiler does not load the library that contains the C code.
How to make it directly accessible within the proc_macro crate ?

I have to admit, that I do not know where to search documentation about that.
Thanks

bjorn3 · September 27, 2020, 8:09am

What is the full rustc invocation used to compile the proc macro? You can use cargo clean && cargo test -v to show all rustc invocations. What is the output of ldd /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_macros-33b7b1bb80af8bf9.so?

rgiot · September 27, 2020, 4:26pm

Thanks for the tip.
Here are the arguments provided to compile the proc_macro:

   Compiling cpclib-macros v0.4.2-alpha (/home/rgiot/Perso/CPC/rust.cpcdemotools/cpclib-macros)
 Running `rustc --crate-name cpclib_macros --edition=2018 cpclib-macros/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C prefer-dynamic -Cembed-bitcode=no -C debuginfo=2 --test -C metadata=9c56c441baa4aebb -C extra-filename=-9c56c441baa4aebb --out-dir /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps -C incremental=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/incremental -L dependency=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps --extern cpclib_asm=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_asm-c39c1efb841e4856.rlib --extern proc_macro2=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libproc_macro2-ac1024b3ca8a8174.rlib --extern quote=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libquote-54e58014e472048d.rlib --extern syn=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libsyn-f7fe57e16fc85744.rlib --extern proc_macro -L native=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/build/cpclib-asm-863dc4ee93983424/out`
 Running `rustc --crate-name cpclib_macros --edition=2018 cpclib-macros/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -Cembed-bitcode=no -C debuginfo=2 -C metadata=a10b80321666be36 -C extra-filename=-a10b80321666be36 --out-dir /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps -C incremental=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/incremental -L dependency=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps --extern cpclib_asm=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_asm-c39c1efb841e4856.rlib --extern proc_macro2=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libproc_macro2-ac1024b3ca8a8174.rlib --extern quote=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libquote-54e58014e472048d.rlib --extern syn=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libsyn-f7fe57e16fc85744.rlib --extern proc_macro -L native=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/build/cpclib-asm-863dc4ee93983424/out`

I guess this one -L native=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/build/cpclib-asm-863dc4ee93983424/out provides the path to the problematic crate.
In this folder I see the library libapultra.a.

And here the arguments to compile the test of the macro
rustc --crate-name macros --edition=2018 cpclib/tests/macros.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -Cembed-bitcode=no -C debuginfo=2 --test -C metadata=67609f209d763c98 -C extra-filename=-67609f209d763c98 --out-dir /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps -C incremental=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/incremental -L dependency=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps --extern anyhow=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libanyhow-44df969aa9e20fb0.rlib --extern arrayref=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libarrayref-e82c5385d42fa584.rlib --extern as_slice=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libas_slice-d9cfa95945fd184f.rlib --extern bitfield=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libbitfield-1832265fd2a21bd5.rlib --extern bitflags=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libbitflags-56eca94422237621.rlib --extern bitsets=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libbitsets-38c62387ea5a839a.rlib --extern cfg_if=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcfg_if-b00c34326580381c.rlib --extern cpclib=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib-da8e375307415090.rlib --extern cpclib_asm=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_asm-ff927c6b3099ff81.rlib --extern cpclib_basic=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_basic-a1c8e3f6299602da.rlib --extern cpclib_disc=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_disc-00e9549bd4c5f976.rlib --extern cpclib_macros=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_macros-c88db4cd68245f95.so --extern cpclib_sna=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_sna-343050be9b20bd67.rlib --extern cpclib_xfer=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_xfer-14f2f5f98c8df5fb.rlib --extern cpclib_z80emu=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_z80emu-ea2bee3397c6d9b0.rlib --extern custom_error=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcustom_error-504b2e846c5ee0af.rlib --extern dbg=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libdbg-049c630014445224.rlib --extern delegate=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libdelegate-d8222f49fead4b0e.so --extern either=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libeither-0888f912497d85d2.rlib --extern failure=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libfailure-e7a5b45acb0acf34.rlib --extern getset=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libgetset-45f2d69b74d49ade.so --extern gif=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libgif-92ca30934fde21ad.rlib --extern gif_dispose=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libgif_dispose-78509012b3aa28e8.rlib --extern image=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libimage-e9bee0bb5bcb191c.rlib --extern itertools=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libitertools-c63690a293bf4b76.rlib --extern lazy_static=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/liblazy_static-7c0477ae3415ac1f.rlib --extern matches=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libmatches-84413336e46be589.rlib --extern memchr=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libmemchr-1e6b68a09fbeab95.rlib --extern nom=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libnom-31ec0b159dd5cdeb.rlib --extern num=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libnum-099549bc5848d022.rlib --extern num_enum=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libnum_enum-f27d0da4a602f7c2.rlib --extern paste=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libpaste-930acbb686c9fbea.so --extern pretty_assertions=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libpretty_assertions-764d491a2478e0c7.rlib --extern serde=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libserde-ef931f378a4039d7.rlib --extern smallvec=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libsmallvec-d9384f4323c19da4.rlib --extern strsim=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libstrsim-fba319bfb1b25934.rlib --extern tempfile=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libtempfile-81c85eee1c8bd83b.rlib -L native=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/build/cpclib-asm-1228dd20c34deac9/out -L native=/home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/build/curl-sys-f9d6b93b59433fb8/out/build
error: /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_macros-c88db4cd68245f95.so: undefined symbol: apultra_read_bit

And here is the result of the ldd command:

$ ldd /home/rgiot/Perso/CPC/rust.cpcdemotools/target/debug/deps/libcpclib_macros-c88db4cd68245f95.so 
linux-vdso.so.1 (0x00007fff4d3dc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f18ce081000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f18ce05e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f18ce043000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f18cde51000)
/lib64/ld-linux-x86-64.so.2 (0x00007f18cf276000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f18cdd02000)

bjorn3 · September 27, 2020, 5:06pm

github.com

cpcsdk/rust.cpclib/blob/23630cc91c101a63fe29057d784ac37dfefcdda0/cpclib-asm/extra/apultra.c#L1169



   nBit = ((*bits) & 128) ? 1 : 0;

   (*bits) <<= 1;
   (*nCurBitMask) >>= 1;

   *ppInBlock = pInBlock;
   return nBit;
}

 FORCE_INLINE int apultra_read_gamma2(const unsigned char **ppInBlock, const unsigned char *pDataEnd, int *nCurBitMask, unsigned char *bits) {
   int bit;
   unsigned int v = 1;

   do {
      v = (v << 1) + apultra_read_bit(ppInBlock, pDataEnd, nCurBitMask, bits);
      bit = apultra_read_bit(ppInBlock, pDataEnd, nCurBitMask, bits);
      if (bit < 0) return bit;
   } while (bit);

   return v;

The apultra_read_gamma2 function is marked as inline, which means that it will not be included in the generated object file. Instead it is only available for inlining in C files that include that function.

rgiot · September 27, 2020, 5:16pm

Ohh I'm stupid; I have .h version in my folder that is the original source of this .c. It is this one I modified to remove this FORCE_INLINE stuff...
rookie mistake; sorry to make you lost your time !

Obviously doing that on the .c file solve the issue.

thanks

edit: BTW, this apultra_read_gamma2 is a function that is never called from the rust code and should also not be called from a function called from rust. Is it still an expected behavior to have this linking issue ?