Undefined references on link with static C library

Hi,

I've created a C binding to the wav2letter c++ framework. The Cmake build works fine.

Next I want to build a Rust executable with it. I link the Cmake output of the static library (libw2l_api.a) in the build.rs. When using cargo build I encounter the first error: the C++ library with the C binding doesn't support PIE.

So I either compile the C library with PIE or I disable PIE via a Rustflag. No matter what I choose I receive countless undefined reference errors (see below). I think the errors are largely the same for both options.

Where does the linking go wrong? From my noob understanding I assumed that the linking is already done (the internal one, from C API to the C++ library etc). That's why I built it with CMake.
Is it correct that Cargo does the linking again?

In more detail the steps I followed:

  1. build C library with -fPIE compile option and then cargo build-> undefined reference errors

see here:

  1. take C lib without PIE enabled and then run cargo build with pie disabled RUSTFLAGS='-C link-args=-no-pie' cargo build

see here:

I also checked whether a missing reference is included in the static library by example via

nm -C ../w2l/build/libw2l_api.a | grep w2l::streaming::IOBuffer::debug
                 U w2l::streaming::IOBuffer::debugString[abi:cxx11]() const
                 U w2l::streaming::IOBuffer::debugString[abi:cxx11]() const

to me it seems that the function is included. So it seems that the C(++) library is fine.

I can provide the source repo if needed/it helps.

This is my build.rs

use std::env;
use std::path::PathBuf;

fn main() {
    println!("cargo:rustc-link-lib=static=stdc++");
    println!("cargo:rustc-link-lib=static=w2l_api");
    println!("cargo:rerun-if-changed=../w2l/src/api.h");
    println!("cargo:rustc-link-search=native=../w2l/build");

    let bindings = bindgen::Builder::default()
        // The input header we would like to generate
        // bindings for.
        .header("../w2l/src/api.h")
        // Tell cargo to invalidate the built crate whenever any of the
        // included header files changed.
        .parse_callbacks(Box::new(bindgen::CargoCallbacks))
        // Finish the builder and generate the bindings.
        .generate()
        // Unwrap the Result and panic on failure.
        .expect("Unable to generate bindings");

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("Couldn't write bindings!");
}

This is my cmake file.

cmake_minimum_required(VERSION 3.5.1)
cmake_policy(VERSION 3.5.1)

set(project_name w2l) ## rename your project here

project(${project_name})

list(APPEND CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake" )
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

set(cereal_INCLUDE_DIRS /root/wav2letter/build/inference/cereal/src/cereal/include)

find_package(Wav2Letter REQUIRED)
find_package(Threads REQUIRED)
find_package(GFlags REQUIRED)

add_subdirectory(3rdparty)

add_library(${project_name}_api
  STATIC
  src/api.cpp
  src/api.h
  ${CMAKE_CURRENT_LIST_DIR}/src/w2l.cpp
  ${CMAKE_CURRENT_LIST_DIR}/src/AudioToWords.cpp
  ${CMAKE_CURRENT_LIST_DIR}/src/Util.cpp
)

target_link_libraries(
  ${project_name}_api
  PUBLIC
    streaming_inference_common
    streaming_inference_modules_nn_backend
    streaming_inference_decoder
    streaming_inference_modules_feature
    decoder-library
    Threads::Threads
)

#target_compile_options(${project_name}_api PRIVATE -fpie)

target_include_directories(
  ${project_name}_api
  PUBLIC
    ${CMAKE_CURRENT_LIST_DIR}
    ${cereal_INCLUDE_DIRS}
    ${wav2letter-inference_SOURCE_DIR}
    ${wav2letter++_SOURCE_DIR}/src
)

This is the resulting cc command:

"cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823.1hdts60oj746ehis.rcgu.o" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823.2jd5ckk6rfm690qi.rcgu.o" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823.37247l7atvuwnzac.rcgu.o" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823.3hw7nz2p4p61apjt.rcgu.o" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823.4tahm12i4sfibyvi.rcgu.o" "-o" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823" "/workspaces/monorepo/asr/api/target/debug/deps/api-f93528ac957cf823.1112szmtp4pnji5a.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/workspaces/monorepo/asr/api/target/debug/deps" "-L" "../w2l/build" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/workspaces/monorepo/asr/api/target/debug/deps/libw2l-0cb89ddcc20a81cd.rlib" "-Wl,--start-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-b1b61f01951b016b.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-1a219005f9510085.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-de703a5f53bf135e.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-f8b5f83d4ba2b90f.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-c6dc8f69734bcae2.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-4546f4207f7495c9.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-2ac782148854dc56.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-cf897c39850c16b7.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-9d727da20068fc09.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-bd6ee87558a376f2.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-182e50caadead100.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-88083985464b6af5.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-83ed8731003cd087.rlib" "-Wl,--end-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d61dbae1fa88f6f5.rlib" "-Wl,-Bdynamic" "-lstdc++" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"

This may not solve all problems, but it seems to depend on fbgemm.

So fbgemm is the problem? How?

One of the problems seems to be that fbgemm is not linked. Linking it while likely not solve all problems as I already said though.

How can I check whether it's linked or not?

nm -C ../w2l/build/libw2l_api.a | grep "fbgemm::fbgemmAlignedFree(void"
                 U fbgemm::fbgemmAlignedFree(void*)
                 U fbgemm::fbgemmAlignedFree(void*)
                 U fbgemm::fbgemmAlignedFree(void*)

I thought this was enough.

There are loads of symbols like that in the libw2l_api. It's statically linked in that library via Cmake. Are you telling me that I need to link it again for Rust?

U means that the symbol is undefined. You will need to add a line similar to that used to link w2l_api for fbgemm.

1 Like

That seems to be it... so far. Didn't know about the U flag :flushed:. Thank you.

Can Cargo somehow grab all these additional static libraries automatically? I think I have dozens of them. Not very maintainable, if I have to specify the exact path for all of them.

I.e. println!("cargo:rustc-link-search=../w2l/build/3rdparty/wav2letter/inference/inference/common");

You can add some code to walk the build directory and emit the right link instructions.