Question on building a RUST shared library with a custom memory alllocator

I am building an LD_PRELOAD shared library that intercepts a few libc API like open and close.
I ran into the following traceback using the shared library with a Make program, where the make program hung.
Because one of the processes it forked exit successfully. This seems to happen when the make program is calling malloc. At this point it gets a sginal, sigchild exit signal handler, from a process it forks, and this handler calls "close()" which is also an itnercepted call. my intercepted my_close() implementation does a malloc() which I suspect is hanging.

This is the traceback i got when i gdb attached to the hung make process.

(gdb) bt
#0  0x00007f73f9fb85bc in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f73f9f34b35 in _L_lock_17166 () from /lib64/libc.so.6
#2  0x00007f73f9f31b73 in malloc () from /lib64/libc.so.6
#3  0x00007f73fa460d28 in alloc::alloc::alloc () at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/alloc.rs:80
#4  <alloc::alloc::Global as core::alloc::AllocRef>::alloc ()
    at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/alloc.rs:173
#5  alloc::raw_vec::RawVec<T,A>::allocate_in () at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/raw_vec.rs:186
#6  alloc::raw_vec::RawVec<T,A>::with_capacity_in () at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/raw_vec.rs:161
#7  alloc::raw_vec::RawVec<T>::with_capacity () at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/raw_vec.rs:92
#8  alloc::vec::Vec<T>::with_capacity () at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/vec.rs:362
#9  <&[u8] as std::ffi::c_str::CString::new::SpecIntoVec>::into_vec () at library/std/src/ffi/c_str.rs:383
#10 std::ffi::c_str::CString::new () at library/std/src/ffi/c_str.rs:396
#11 std::sys::unix::os::getenv () at library/std/src/sys/unix/os.rs:516
#12 std::env::_var_os [_ZN3std3env7_var_os1...] () at library/std/src/env.rs:250
#13 0x00007f73fa460bc6 in std::env::var_os () at library/std/src/env.rs:246
#14 std::env::_var [_ZN3std3env4_var17h7...] () at library/std/src/env.rs:216
#15 0x00007f73fa42981c in std::env::var [_ZN3std3env3var17hdb...] (key=...)
    at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/env.rs:212
#16 0x00007f73fa2d017b in wisktrack::my_close [_ZN9wisktrack8my_clo...] (fd=8) at src/lib.rs:138
#17 0x00007f73fa2e9e70 in wisktrack::close::{{closure}} [_ZN9wisktrack5close2...] ()
    at /users/sarvi/.cargo/git/checkouts/redhook-16e0a55232cdd48a/55ac3da/src/ld_preload.rs:67
#18 0x00007f73fa2f26e7 in std::panicking::try::do_call [_ZN3std9panicking3tr...] (data=0x7ffc4e41ce28 "\264\316AN\374\177")
    at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
#19 0x00007f73fa2f3b2d in __rust_try () from /ws/sarvi-sjc/cel7root/lib64/libwisktrack.so
#20 0x00007f73fa2f0627 in std::panicking::try [_ZN3std9panicking3tr...] (f=...)
---Type <return> to continue, or q <return> to quit--- 
    at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
#21 0x00007f73fa2b11b6 in std::panic::catch_unwind [_ZN3std5panic12catch...] (f=...)
    at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
#22 0x00007f73fa2d08cf in wisktrack::close [close] (fd=8)
    at /users/sarvi/.cargo/git/checkouts/redhook-16e0a55232cdd48a/55ac3da/src/ld_preload.rs:67
#23 0x000000000040d88a in child_handler (sig=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/job.c:423
#24 <signal handler called>
#25 0x00007f73f9f2e50d in _int_malloc () from /lib64/libc.so.6
#26 0x00007f73f9f31b1c in malloc () from /lib64/libc.so.6
#27 0x0000000000411779 in xmalloc (size=size@entry=48) at /auto/swtools/kellythw/src/make-3.81/misc.c:342
#28 0x000000000041aacf in define_variable_in_set (name=name@entry=0x41d67a "|", length=length@entry=1, value=0x6e69ba0 "", 
    origin=origin@entry=o_automatic, recursive=recursive@entry=0, set=0xbd4b380, flocp=flocp@entry=0x0)
    at /auto/swtools/kellythw/src/make-3.81/variable.c:209
#29 0x00000000004052c6 in set_file_variables (file=file@entry=0x3172b70) at /auto/swtools/kellythw/src/make-3.81/commands.c:259
#30 0x000000000040581a in execute_file_commands (file=file@entry=0x3172b70) at /auto/swtools/kellythw/src/make-3.81/commands.c:391
#31 0x000000000041889a in remake_file (file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:1123
#32 update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:761
#33 update_file (file=file@entry=0x3172b70, depth=depth@entry=26) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#34 0x0000000000417611 in check_dep (file=0x3172b70, depth=26, depth@entry=25, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41d82c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#35 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#36 update_file (file=file@entry=0x316a8e0, depth=depth@entry=24) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#37 0x0000000000417611 in check_dep (file=0x316a8e0, depth=24, depth@entry=23, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41d92c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#38 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
---Type <return> to continue, or q <return> to quit---
#39 update_file (file=file@entry=0x316f6e0, depth=depth@entry=22) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#40 0x0000000000417611 in check_dep (file=0x316f6e0, depth=22, depth@entry=21, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41da2c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#41 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#42 update_file (file=file@entry=0x165f740, depth=depth@entry=20) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#43 0x0000000000417611 in check_dep (file=0x165f740, depth=20, depth@entry=19, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41db2c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#44 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#45 update_file (file=file@entry=0x1667780, depth=depth@entry=18) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#46 0x0000000000417611 in check_dep (file=0x1667780, depth=18, depth@entry=17, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41dc2c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#47 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#48 update_file (file=file@entry=0x1669dc0, depth=depth@entry=16) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#49 0x0000000000417611 in check_dep (file=0x1669dc0, depth=16, depth@entry=15, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41dd2c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#50 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#51 update_file (file=file@entry=0x165ba20, depth=depth@entry=14) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#52 0x0000000000417611 in check_dep (file=0x165ba20, depth=14, depth@entry=13, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41de2c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#53 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#54 update_file (file=file@entry=0x1657520, depth=depth@entry=12) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#55 0x0000000000417611 in check_dep (file=0x1657520, depth=12, depth@entry=11, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41df2c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#56 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#57 update_file (file=file@entry=0x165bb70, depth=depth@entry=10) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
---Type <return> to continue, or q <return> to quit---
#58 0x0000000000417611 in check_dep (file=0x165bb70, depth=10, depth@entry=9, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41e02c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#59 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#60 update_file (file=file@entry=0x1669e60, depth=depth@entry=8) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#61 0x0000000000417611 in check_dep (file=0x1669e60, depth=8, depth@entry=7, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41e12c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#62 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#63 update_file (file=file@entry=0x1649b30, depth=depth@entry=6) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#64 0x0000000000417611 in check_dep (file=0x1649b30, depth=6, depth@entry=5, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41e22c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#65 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#66 update_file (file=file@entry=0x16439b0, depth=depth@entry=4) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#67 0x0000000000417611 in check_dep (file=0x16439b0, depth=4, depth@entry=3, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41e32c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#68 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#69 update_file (file=file@entry=0x1649cf0, depth=depth@entry=2) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#70 0x0000000000417611 in check_dep (file=0x1649cf0, depth=2, depth@entry=1, this_mtime=this_mtime@entry=1, 
    must_make_ptr=must_make_ptr@entry=0x7ffc4e41e42c) at /auto/swtools/kellythw/src/make-3.81/remake.c:947
#71 0x0000000000417d2e in update_file_1 (depth=<optimized out>, file=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:508
#72 update_file (file=file@entry=0x161c510, depth=<optimized out>) at /auto/swtools/kellythw/src/make-3.81/remake.c:307
#73 0x0000000000418ce6 in update_goal_chain (goals=0x94bedb0) at /auto/swtools/kellythw/src/make-3.81/remake.c:154
#74 0x0000000000403879 in main (argc=<optimized out>, argv=0x7ffc4e41fff8, envp=<optimized out>)
    at /auto/swtools/kellythw/src/make-3.81/main.c:2198
(gdb) 
(gdb) 

bash-4.4$ 

Question : If used a custom rust allocator in my shared library, say for example static_alloc - Rust and define it as such follows in my library, will the static allocator apply only to calls for heap allocation from within my shared library?
I dont want it to affect the rest of the heeap allocation and calls to libc:malloc from the rest of the program or libc itself. I want it to affect only heap allocation from within the shared library.
How do I do that.

use static_alloc::Bump;

#[global_allocator]
static A: Bump<[u8; 1 << 16]> = Bump::uninit();

Question 2: Is the above good choice? is there a recommendation for a alternative allocator to libc. There are no other limitations.

Yes

If your main binary is not written in Rust, yes. But I'm not sure same applies if both your shared library and the binary are written in Rust.

I cant guarantee what language the main program will be written in.
So I am looking for a way to make the allocator private to the current shared library ad not affect anyone the main program or any of the shared librarries in any way.

Also is there a better alternative static_alloc. My problem space is not necessary space limited.
And would prefer not to be space limited. I am only trying to keep the memory management of my shared library not dependent on the global allocator like libc or jemalloc statically bound to the rustc as in that case. to avoid the above such problems

Just statically link in any allocator and make sure to link with -Bsymbolic. The allocator should also call syscalls directly, not through libc.

Is there an allocator that fits that definition? calling syscalls directly? I suspect static_alloc might fit the bill.
I know jemalloc does do libc calls like readlink() and so does have some libc dependencies. it does use libc malloc.

How do I do static linking of crates into a .so ?
I am reading
https://doc.rust-lang.org/reference/linkage.html
Ad it talks about creating a staic library or a cdylib/.so, but not about statically linking specific crates like a custom allocator into .so shared library?

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.