Debugging a memory leak


#1

I’m working on GRPC bindings to rust and I’m getting some sad crashes. The code is somewhat a mess of raw pointers and FFI and I’m not exactly sure how to debug double-free with rust.

(lldb) bt
* thread #2: tid = 0xd32006, 0x000000010006aac6 helloworld`je_arena_dalloc_bin_locked [inlined] je_bitmap_unset(bit=0) + 71 at bitmap.h:215, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x000000010006aac6 helloworld`je_arena_dalloc_bin_locked [inlined] je_bitmap_unset(bit=0) + 71 at bitmap.h:215 [opt]
    frame #1: 0x000000010006aa7f helloworld`je_arena_dalloc_bin_locked [inlined] arena_run_reg_dalloc(ptr=<unavailable>, run=&0x101802480) + 95 at arena.c:217 [opt]
    frame #2: 0x000000010006aa20 helloworld`je_arena_dalloc_bin_locked(arena=&0x1010395c0, chunk=&0x101800000, ptr=<unavailable>, bitselm=&0x0) + 208 at arena.c:1780 [opt]
    frame #3: 0x000000010007b131 helloworld`je_tcache_bin_flush_small(tbin=&0x10181c048, binind=1, rem=0, tcache=&0x10181c000) + 241 at tcache.c:120 [opt]
    frame #4: 0x000000010007bd00 helloworld`je_tcache_cleanup [inlined] tcache_destroy + 42 at tcache.c:328 [opt]
    frame #5: 0x000000010007bcd6 helloworld`je_tcache_cleanup(tsd=&0x10141c088) + 38 at tcache.c:385 [opt]
    frame #6: 0x000000010007d7af helloworld`je_tsd_cleanup(arg=&0x10141c088) + 159 at tsd.c:81 [opt]
    frame #7: 0x000000010007dbda helloworld`je_tsd_cleanup_wrapper(arg=&0x10141c080) + 26 at tsd.h:486 [opt]
    frame #8: 0x00007fff85f2a739 libsystem_pthread.dylib`_pthread_tsd_cleanup + 86
    frame #9: 0x00007fff85f2a460 libsystem_pthread.dylib`_pthread_exit + 117
    frame #10: 0x00007fff85f2b065 libsystem_pthread.dylib`_pthread_body + 142
    frame #11: 0x00007fff85f2afd7 libsystem_pthread.dylib`_pthread_start + 176
    frame #12: 0x00007fff85f283ed libsystem_pthread.dylib`thread_start + 13

As far as I know, jemalloc is used in rust (and GRPC C code uses usual malloc() calls). Any suggestions on where do I start? The usual memory debugging techniques wouldn’t work as the allocator is statically linked into the binary itself.


#2

Just to clarify: are you trying to have Rust free pointers allocated by C? Because that’s basically guaranteed to not work.


#3

No, it’s not the case. So far I’m actually leaking all the memory that is allocated on C side. Thing is that I have quite a bunch of unsafe code and something in there doesn’t work as expected.


#4

[quote=“Gankro, post:2, topic:2959”]
have Rust free pointers allocated by C
[/quote]Could you clarify this further please? Does libc::free count as Rust freeing pointers allocated by C?


#5

Actually traced it down to:

unsafe { Vec::from_raw_parts(out.as_mut_ptr(), ffi::grpc_byte_buffer_length(buffer) as usize, ffi::grpc_byte_buffer_length(buffer) as usize) }

then modified it to use Vec::set_len() and the double free is gone. The original question still stands though, out of general curiosity.


#6

The docs make it pretty clear that the pointer should not have been allocated with system malloc.


#7

out is actually another Vec, but what I should have done is mem::forget(out) (well, as a very weird hack to set_len() which I was looking for but didn’t notice in the docs.


#8

@gkoz this is a point of debate – if you control all the C then you can be reasonably confident that this is fine, but in general those pointers can be allocated using whatever strategy, and this will only correctly handle the default C allocator.


#9

@farcaller Is your grpc binding going to be public / open source?


#10

It is definitely a plan. unfortunately I can’t release half-baked implementation and getting rust to same level of functionality coverage as C++ takes time.


#11

Sweet. Let me know if you want any help :smile:


#12

@Gankro

This is a necro, but of course calling free from Rust will work under the same circumstances that calling free from C will - exactly when the pointer was allocated by a malloc, except on Windows when it must also have been allocated by the correct CRT.


#13

Any progress on gRPC? I’ve been working with the ruby implementation recently and was contemplating looking at rust bindings.


#14

I’ve been building large project at work based on gRPC and RocksDB. Being able to work on the core services with Rust instead of C++ would be pretty great.


#15

There is progress, but nothing I can release yet, sorry.