Debugging a memory leak

1

I'm working on GRPC bindings to rust and I'm getting some sad crashes. The code is somewhat a mess of raw pointers and FFI and I'm not exactly sure how to debug double-free with rust.

(lldb) bt
* thread #2: tid = 0xd32006, 0x000000010006aac6 helloworld`je_arena_dalloc_bin_locked [inlined] je_bitmap_unset(bit=0) + 71 at bitmap.h:215, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x000000010006aac6 helloworld`je_arena_dalloc_bin_locked [inlined] je_bitmap_unset(bit=0) + 71 at bitmap.h:215 [opt]
    frame #1: 0x000000010006aa7f helloworld`je_arena_dalloc_bin_locked [inlined] arena_run_reg_dalloc(ptr=<unavailable>, run=&0x101802480) + 95 at arena.c:217 [opt]
    frame #2: 0x000000010006aa20 helloworld`je_arena_dalloc_bin_locked(arena=&0x1010395c0, chunk=&0x101800000, ptr=<unavailable>, bitselm=&0x0) + 208 at arena.c:1780 [opt]
    frame #3: 0x000000010007b131 helloworld`je_tcache_bin_flush_small(tbin=&0x10181c048, binind=1, rem=0, tcache=&0x10181c000) + 241 at tcache.c:120 [opt]
    frame #4: 0x000000010007bd00 helloworld`je_tcache_cleanup [inlined] tcache_destroy + 42 at tcache.c:328 [opt]
    frame #5: 0x000000010007bcd6 helloworld`je_tcache_cleanup(tsd=&0x10141c088) + 38 at tcache.c:385 [opt]
    frame #6: 0x000000010007d7af helloworld`je_tsd_cleanup(arg=&0x10141c088) + 159 at tsd.c:81 [opt]
    frame #7: 0x000000010007dbda helloworld`je_tsd_cleanup_wrapper(arg=&0x10141c080) + 26 at tsd.h:486 [opt]
    frame #8: 0x00007fff85f2a739 libsystem_pthread.dylib`_pthread_tsd_cleanup + 86
    frame #9: 0x00007fff85f2a460 libsystem_pthread.dylib`_pthread_exit + 117
    frame #10: 0x00007fff85f2b065 libsystem_pthread.dylib`_pthread_body + 142
    frame #11: 0x00007fff85f2afd7 libsystem_pthread.dylib`_pthread_start + 176
    frame #12: 0x00007fff85f283ed libsystem_pthread.dylib`thread_start + 13

As far as I know, jemalloc is used in rust (and GRPC C code uses usual malloc() calls). Any suggestions on where do I start? The usual memory debugging techniques wouldn't work as the allocator is statically linked into the binary itself.

2

Just to clarify: are you trying to have Rust free pointers allocated by C? Because that's basically guaranteed to not work.

1 Like
3

No, it's not the case. So far I'm actually leaking all the memory that is allocated on C side. Thing is that I have quite a bunch of unsafe code and something in there doesn't work as expected.

4

[quote="Gankra, post:2, topic:2959"]
have Rust free pointers allocated by C
[/quote]Could you clarify this further please? Does libc::free count as Rust freeing pointers allocated by C?

5

Actually traced it down to:

unsafe { Vec::from_raw_parts(out.as_mut_ptr(), ffi::grpc_byte_buffer_length(buffer) as usize, ffi::grpc_byte_buffer_length(buffer) as usize) }

then modified it to use Vec::set_len() and the double free is gone. The original question still stands though, out of general curiosity.

6

The docs make it pretty clear that the pointer should not have been allocated with system malloc.

7

out is actually another Vec, but what I should have done is mem::forget(out) (well, as a very weird hack to set_len() which I was looking for but didn't notice in the docs.

8

@gkoz this is a point of debate -- if you control all the C then you can be reasonably confident that this is fine, but in general those pointers can be allocated using whatever strategy, and this will only correctly handle the default C allocator.

9

@farcaller Is your grpc binding going to be public / open source?

10

It is definitely a plan. unfortunately I can't release half-baked implementation and getting rust to same level of functionality coverage as C++ takes time.

11

Sweet. Let me know if you want any help :smile:

12

@Gankro

This is a necro, but of course calling free from Rust will work under the same circumstances that calling free from C will - exactly when the pointer was allocated by a malloc, except on Windows when it must also have been allocated by the correct CRT.

13

Any progress on gRPC? I've been working with the ruby implementation recently and was contemplating looking at rust bindings.

14

I've been building large project at work based on gRPC and RocksDB. Being able to work on the core services with Rust instead of C++ would be pretty great.

15

There is progress, but nothing I can release yet, sorry.