SegFault in Rust when dropping Option<Weak<()>> that is set to None

help
1

I'm not quite sure how to debug this as its happening in a large application, but only some of the devices we have deployed to.

In our code we have an iterator that produces a "BoundingBox" and then we call collect on that BoundingBox.

so we have

filter_map(|| {
    // ...
    let bbox = BoundingBox {
     id: 0,
     rect,
     score: confidence as f64,
     class: mapping.from_detector_class(id),
     raw_class: id as usize,
     ..Default::default()
  };

  Some(bbox)
})
.collect::<Vec<_>>()

BoundingBox looks like this:


#[derive(Debug, Clone)]
pub struct BoundingBox {
    pub id: u64,
    pub rect: Rect,
    pub score: f64,
    pub class: DetectionClass,
    pub raw_class: usize,
    pub handle: Option<std::sync::Weak<()>>,
}

Where we have a Default impl where handle is None. We noticed that some devices (Rust 1.87.0 and Rust 1.92.0, on Ubuntu Linux x86) were segfaulting so we deployed a debug build and got this trace:

#0  core::sync::atomic::atomic_sub (dst=0x208) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/sync/atomic.rs:4002
#1  core::sync::atomic::AtomicUsize::fetch_sub (self=0x208) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/sync/atomic.rs:3195
#2  <alloc::sync::Weak<T,A> as core::ops::drop::Drop>::drop (self=0x7f85ee5630d8) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/alloc/src/sync.rs:3301
#3  0x0000558f04b2a2eb in core::ptr::drop_in_place<alloc::sync::Weak<()>> () at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/ptr/mod.rs:805
#4  0x0000558f02c66026 in core::ptr::drop_in_place<core::option::Option<alloc::sync::Weak<()>>> () at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/ptr/mod.rs:805
#5  0x0000558f02c62608 in core::ptr::drop_in_place<database::types::bounding_box::BoundingBox> () at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/ptr/mod.rs:805
#6  0x0000558f01cc5719 in core::iter::traits::iterator::Iterator::find::check::{{closure}} (x=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/traits/iterator.rs:2886
#7  0x0000558f01cc5542 in core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut (self=0x7f85ee563188, args=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/ops/function.rs:298
#8  0x0000558f01cce10a in <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold (self=0x7f85ee567020, accum=(), f=0x7f85ee564e38)
    at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/alloc/src/vec/into_iter.rs:351
#9  0x0000558f01cd5537 in <core::iter::adapters::flatten::FlattenCompat<I,U> as core::iter::traits::iterator::Iterator>::try_fold::flatten::{{closure}} (acc=(), iter=0x7f85ee567020)
    at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/flatten.rs:563
#10 0x0000558f01cd9074 in core::iter::adapters::flatten::FlattenCompat<I,U>::iter_try_fold::flatten::{{closure}} (acc=(), iter=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/flatten.rs:420
#11 0x0000558f01cde2df in core::iter::adapters::map::map_try_fold::{{closure}} (acc=(), elt=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/map.rs:95
#12 0x0000558f01ceda19 in core::iter::traits::iterator::Iterator::try_fold (self=0x7f85ee566fa0, init=(), f=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/traits/iterator.rs:2427
#13 0x0000558f01cddb50 in <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold (self=0x7f85ee566fa0, init=(), g=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/map.rs:121
#14 0x0000558f01cdd3de in <core::iter::adapters::fuse::Fuse<I> as core::iter::adapters::fuse::FuseImpl<I>>::try_fold (self=0x7f85ee566fa0, acc=(), fold=...)
    at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/fuse.rs:317
#15 0x0000558f01cd881d in <core::iter::adapters::fuse::Fuse<I> as core::iter::traits::iterator::Iterator>::try_fold (self=0x7f85ee566fa0, acc=(), fold=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/fuse.rs:89
#16 core::iter::adapters::flatten::FlattenCompat<I,U>::iter_try_fold (self=0x7f85ee566fa0, acc=(), fold=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/flatten.rs:428
#17 0x0000558f01cd547c in <core::iter::adapters::flatten::FlattenCompat<I,U> as core::iter::traits::iterator::Iterator>::try_fold (self=0x7f85ee566fa0, init=(), fold=...)
    at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/flatten.rs:566
#18 0x0000558f01cd52bc in <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::try_fold (self=0x7f85ee566fa0, init=(), fold=...)
    at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/flatten.rs:79
#19 0x0000558f01cd7c0c in core::iter::traits::iterator::Iterator::find (self=0x7f85ee566fa0, predicate=0x7f85ee567060) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/traits/iterator.rs:2889
#20 0x0000558f01cf4991 in <core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::next (self=0x7f85ee566fa0) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/iter/adapters/filter.rs:98
#21 0x0000558f01c93ad0 in <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iterPython Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x8: 
 (iterator=#22 0x0000558f01c9535e in <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iterPython Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x8: 

    (iterator=#23 0x0000558f01c9530a in <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter (iter=...) at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/alloc/src/vec/mod.rs:3689
#24 0x0000558f01cf4b4e in core::iter::traits::iterator::Iterator::collectPython Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x8: 
 (self=#25 0x0000558f01cc3694 in trt::yolo::parse
    (preds=0x7f884e76b380, output_shape=..., classes=44, mapping=0x7f863058ef10, confidence=0x7f863058f028, iou_threshold=0x7f863058f0f8, scaling=0x7f863058f248) at trt/src/yolo.rs:90

It seems that fetch_sub is being called on Weak that was never initialized? I'm not sure when this bug was introduced as our application runs on thousands of devices that are mostly homogenous. However it's not clear to me why drop is called at all for Weak.

I've tried to repro in an example but naturally an isolated example doesn't work; so I'm mainly looking for pointers here on what could be the issue.

2

shot in the dark: is there some instance of Arc::from_raw() or Weak::from_raw() somewhere in the code base?

3

(It doesn't help with your problem, but) Rc<T>s have both a weak count and a strong count,[1] and dropping Weak<T>s needs to decrement the weak count, or the Rc<T> won't know when the counters themselves can be freed.[2]

  1. the T is dropped when the strong count reaches 0 ↩︎

  2. And every Weak<T> needs to be able to access the strong count when attempting to upgrade, for example. ↩︎

4

No, although I did check.

The Arc here is just a makeshift liveness check (so the type is really ()). The system maintains a tracker (that holds an Arc), and if an object is being tracked, its given a Weak<()>. Other modules can test if a BoundingBox is "live" by trying to upgrade the handle.

That said, all of that comes after this point. At this point all I'm doing is initializing it with None, and then performing a collect.

If this isn't an issue with the Rust compiler (which it probably isn't), I think I probably have a heap corruption somewhere else.

Actually looking a bit closer, the application always crashes in this collect, but at different points. I think this might just be a heap corruption.

5

It's more so I don't know why it's being called because handle is None.

let handle: Option<Weak<()>> = None;
drop(handle) // Weak<()>::drop() surely shouldn't be called?
6

Right -- if nothing (legitimately) sets handle to Some, the drop for Weak<()> wouldn't be called (there's nothing to call it on).

In which case it does sound like some sort of UB before the attempted drop (but it's hard to say where).

2 Likes
7

Yes I'm pretty sure this is UB somewhere and the above code isn't enough to diagnose it.

Thanks everyone for the quick responses.

2 Likes