Make multithreaded/async app more readable in htop/gdb?

arifd · November 12, 2022, 12:40pm

Hello,

So my poor application that I have been working very hard on, sadly over time starts to emit "hanging" like behaviour. It is a complex beast involving tokio tasks, some of which spawn_blocking rayon par_iter() tasks.

I have a feeling it is somewhere in rayon that I am failing because looking in the tokio-console, i can see all the tasks that call rayon get stuck and none of them make any progress.

The problem is, rayon i believe is a black box. I would really love to inspect what is happening in rayon, but I don't know how.

If I look in htop, I can see all the threads my app spawns, but they are unnamed. would this help thread::Builder::new().name("foo".into())? It'd be great if i could identify them somehow.

I'm about to run rust-gdb thread apply all where so that I can inspect all 80 threads, again, perhaps if named i might be able to discard more data quicker.

If you have any advice? I'm kind of new to such deep debugging.

Thanks in advance!

Michael-F-Bryan · November 12, 2022, 12:45pm

It's not part of gdb, but the Tokio Console project could help you figure out whether the cause is async tasks being deadlocked or because you are accidentally locking up Tokio's threads with long-running sync tasks or whatever.

arifd · November 12, 2022, 12:51pm

looking through the tokio console

the first Semaphore has 0 permits available, but this is Ok because this is the semaphore that controls the number of workers... they are all busy.... but hanging

The second Semaphore (with id 37) is of <unknown location> I don't know whose semaphore this is, I don't believe it's in my code, and it has 0 permits available, this kind of worries me.

And sometimes it just unhangs for a ltittle bit, then hangs again. Unfortunately i wasn't able to look at the console in time to see if Semaphore 37 suddenly had permits available or something (but it's back to 0 again)

Michael-F-Bryan · November 12, 2022, 12:57pm

Interesting.

Looking through tokio's source code, it seems like the Semaphore (from the tokio::sync::semaphore module) struct is implemented in terms of a lower-level semaphore in the tokio::sync::batch_semaphore module, so maybe we're just seeing the same thing twice?

github.com

tokio-rs/tokio/blob/b7812c85ca2d051d47cec023b880cbf8cdcbc313/tokio/src/sync/semaphore.rs#L79-L84


      
          pub struct Semaphore {
              /// The low level semaphore
              ll_sem: ll::Semaphore,
              #[cfg(all(tokio_unstable, feature = "tracing"))]
              resource_span: tracing::Span,
          }

arifd · November 12, 2022, 12:59pm

btw, when it hangs, if i quit and restart the console, the console doesn't want to connect either.... What could this be!?!?!

this means tokio is stuck, right? and not necessarily rayon?

btw, all my app's threads take up 0% CPU when it is exhibiting this "hanging" behaviour

CRAAP ignore the above, gdb had interuppted the program!!!

Wait, it did it again, why is gdb suddenly going into this prompt?

Okay, detaching gdb when I'm not using it... I cant have it create more confusion for me!

you are accidentally locking up Tokio's threads with long-running sync tasks or whatever

Could you explain what you meant by this?

Michael-F-Bryan · November 12, 2022, 1:15pm

Not necessarily. It might just be a port.

From their docs,

by default, this will attempt to connect to an instrumented application running on localhost on port 6669. if the application is running somewhere else, or is serving the console endpoint on a different port, a target address can be passed as an argument to the console (either as an <IP>:<PORT> or <DNS_NAME>:<PORT> )

If you can't reconnect, you might want to check whether the restarted app is serving on a different port. I'd normally use nmap localhost for that, but there are a bunch of other tools for seeing which ports are open on your machine.

Rust's Futures and async code work around the idea of polling - in order to make any progress on an operation you (well, technically tokio's executor) will need to call some sort of poll() method which will do some work and let you know whether the future has completed or will need to be polled again later on.

This sort of cooperative multi-tasking is great for performance, but has one critical weakness - calls to poll() can't be pre-empted, so if your async function blocks for a while, the Tokio runtime will have no choice but to wait until that slow poll() call completes before it can make progress on any other tasks.

Tokio has a runner which uses a thread pool to poll futures. That lets us both do more stuff and helps mitigate this risk (i.e. if one thread is stuck on a long poll() call, futures can still be polled on other threads), but if we have lots of blocking calls it's possible for all of the threads in Tokio's thread pool to be stuck at the same time. If one task relies on a mutex held by another task, this can result in deadlocks.

@alice has an article which does a much better job of explaining this than I could. I'd recommend giving it a read.

cuviper · November 12, 2022, 2:39pm

Rayon does have a builder that can set thread_name, if that helps. You can use that and then build_global to initialize the global pool, or just build a private instance and use that.

arifd · November 12, 2022, 2:44pm

The thing is, all my tasks that spawn_blocking -> par_iter() suddenly get stuck at the point this "hanging" occurs. and the tasks don't share any sync-primitives between them or anything.

Like I could understand if one of the closures sent to rayon got stuck, but they all (appear to) get stuck at the same time?... the CPU just suddenly drops to 0% and then nothing progresses

Do you have any ideas where to begin to look?

I'll set a numbered thread-name, see if that offers any insight.

but I can't seem to see anything in htop that suggests rayon anyway, all thread names are 'tokio-runtime-worker'...

Thanks!

cuviper · November 12, 2022, 2:53pm

What do the backtraces of those stuck threads look like? That should give a clue what they're stuck on.

arifd · November 12, 2022, 3:04pm

Thank you for asking! I'm not really sure how to make sense of the backtraces, but in gdb they all look like this: ( using rust-gdb thread apply all where. I stop the backtrace at the first function of my code, and only pasted a few relevant threads)

Thread 53 (Thread 0x7f1c90bc5700 (LWP 964556) "tokio-runtime-w"):
#0  0x00007f1ea9c122e9 in syscall () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555eb6b5c1a5 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x0000555eb6b60bb7 in std::sys::unix::locks::futex_condvar::Condvar::wait_optional_timeout () at library/std/src/sys/unix/locks/futex_condvar.rs:51
#3  std::sys::unix::locks::futex_condvar::Condvar::wait () at library/std/src/sys/unix/locks/futex_condvar.rs:35
#4  0x0000555eb5c384c0 in std::sys_common::condvar::Condvar::wait (self=0x7f1c90bc54dc, mutex=0x7f1c90bc54d4) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/condvar.rs:42
#5  0x0000555eb5c670e6 in std::sync::condvar::Condvar::wait<bool> (self=0x7f1c90bc54dc, guard=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sync/condvar.rs:191
#6  0x0000555eb5c5f847 in rayon_core::latch::LockLatch::wait_and_reset (self=0x7f1c90bc54d4) at src/latch.rs:242
#7  0x0000555eb56220b0 in rayon_core::registry::{impl#6}::in_worker_cold::{closure#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (l=0x7f1c90bc54d4) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:473
#8  0x0000555eb5627665 in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::try_with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x555eb6bbafc0, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445
#9  0x0000555eb5626e2f in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x555eb6bbafc0, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421
#10 0x0000555eb5620f24 in rayon_core::registry::Registry::in_worker_cold<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x7f1e6--Type <RET> for more, q to quit, c to continue without paging--
004e480, op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:461
#11 0x0000555eb5626039 in rayon_core::registry::in_worker<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:879
#12 0x0000555eb561875a in rayon_core::join::join_context<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:132
#13 0x0000555eb5620db1 in rayon_core::join::join<cabinet_document::parse::_parse_document::{closure_env#1}, cabinet_document::parse::_parse_document::{closure_env#2}, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:105
#14 0x0000555eb55e9fb3 in cabinet_document::parse::_parse_document (pdf=...) at cabinet_document/src/parse.rs:211

arifd · November 12, 2022, 3:05pm

Thread 52 (Thread 0x7f1da3bfd700 (LWP 963539) "tokio-runtime-w"):
#0  0x00007f1ea9c122e9 in syscall () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555eb6b5c1a5 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x0000555eb6b60bb7 in std::sys::unix::locks::futex_condvar::Condvar::wait_optional_timeout () at library/std/src/sys/unix/locks/futex_condvar.rs:51
#3  std::sys::unix::locks::futex_condvar::Condvar::wait () at library/std/src/sys/unix/locks/futex_condvar.rs:35
#4  0x0000555eb5c384c0 in std::sys_common::condvar::Condvar::wait (self=0x7f1da3bfd4dc, mutex=0x7f1da3bfd4d4) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/condvar.rs:42
#5  0x0000555eb5c670e6 in std::sync::condvar::Condvar::wait<bool> (self=0x7f1da3bfd4dc, guard=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sync/condvar.rs:191
#6  0x0000555eb5c5f847 in rayon_core::latch::LockLatch::wait_and_reset (self=0x7f1da3bfd4d4) at src/latch.rs:242
#7  0x0000555eb53fdfdc in rayon_core::registry::{impl#6}::in_worker_cold::{closure#0}<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (l=0x7f1da3bfd4d4) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:473
#8  0x0000555eb54016b1 in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::try_with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_inde--Type <RET> for more, q to quit, c to continue without paging--
x::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (self=0x555eb6bafb80, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445
#9  0x0000555eb540126f in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (self=0x555eb6bafb80, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421
#10 0x0000555eb53fd436 in rayon_core::registry::Registry::in_worker_cold<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (self=0x7f1e6004e480, op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:461
#11 0x0000555eb5400869 in rayon_core::registry::in_worker<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:879
#12 0x0000555eb53b6518 in rayon_core::join::join_context<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:132
#13 0x0000555eb53b12e5 in rayon::iter::plumbing::bridge_producer_consumer::helper<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (len=3, migrated=false, splitter=..., producer=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:416
#14 0x0000555eb53aeda9 in rayon::iter::plumbing::bridge_producer_consumer<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (len=3, producer=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:397
#15 0x0000555eb53aead2 in rayon::iter::plumbing::bridge::{impl#0}::callback<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_types::document::c_region::CRegion, rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>> (self=..., producer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:373
#16 0x0000555eb539e4ba in rayon::vec::{impl#7}::with_producer<cabinet_types::document::c_region::CRegion, rayon::iter::plumbing::bridge::Callback<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>> (self=..., callback=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/vec.rs:147
--Type <RET> for more, q to quit, c to continue without paging--
#17 0x0000555eb539fa7c in rayon::vec::{impl#4}::with_producer<cabinet_types::document::c_region::CRegion, rayon::iter::plumbing::bridge::Callback<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>> (self=..., callback=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/vec.rs:83
#18 0x0000555eb53b55fc in rayon::iter::plumbing::bridge<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (par_iter=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:357
#19 0x0000555eb539e00f in rayon::vec::{impl#3}::drive_unindexed<cabinet_types::document::c_region::CRegion, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/vec.rs:58
#20 0x0000555eb53975fe in rayon::iter::flat_map::{impl#2}::drive_unindexed<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}, alloc::vec::Vec<(cabinet_index::schemas::c_spans::CSpanSliceKind, alloc::vec::Vec<cabinet_types::document::c_span::CSpan, alloc::alloc::Global>, alloc::sync::Arc<alloc::vec::Vec<alloc::string::String, alloc::alloc::Global>>), alloc::alloc::Global>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/flat_map.rs:44
#21 0x0000555eb5397803 in rayon::iter::flat_map::{impl#2}::drive_unindexed<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}, alloc::vec::Vec<(usize, cabinet_index::schemas::c_spans::CSpanSliceKind, cabinet_types::document::c_span::CSpan, alloc::sync::Arc<alloc::vec::Vec<alloc::string::String, alloc::alloc::Global>>), alloc::alloc::Global>, rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/flat_map.rs:44
#22 0x0000555eb5393e47 in rayon::iter::map::{impl#2}::drive_unindexed<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}, (cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), rayon::iter::extend::ListVecConsumer> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/map.rs:49
#23 0x0000555eb53872eb in rayon::iter::extend::{impl#29}::par_extend<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (self=0x7f1da3bfba38, par_iter=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/extend.rs:576
#24 0x0000555eb53dd4cf in rayon::iter::from_par_iter::collect_extended<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>, rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (par_iter=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/from_par_iter.rs:17
#25 0x0000555eb538718d in rayon::iter::from_par_iter::{impl#0}::from_par_iter<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (par_iter=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/from_par_iter.rs:30
#26 0x0000555eb5393bfd in rayon::iter::ParallelIterator::collect<rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>> (self=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/mod.rs:2046
#27 0x0000555eb53ec422 in cabinet_index::schemas::c_spans::tps_to_cspans (id=..., hash=..., dtps=0x7f1d5404f520) at cabinet_index/src/schemas/c_spans.rs:266

arifd · November 12, 2022, 3:05pm

Thread 51 (Thread 0x7f1c96bf5700 (LWP 960112) "tokio-runtime-w"):
#0  0x00007f1ea9c122e9 in syscall () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555eb6b5c1a5 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x0000555eb6b60bb7 in std::sys::unix::locks::futex_condvar::Condvar::wait_optional_timeout () at library/std/src/sys/unix/locks/futex_condvar.rs:51
#3  std::sys::unix::locks::futex_condvar::Condvar::wait () at library/std/src/sys/unix/locks/futex_condvar.rs:35
#4  0x0000555eb5c384c0 in std::sys_common::condvar::Condvar::wait (self=0x7f1c96bf54dc, mutex=0x7f1c96bf54d4) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/condvar.rs:42
#5  0x0000555eb5c670e6 in std::sync::condvar::Condvar::wait<bool> (self=0x7f1c96bf54dc, guard=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sync/condvar.rs:191
#6  0x0000555eb5c5f847 in rayon_core::latch::LockLatch::wait_and_reset (self=0x7f1c96bf54d4) at src/latch.rs:242
#7  0x0000555eb53fdfdc in rayon_core::registry::{impl#6}::in_worker_cold::{closure#0}<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (l=0x7f1c96bf54d4) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:473
#8  0x0000555eb54016b1 in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::try_with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (self=0x555eb6bafb80, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445
#9  0x0000555eb540126f in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (self=0x555eb6bafb80, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421
#10 0x0000555eb53fd436 in rayon_core::registry::Registry::in_worker_cold<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (self=0x7f1e6004e480, op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:461
#11 0x0000555eb5400869 in rayon_core::registry::in_worker<rayon_core::join::join_context::{closure_env#0}<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_t--Type <RET> for more, q to quit, c to continue without paging--
o_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>>, (alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>)> (op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:879
#12 0x0000555eb53b6518 in rayon_core::join::join_context<rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#0}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, rayon::iter::plumbing::bridge_producer_consumer::helper::{closure_env#1}<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>, alloc::collections::linked_list::LinkedList<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:132
#13 0x0000555eb53b12e5 in rayon::iter::plumbing::bridge_producer_consumer::helper<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (len=3, migrated=false, splitter=..., producer=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:416
#14 0x0000555eb53aeda9 in rayon::iter::plumbing::bridge_producer_consumer<rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (len=3, producer=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:397
#15 0x0000555eb53aead2 in rayon::iter::plumbing::bridge::{impl#0}::callback<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_types::document::c_region::CRegion, rayon::vec::DrainProducer<cabinet_types::document::c_region::CRegion>> (self=..., producer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:373
#16 0x0000555eb539e4ba in rayon::vec::{impl#7}::with_producer<cabinet_types::document::c_region::CRegion, rayon::iter::plumbing::bridge::Callback<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>> (self=..., callback=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/vec.rs:147
#17 0x0000555eb539fa7c in rayon::vec::{impl#4}::with_producer<cabinet_types::document::c_region::CRegion, rayon::iter::plumbing::bridge::Callback<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>>> (self=..., callback=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/vec.rs:83
#18 0x0000555eb53b55fc in rayon::iter::plumbing::bridge<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (par_iter=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/plumbing/mod.rs:357
#19 0x0000555eb539e00f in rayon::vec::{impl#3}::drive_unindexed<cabinet_types::document::c_region::CRegion, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/vec.rs:58
#20 0x0000555eb53975fe in rayon::iter::flat_map::{impl#2}::drive_unindexed<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}, alloc::vec::Vec<(cabinet_index::schemas::c_spans::CSpanSliceKind, alloc::vec::Vec<cabinet_types::document::c_span::CSpan, alloc::alloc::Global>, alloc::sync::Arc<alloc::vec::Vec<alloc::string::String, alloc::alloc::Global>>), alloc::alloc::Global>, rayon::iter::flat_map::FlatMapConsumer<rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/flat_map.rs:44
#21 0x0000555eb5397803 in rayon::iter::flat_map::{impl#2}::drive_unindexed<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}, alloc::vec::Vec<(usize, cabinet_index::schemas::c_spans::CSpanSliceKind, cabinet_types::document::c_span::CSpan, alloc::sync::Arc<alloc::vec::Vec<alloc::string::String, alloc::alloc::Global>>), alloc::alloc::Global>, rayon::iter::map::MapConsumer<rayon::iter::extend::ListVecConsumer, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/flat_map.rs:44
#22 0x0000555eb5393e47 in rayon::iter::map::{impl#2}::drive_unindexed<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}, (cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), rayon::iter::extend::ListVecConsumer> (self=..., consumer=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/map.rs:49
#23 0x0000555eb53872eb in rayon::iter::extend::{impl#29}::par_extend<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (self=0x7f1c96bf3a38, par_iter=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/extend.rs:576
#24 0x0000555eb53dd4cf in rayon::iter::from_par_iter::collect_extended<alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>, rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (par_iter=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/from_par_iter.rs:17
#25 0x0000555eb538718d in rayon::iter::from_par_iter::{impl#0}::from_par_iter<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::--Type <RET> for more, q to quit, c to continue without paging--
tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>> (par_iter=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/from_par_iter.rs:30
#26 0x0000555eb5393bfd in rayon::iter::ParallelIterator::collect<rayon::iter::map::Map<rayon::iter::flat_map::FlatMap<rayon::iter::flat_map::FlatMap<rayon::vec::IntoIter<cabinet_types::document::c_region::CRegion>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#0}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#1}>, cabinet_index::schemas::c_spans::tps_to_cspans::{closure_env#2}>, alloc::vec::Vec<(cabinet_types::document::c_span::CSpan, cabinet_index::schemas::c_spans::CSpanInputMetadata), alloc::alloc::Global>> (self=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/mod.rs:2046
#27 0x0000555eb53ec422 in cabinet_index::schemas::c_spans::tps_to_cspans (id=..., hash=..., dtps=0x7f1e80005400) at cabinet_index/src/schemas/c_spans.rs:266

Thread 50 (Thread 0x7f1dfa3f1700 (LWP 904603) "tokio-runtime-w"):
#0  0x00007f1ea9c122e9 in syscall () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555eb6b5c1a5 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x0000555eb6b60bb7 in std::sys::unix::locks::futex_condvar::Condvar::wait_optional_timeout () at library/std/src/sys/unix/locks/futex_condvar.rs:51
#3  std::sys::unix::locks::futex_condvar::Condvar::wait () at library/std/src/sys/unix/locks/futex_condvar.rs:35
#4  0x0000555eb5c384c0 in std::sys_common::condvar::Condvar::wait (self=0x7f1dfa3f14dc, mutex=0x7f1dfa3f14d4) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/condvar.rs:42
#5  0x0000555eb5c670e6 in std::sync::condvar::Condvar::wait<bool> (self=0x7f1dfa3f14dc, guard=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sync/condvar.rs:191
#6  0x0000555eb5c5f847 in rayon_core::latch::LockLatch::wait_and_reset (self=0x7f1dfa3f14d4) at src/latch.rs:242
#7  0x0000555eb56220b0 in rayon_core::registry::{impl#6}::in_worker_cold::{closure#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (l=0x7f1dfa3f14d4) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:473
#8  0x0000555eb5627665 in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::try_with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x555eb6bbafc0, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445
#9  0x0000555eb5626e2f in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x555eb6bbafc0, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421
#10 0x0000555eb5620f24 in rayon_core::registry::Registry::in_worker_cold<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x7f1e6004e480, op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:461
#11 0x0000555eb5626039 in rayon_core::registry::in_worker<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:879
#12 0x0000555eb561875a in rayon_core::join::join_context<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:132
#13 0x0000555eb5620db1 in rayon_core::join::join<cabinet_document::parse::_parse_document::{closure_env#1}, cabinet_document::parse::_parse_document::{closure_env#2}, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:105
#14 0x0000555eb55e9fb3 in cabinet_document::parse::_parse_document (pdf=...) at cabinet_document/src/parse.rs:211

Thread 49 (Thread 0x7f1c9f9fc700 (LWP 888162) "tokio-runtime-w"):
#0  0x00007f1ea9c122e9 in syscall () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555eb6b5c1a5 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x0000555eb6b60bb7 in std::sys::unix::locks::futex_condvar::Condvar::wait_optional_timeout () at library/std/src/sys/unix/locks/futex_condvar.rs:51
#3  std::sys::unix::locks::futex_condvar::Condvar::wait () at library/std/src/sys/unix/locks/futex_condvar.rs:35
#4  0x0000555eb5c384c0 in std::sys_common::condvar::Condvar::wait (self=0x7f1c9f9fc4dc, mutex=0x7f1c9f9fc4d4) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/condvar.rs:42
#5  0x0000555eb5c670e6 in std::sync::condvar::Condvar::wait<bool> (self=0x7f1c9f9fc4dc, guard=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sync/condvar.rs:191
#6  0x0000555eb5c5f847 in rayon_core::latch::LockLatch::wait_and_reset (self=0x7f1c9f9fc4d4) at src/latch.rs:242
#7  0x0000555eb56220b0 in rayon_core::registry::{impl#6}::in_worker_cold::{closure#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (l=0x7f1c9f9fc4d4) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:473
#8  0x0000555eb5627665 in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::try_with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x555eb6bbafc0, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445
#9  0x0000555eb5626e2f in std::thread::local::LocalKey<rayon_core::latch::LockLatch>::with<rayon_core::latch::LockLatch, rayon_core::registry::{impl#6}::in_worker_cold::{closure_env#0}<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x555eb6bbafc0, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421
#10 0x0000555eb5620f24 in rayon_core::registry::Registry::in_worker_cold<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (self=0x7f1e6004e480, op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:461
#11 0x0000555eb5626039 in rayon_core::registry::in_worker<rayon_core::join::join_context::{closure_env#0}<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>>, (cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>)> (op=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/registry.rs:879
#12 0x0000555eb561875a in rayon_core::join::join_context<rayon_core::join::join::call::{closure_env#0}<cabinet_types::document::c_link_position::DocumentCLinkPositions, cabinet_document::parse::_parse_document::{closure_env#1}>, rayon_core::join::join::call::{closure_env#0}<alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>, cabinet_document::parse::_parse_document::{closure_env#2}>, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:132
#13 0x0000555eb5620db1 in rayon_core::join::join<cabinet_document::parse::_parse_document::{closure_env#1}, cabinet_document::parse::_parse_document::{closure_env#2}, cabinet_types::document::c_link_position::DocumentCLinkPositions, alloc::vec::Vec<cabinet_document::poppler::xml::text::CTextPositionBuilder, alloc::alloc::Global>> (oper_a=..., oper_b=...) at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/join/mod.rs:105
#14 0x0000555eb55e9fb3 in cabinet_document::parse::_parse_document (pdf=...) at cabinet_document/src/parse.rs:211

cuviper · November 12, 2022, 3:35pm

Ok, yes, those are all waiting for work in the rayon pool, in_worker_cold. Next we should look at the backtraces of the rayon threads.

If you can reproduce this with a smaller number of threads, it will probably be easier to make sense of it.

arifd · November 12, 2022, 3:39pm

Okay, i will redeploy with a number of unrelated threads disabled, not sure i'm able to reduce many threads tbh though, but it should be for me easy to look through the traces and discard any that are irrelevant. This will take about an hour before it is deployed and the bug suddenly starts showing itself.

How do i identify these "rayon threads" or are you just asking to see the backtraces of all threads that currently have rayon invoked at the time of the bug?

kpreid · November 12, 2022, 4:13pm

In general, the way to identify “who owns a thread” — what that thread is supposed to be doing — is to look at the bottom of the call stack, to find the name of the functions/closures that the thread was spawned to call (rather than the top, where you see what it's currently calling).

Regarding your original problem, the thing I would look for is if there is anything in the code that you are running within a par_iter that is doing any kind of synchronization. The deadlock you describe sounds very much like what would happen if the code running inside some rayon operation is blocking on some lock that is being held by something else also blocking. In order to guarantee that rayon can make progress you need to avoid any such locking that introduces dependencies between threads ("this work must complete before the lock is released"). Only locking which can be understood as for message-passing (“lock while we write a new value, which we already computed, to this cell” sort of thing) is safe from deadlock, given how rayon uses its threads (stacking work on top of other work that's waiting).

arifd · November 12, 2022, 5:54pm

Okay great well I've got thread names so that's good

(although kinda curious that only two are busy most of the time)

is it possible that a par_iter() that calls another par_iter() could be the cause of a deadlock? I don't know that this is the situation, but I'm not doing anything else crazy inside these par_iters!

Also, here's the bit i don't get. if it is a deadlock, why is it not deterministic? why does it fail over time?

cuviper · November 12, 2022, 6:09pm

In general, it's fine for par_iter() to be nested -- they're all nested joins as far as the core runtime is concerned.

The sort of deadlock that kpreid describes is artificially introduced by work stealing, which can definitely be non-deterministic. If one job makes a rayon-blocking call, like waiting for a nested par_iter(), then it will try to steal other work in the meantime. If the stolen job has an outside dependency on the waiting job, then this will never be able to complete.

kpreid · November 12, 2022, 6:11pm

if it is a deadlock, why is it not deterministic? why does it fail over time?

This indicates that there is something about the ordering of events that is problematic. Individual pieces get stuck or not depending on whether the order they are scheduled in is a lucky or unlucky one. The way to make the program work correctly will be eliminating the “unlucky” case from the space of possibilities.

I'm not doing anything else crazy inside these par_iters!

Well, what are you doing? What is in the body of this iteration?

arifd · November 12, 2022, 6:33pm

One of my par_iter closures had an Arc<OnceCell>> passed into it. the only thing it did was cache some computation if any of its fellows happened to already call it. But I think it was perfectly safe because at worst, all threads would just wait while the OnceCell was being initiated. Nonetheless i have disabled this, lets see if i can reproduce the bug... <-- I think this might have been the buggy code!! I need to observe the system for another hour or so, but if I determine this was it, i will post the code!

Currently, none of the work done inside any of the other par_iters() depend on each other.
The things they do (apart from pure rust calculations)

Call std::process::Command
create tracing spans and .entered() them
use the serde_json::json! macro
there is one case of a into_par_iter() calling a par_iter().

I will now selectively remove a bunch of the par_iter()s and see if the bug disappears

cuviper · November 12, 2022, 7:10pm

Does that initialization make any rayon calls? If so, that's where you could get blocked. For example, a thread holds the OnceCell open for initialization and calls par_iter(); that iter finishes local work but has to wait for parts that got farmed out to other threads, so it starts work-stealing; the stolen job also tries to check the cell and blocks because it's currently being initialized -- deadlock!