Tests fail on ubuntu but not when run separately

[tangentstorm]

I don't know what to make of this. I have a collection of tests that pass when run individually, and which pass together on my windows box, but a handful fail when I run them all on ubuntu.

I am running the exact same version of rust on both machines:

• stable-x86_64-unknown-linux-gnu unchanged - rustc 1.64.0 (a55dd71d5 2022-09-19)
• stable-x86_64-pc-windows-msvc unchanged - rustc 1.64.0 (a55dd71d5 2022-09-19)

The failures are all panics are all inside a macro expansion, and relate to an "attempt to subtract with overflow". As far as I know, there is no platform-specific code in my crate, and (as far as I can tell) there is no global state that could be being corrupted by the tests running in batch mode (with plain old cargo test).

Running cargo test -- --test-threads=1 does not remove the errors, so I don't think it's related to threading.

There are no compiler warnings.

I'm at a loss here. How should I go about trying to debug this?

(The code is at GitHub - tangentstorm/bex: A rust crate for boolean expressions (expression trees, decision diagrams, etc.) and I'm literally just running cargo test --lib with the latest stable rust.)

[bjorn3]

Can you show the test failures?

[tangentstorm]

@bjorn3 the tests include a massive amount of debug output, but they're all failing on different instances of the same macro invocation:

thread 'main' panicked at 'attempt to subtract with overflow', src/solve.rs:303:3


failures:
    solve::test_multi_bdd
    solve::test_nano_anf
    solve::test_nano_bdd
    solve::test_nano_swap
    solve::test_tiny_anf
    solve::test_tiny_bdd
    solve::test_tiny_swap

It's basically setting up a big graph that represents the boolean logic involved in factoring a number, and then simplifying the graph to get the answers.

[bjorn3]

You seem to be using thread local storage in several places. Maybe this semi-global state messes things up? Would it be possible to reduce your dependency on (semi-)global state?

[tangentstorm]

Hrm. It seems to be failing on the very last line of the macro:

    let tests = bdd::COUNT_XMEMO_TEST.with(|c| *c.borrow() );
    let fails = bdd::COUNT_XMEMO_FAIL.with(|c| *c.borrow() );
    println!("TOTAL XMEMO STATS: tests: {} fails: {} hits: {}", tests, fails, tests-fails);

These are supposed to be thread-local counters:

// cache lookup counters:
thread_local!{
  pub static COUNT_XMEMO_TEST: RefCell<u64> = RefCell::new(0);
  pub static COUNT_XMEMO_FAIL: RefCell<u64> = RefCell::new(0); }

Since all my other assertions are passing, I'm guessing something is causing these values to get corrupted, so that the number of fails appears to be higher than the number of tests. This can't actually happen unless these thread local variables are getting corrupted somehow ....

Is it possible the thread-local variables are re-used on some platforms and I need to reset them on each test run?

[bjorn3]

When using --num-threads 1 or when running on a target without thread support like wasm, all tests run on the main thread and thus reuse thread local storage. In all other cases each test should get their own thread and thus thread local storage.

[tangentstorm]

Oddly, I had already accounted for this, but I was doing this:

bdd::COUNT_XMEMO_TEST.with(|c| *c.borrow_mut()=0 );

Changing it to this fixed the problem:

    bdd::COUNT_XMEMO_TEST.with(|c| c.replace(0) );

Thanks!

[tangentstorm]

Well. Hmm. I was fooling myself there (I had commented out the line that used the variables).

I'm not sure why my logic isn't correctly clearing the counters but at least now I know where to look.

This was just something I added for debugging anyway, so maybe I'll just take it out.