Mixing SeqCst with other orderings

[manganhedenbergite]

hi folks! could you please provide some interesting examples of mixing SeqCst with other memory orderings. Also, does mixing SeqCst with other memory orderings always lead to something incorrect or is it just about understanding how to properly use other orderings with SeqCst? I'm trying my best to understand all of this memory ordering stuff, it seems a bit hard but I really really want to understand it thanks!

[yannham]

Mixing SeqCst with other oderings is not "incorrect" per se. By that I mean this is not UB and is properly defined by the C++20 memory model and thus the Rust memory model as well.

However, as Ralf Jung put it on some random GitHub comment (I don't remember where), I just have no idea how to simply reason about mixed SeqCst and non SeqCst operations. The C++ memory model is already quite intricate as it is. I personally have never seen such a case where you'd want to mix.

In particular because it tends to weaken the guarantees that you usually seek with SeqCst ordering: if you look at the specification for sequentially consistent ordering in particular, you can see stuff like doing X gets you what you expect from a seq cst operation or something totally unrelated, e.g:

OR, if there wasn't such an A, B may observe the result of some unrelated modification of M that is not memory_order_seq_cst.

It's hard to tell that nobody will ever have an actual use-case for mixing sequentially consistent memory ordering with other orderings, but so far it seems to never be what you actually want nor need, and is very hard to reason about.

Note that it's totally fine to have different orderings on different sets of atomics: it's legit to use SeqCst for atomics A and B but other non-SeqCst orderings for C and D, if they don't really interact. However it's fishy to mix, for a given atomic A, both SeqCst and non-SeqCst orderings, IMHO.

[steffahn]

Ultimately, to get a more detailed understanding it’s probably helpful to read the documentation of how things work in C++, since Rust delegates these details to the C++20 memory model (without the “consume” memory ordering).

This doesn’t make things particularly easy to understand, but that page does list an explicit caveat about mixing SeqCst with non-SeqCst operations (I’m translating code and identifiers into Rust)

Atomic operations tagged Ordering::SeqCst not only order memory the same way as release/acquire ordering (everything that happened-before a store in one thread becomes a visible side effect in the thread that did a load), but also establish a single total modification order of all atomic operations that are so tagged.

[…]

The single total order might not be consistent with happens-before. This allows more efficient implementation of Ordering::Acquire and Ordering::Release on some CPUs. It can produce surprising results when Ordering::Acquire and Ordering::Release are mixed with Ordering::SeqCst.

For example, with x and y initially zero,

// Thread 1:
x.store(1, Ordering::SeqCst); // A
y.store(1, Ordering::Release); // B
// Thread 2:
r1 = y.fetch_add(1, Ordering::SeqCst); // C
r2 = y.load(Ordering::Relaxed); // D
// Thread 3:
y.store(3, Ordering::SeqCst); // E
r3 = x.load(Ordering::SeqCst); // F

is allowed to produce r1 == 1 && r2 == 3 && r3 == 0, where A happens-before C, but C precedes A in the single total order C-E-F-A of Ordering::SeqCst (see Lahav et al).

Assuming that the above is true for Rust as well, at least miri seems unable to simulate it (though they do call out limitations w.r.t. weak memory orderings^[1].)

---

1. Further caveats that Miri users should be aware of:

   • […]
   • Weak memory emulation is not complete: there are legal behaviors that Miri will never produce. However, Miri produces many behaviors that are hard to observe on real hardware, so it can help quite a bit in finding weak memory concurrency bugs. To be really sure about complicated atomic code, use specialized tools such as loom.

   ↩︎

[manganhedenbergite]

yeah, I see. That makes sense. thanks!

[manganhedenbergite]

@steffahn

The single total order might not be consistent with happens-before

hey, could you please elaborate a bit more on this? Are you talking about the global single order maintained by SeqCst operations? also, how it cannot be consistent with happens-before?

@steffahn

(see Lahav et al)

Do you think after reading this paper, these kind of mixing will make sense to me? **get's excited**

also, could you please mention some other papers that one should definitely read, if they want to become very good with atomics and memory ordering, thanks!

[steffahn]

I have no complete understanding of the details here myself. At some point in the past - while trying to understand how Arc can be sound - I've reread parts of that page so many times until I had a pretty decent understanding of everything but SeqCst ordering^[1] (because those don't appear in Arc's internals at all).

The whole thing is quoted from the linked website, as mentioned. I only replaced some small things, e. g. replacing std::memory_order_seq_cst with Ordering::SeqCst, and things like that, to make it more readable in a context of Rust. The "single total order" is introduced a bit further up as

There is a single total order S on all memory_order_seq_cst operations, including fences, that satisfies the following constraints:

followed by a lot of formal details - so yes it's referring to that single order for all SeqCst operations.

---

1. and of course also excluding "consume" order which isn't relevant to Rust to begin with ↩︎

[manganhedenbergite]

yeah, I see.