Show and tell: trains-valkey — loss-free Redis/Valkey failover via a total-order ring proxy (Rust)

A practical follow-up to my earlier TRAINS post. TRAINS is a formally-verified total-order broadcast primitive; trains-valkey is the payoff — and it is probably the part most people actually care about.

The problem: Redis/Valkey replication is asynchronous, so a failover can drop acknowledged writes. Sentinel trades durability for availability — if the primary dies after acking a write but before it replicates, that write is gone.

What it is: a proxy (Rust) that puts an unmodified Valkey/Redis behind the TRAINS ring. Writes are totally ordered and replicated across the ring before they are acknowledged to the client, so a node can be SIGKILLed mid-flight without losing an acked write. Reads bypass the ring.

The Rust-relevant parts:

  • async proxy speaking RESP, with an unmodified valkey underneath — no fork, no patch to the server
  • rustls for the ring transport (no OpenSSL)
  • the durability guarantee — no acknowledged-write loss across a crash — falls out of the ring total-order + pre-ack replication, which is the property I checked seven ways in the parent project
  • chaos-tested on EC2: N-node clusters, random SIGKILL of the primary under load, asserting no acked write is lost and survivors converge byte-identically

Honest about limits: it is consistency-first and control-plane-scale — small messages, small clusters. It halts under partition by design (PC/EC in PACELC), so it is not a drop-in for a high-throughput data plane. The point is loss-free failover for the cases where losing a write is worse than briefly pausing.

Code (MIT): GitHub - yeychenne/trains-valkey: Loss-free Valkey/Redis failover via TRAINS total-order broadcast (state-machine replication) · GitHub
Write-up: https://yeychenne.hashnode.dev/trains-valkey-no-write-loss

Happy to go into the proxy architecture, the RESP handling, or how the durability property maps onto the ring.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.