A practical follow-up to my earlier TRAINS post. TRAINS is a formally-verified total-order broadcast primitive; trains-valkey is the payoff — and it is probably the part most people actually care about.
The problem: Redis/Valkey replication is asynchronous, so a failover can drop acknowledged writes. Sentinel trades durability for availability — if the primary dies after acking a write but before it replicates, that write is gone.
What it is: a proxy (Rust) that puts an unmodified Valkey/Redis behind the TRAINS ring. Writes are totally ordered and replicated across the ring before they are acknowledged to the client, so a node can be SIGKILLed mid-flight without losing an acked write. Reads bypass the ring.
The Rust-relevant parts:
- async proxy speaking RESP, with an unmodified valkey underneath — no fork, no patch to the server
- rustls for the ring transport (no OpenSSL)
- the durability guarantee — no acknowledged-write loss across a crash — falls out of the ring total-order + pre-ack replication, which is the property I checked seven ways in the parent project
- chaos-tested on EC2: N-node clusters, random SIGKILL of the primary under load, asserting no acked write is lost and survivors converge byte-identically
Honest about limits: it is consistency-first and control-plane-scale — small messages, small clusters. It halts under partition by design (PC/EC in PACELC), so it is not a drop-in for a high-throughput data plane. The point is loss-free failover for the cases where losing a write is worse than briefly pausing.
Code (MIT): GitHub - yeychenne/trains-valkey: Loss-free Valkey/Redis failover via TRAINS total-order broadcast (state-machine replication) · GitHub
Write-up: https://yeychenne.hashnode.dev/trains-valkey-no-write-loss
Happy to go into the proxy architecture, the RESP handling, or how the durability property maps onto the ring.