Introducing Wavelet

announcements
1

Introducing Wavelet: A Graph-Based Stream Processing Runtime

Hey Rustaceans! :waving_hand:

I've just published Wavelet, a stream processing runtime that takes a graph-based approach to reactive systems. Instead of async/await, you build processing pipelines as computation graphs with deterministic execution.

Why Graph-Based Architecture?

The computation graph isn't just a different way to organize code - it solves fundamental problems in complex data processing systems:

Incremental Computation & Consistency

The Problem: In traditional imperative code, it's easy to create data consistency bugs where you use data A to compute data B, then later mutate data A, leaving data B stale and invalid. In sufficiently sophisticated systems, these dependency chains can become nearly impossible to track manually until a bug crops up.

The Solution: Nodes automatically broadcast updates to their children when they change, ensuring dependent computations stay consistent. The graph makes data dependencies explicit and handles propagation automatically.

Zero-Copy Fanout & Reusability

The Problem: When multiple components need the same data, you often end up copying it around or creating complex shared ownership patterns.

The Solution: Child nodes can reference parent data directly through the graph relationships. One data source can efficiently fan out to multiple consumers without copying, and expensive computations get reused across multiple downstream dependencies.

Testable State Machines

The Problem: Testing complex stateful logic often requires setting up elaborate mock environments and managing intricate state transitions.

The Solution: Each node (even a full subgraph) is essentially a mini state machine with well-defined inputs and outputs. You can test individual nodes in isolation, mock their dependencies easily, and compose them into larger testable systems.

How it works

Nodes process data and control downstream propagation:

let processor = NodeBuilder::new(DataProcessor::new())
    .triggered_by(&parent_node)
    .build(&mut executor, |data, ctx| {
        data.transform();
        Control::Broadcast  // Triggers downstream nodes
    });

Multi-depth scheduling ensures nodes execute in dependency order with predictable latency. The runtime handles I/O events, timers, and cooperative scheduling automatically.

Key characteristics

  • Single-threaded by design - deterministic execution, no hidden concurrency bugs
  • Event-driven - integrates I/O, timers, and yield events seamlessly
  • Dependency injection - swap implementations (live/test/mock) while keeping graph topology, build DSLs via factories creating complete memoized subgraphs (behaviors)
  • Cross-thread channels - ring buffers for external data injection
  • Garbage collection - automatic cleanup of completed subgraphs

Sweet spot

Built for continuous stream processing:

  • Financial market data processing
  • IoT sensor analytics
  • Real-time protocol handling
  • Live media processing

Not trying to replace Tokio for request/response - different tool for different jobs.

Current state & opportunities

Wavelet is production-capable for its intended use cases, but there are exciting areas for contribution:

:rocket: High impact: Networking module (WebSocket, UDP protocols), standard library nodes, real-world examples

:microscope: Research: Multi-threading approaches, observability

The runtime handles sophisticated internals (epoch-based scheduling, cross-platform precision timing) but keeps the API straightforward.

Check it out

Would love feedback from the community, especially if you're working on streaming/reactive systems! And if any of the contribution areas sound interesting, I'd be excited to collaborate.

Built this after working with various stream processing frameworks and wanting something that felt more native to Rust's ownership model. Hope others find it useful! :crab:

2 Likes