Hi,
first of all, a warm hello to the rust community.
I am experimenting with the futures and futures-cpupool libraries in order to get a parallel pipelined processing of a stream of inputs. The execution should happen like this:
- The
source
produces data 0. -
stage 0
processes data 0. At the same time, thesource
produces data 1. -
stage 1
processes data 0. At the same time, thestage 0
processes data 1. At the same time,source
produces data 2. - ...
However, it is mandatory, that every stage processes the input in the correct order (the stages might have state). The actual amount of parallelism should be handled by the thread pool.
Unfortunately, I just cannot get it to run in parallel.
My example code uses a random time of execution between the stages. I get the output
Source produced 1 (in thread "source")
Source produced 2 (in thread "source")
Stage 0 received 1 (in thread "pool-0")
Source produced 3 (in thread "source")
Stage 1 received 1 (in thread "pool-1")
Source produced 4 (in thread "source")
Stage 2 received 1 (in thread "pool-2")
Stage 0 received 2 (in thread "pool-3")
Source produced 5 (in thread "source")
Stage 1 received 2 (in thread "pool-0")
Stage 2 received 2 (in thread "pool-1")
Source produced 6 (in thread "source")
Source produced 7 (in thread "source")
Stage 0 received 3 (in thread "pool-2")
Source produced 8 (in thread "source")
Stage 1 received 3 (in thread "pool-3")
Stage 2 received 3 (in thread "pool-0")
Source produced 9 (in thread "source")
Stage 0 received 4 (in thread "pool-1")
Stage 1 received 4 (in thread "pool-2")
Stage 2 received 4 (in thread "pool-3")
Stage 0 received 5 (in thread "pool-0")
Stage 1 received 5 (in thread "pool-1")
Stage 2 received 5 (in thread "pool-2")
Stage 0 received 6 (in thread "pool-3")
Stage 1 received 6 (in thread "pool-0")
Stage 2 received 6 (in thread "pool-1")
Stage 0 received 7 (in thread "pool-2")
Stage 1 received 7 (in thread "pool-3")
Stage 2 received 7 (in thread "pool-0")
Stage 0 received 8 (in thread "pool-1")
Stage 1 received 8 (in thread "pool-2")
Stage 2 received 8 (in thread "pool-3")
Stage 0 received 9 (in thread "pool-0")
Stage 1 received 9 (in thread "pool-1")
Stage 2 received 9 (in thread "pool-2")
Obviously, apart from the producing stage (which is running in its own thread) I cannot get the next stages to interleave.
Is it possible to get the futures library to behave the way I want?