First, thank you for the futures crate, I like it very much so far, both with tokio_core::reactor::Core and futures_cpupool.
The goal was to modularize the application logic into many small futures and then compose them to act on a Stream.
I worry though about the rustc resource usage w.r.t. time and memory. Maybe it is wrong in the first place to try to design the application as a big future-driven state machine, but already with about 10 .map() or .and_then() chained on a Stream, compilation takes more than 2 minutes and peaks at several GB of RAM.
That happens even if the code itself is trivial, like .map( |item| item) or .and_then( |item| ok(item) ). I guess the combinatorics of the different Ok and Err branches grows out of control?
Is there some trick to keep the combinatorics for rustc manageable? I tried to give type hints in some places, but so far that did not help at all.
In general, how can one partition an application in a better way using futures? Thank you!
1 Like
Would you, per chance, have some code that you can share? Unless you accidentally wrote something which cannot possibly be compiled efficiently, I would personally consider such high compile-time resource consumption on a relatively tame future chain to be a good candidate for a bug report in either rustc or futures-rs.
I have produced some numbers for illustration using the attached code, and I also noticed that stable behaves a bit better than nightly
#![allow(unused_variables)]
#![allow(dead_code)]
#![allow(unused_imports)]
#![allow(unreachable_code)]
extern crate futures;
extern crate futures_cpupool;
extern crate tokio_core;
use std::collections::VecDeque;
use futures::{Future, Stream};
use futures::future::{err, lazy, ok, poll_fn, result, Executor};
use futures::sink::Sink;
struct Words {
words: VecDeque<String>,
}
impl Words {
fn new() -> Self {
let mut words = VecDeque::new();
words.extend(
vec!["Hello", "Future", "World"]
.into_iter()
.map(|x| String::from(x)),
);
Self {
words,
}
}
}
impl futures::Stream for Words {
type Item = String;
type Error = f32;
fn poll(&mut self) -> futures::Poll<Option<Self::Item>, Self::Error> {
return Ok(futures::Async::Ready(self.words.pop_front()));
}
}
fn process_stream() {
let mut core = tokio_core::reactor::Core::new().unwrap();
let future = Words::new()
// ==============================================
// The number of .and_then() as referenced on the plot
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.map_err(|err| { () })
.fold( (), |acc, x| {
println!("{:?}", x);
Ok(()) as Result<(), ()>
});
match core.run(future) {
Ok(_) => println!("core.run went fine"),
Err(x) => println!("core.run returned Err {:?}", x),
}
}
#[test]
fn test_process_stream() {
process_stream();
}
3 Likes
Indeed, and that issue also links to a massive number of related ones, such as https://github.com/rust-lang/rust/issues/38528...
@literda: From the issues linked by @vitalyd, the underlying compiler issue seems to relate to deeply nested types, which future combinators naturally build. Until this is fixed, a workaround to reduce type nesting is to use more boxed futures in order to "break" the future chains, like this:
let future = Words::new()
// Add a few boxed() here and there to avoid building a highly nested type
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.boxed()
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.and_then(|item| ok(item))
.boxed()
.and_then(|item| ok(item))
.map_err(|err| { () })
.fold( (), |acc, x| {
println!("{:?}", x);
Ok(()) as Result<(), ()>
});
This will cost you a bit more dynamic memory allocation and dynamic dispatch for the time being (which you can easily get rid of once the compiler is fixed), but that is probably a small price to pay for usable compile times 
1 Like
@HadrienG: This is awesome! I've read the comments about boxing and first tried things like .and_then(|item| Box::new(ok(item)) as Box<Future<Item = _, Error = _>>) which did not help.
Your suggestion using .boxed() works perfectly!
Credit for that trick should go to @sfackler. On rustc issue #38528 linked above, he used it in rust-postgres in order to keep compile times sane.
1 Like
