Futures-rs compile time and memory usage


#1

First, thank you for the futures crate, I like it very much so far, both with tokio_core::reactor::Core and futures_cpupool.

The goal was to modularize the application logic into many small futures and then compose them to act on a Stream.

I worry though about the rustc resource usage w.r.t. time and memory. Maybe it is wrong in the first place to try to design the application as a big future-driven state machine, but already with about 10 .map() or .and_then() chained on a Stream, compilation takes more than 2 minutes and peaks at several GB of RAM.

That happens even if the code itself is trivial, like .map( |item| item) or .and_then( |item| ok(item) ). I guess the combinatorics of the different Ok and Err branches grows out of control?

Is there some trick to keep the combinatorics for rustc manageable? I tried to give type hints in some places, but so far that did not help at all.

In general, how can one partition an application in a better way using futures? Thank you!


#2

Would you, per chance, have some code that you can share? Unless you accidentally wrote something which cannot possibly be compiled efficiently, I would personally consider such high compile-time resource consumption on a relatively tame future chain to be a good candidate for a bug report in either rustc or futures-rs.


#3

I have produced some numbers for illustration using the attached code, and I also noticed that stable behaves a bit better than nightly

#![allow(unused_variables)]
#![allow(dead_code)]
#![allow(unused_imports)]
#![allow(unreachable_code)]

extern crate futures;
extern crate futures_cpupool;
extern crate tokio_core;

use std::collections::VecDeque;
use futures::{Future, Stream};
use futures::future::{err, lazy, ok, poll_fn, result, Executor};
use futures::sink::Sink;

struct Words {
  words: VecDeque<String>,
}

impl Words {
  fn new() -> Self {
    let mut words = VecDeque::new();
    words.extend(
      vec!["Hello", "Future", "World"]
        .into_iter()
        .map(|x| String::from(x)),
    );
    Self {
      words,
    }
  }
}

impl futures::Stream for Words {
  type Item = String;
  type Error = f32;
  fn poll(&mut self) -> futures::Poll<Option<Self::Item>, Self::Error> {
    return Ok(futures::Async::Ready(self.words.pop_front()));
  }
}

fn process_stream() {
  let mut core = tokio_core::reactor::Core::new().unwrap();
  let future = Words::new()

    // ==============================================
    // The number of .and_then() as referenced on the plot
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))

    .map_err(|err| { () })
    .fold( (), |acc, x| {
      println!("{:?}", x);
      Ok(()) as Result<(), ()>
    });
  match core.run(future) {
    Ok(_) => println!("core.run went fine"),
    Err(x) => println!("core.run returned Err {:?}", x),
  }
}

#[test]
fn test_process_stream() {
  process_stream();
}


#4

Probably same issue as https://github.com/rust-lang/rust/issues/43018


#5

Indeed, and that issue also links to a massive number of related ones, such as https://github.com/rust-lang/rust/issues/38528

@literda: From the issues linked by @vitalyd, the underlying compiler issue seems to relate to deeply nested types, which future combinators naturally build. Until this is fixed, a workaround to reduce type nesting is to use more boxed futures in order to “break” the future chains, like this:

let future = Words::new()

    // Add a few boxed() here and there to avoid building a highly nested type
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .boxed()
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .and_then(|item| ok(item))
    .boxed()
    .and_then(|item| ok(item))

    .map_err(|err| { () })
    .fold( (), |acc, x| {
      println!("{:?}", x);
      Ok(()) as Result<(), ()>
    });

This will cost you a bit more dynamic memory allocation and dynamic dispatch for the time being (which you can easily get rid of once the compiler is fixed), but that is probably a small price to pay for usable compile times :slight_smile:


#6

@HadrienG: This is awesome! I’ve read the comments about boxing and first tried things like .and_then(|item| Box::new(ok(item)) as Box<Future<Item = _, Error = _>>) which did not help.
Your suggestion using .boxed() works perfectly!


#7

Credit for that trick should go to @sfackler. On rustc issue #38528 linked above, he used it in rust-postgres in order to keep compile times sane.