Exponential type size of async function

I'm making an async function for a library. It's not very big, but yet I'm meeting the error:


error: reached the type-length limit while instantiating `std::pin::Pin::<&mut std::future...}[1]::poll[0]::{{closure}}[0])]>`
    |
    = note: consider adding a `#![type_length_limit="9479903"]` attribute to your crate

error: aborting due to previous error

Commit: https://github.com/Ploppz/s3-upload/tree/e3168d83b73ad373c928a89b1bdca1d01da4299e
Async function in question: https://github.com/Ploppz/s3-upload/blob/e3168d83b73ad373c928a89b1bdca1d01da4299e/src/lib.rs#L75

I have asked around and it seems nobody knows how to overcome this and only suggest that I do as the error suggests. But that makes for a long compile time; and I suppose it also requires the type_length_limit in any crate that uses my library.

I really want to get to the bottom of what part of my code is the 'bottleneck'.
In the linked-to commit I have adjusted the type length limits to be just enough to compile the tests (cargo test). It required a limit of 10 million. How can it possibly get this high? I know it's the return type that becomes complex due to combinators and what-not that are applied on the futures in the function but I only have some ten-fold of combinators/wrappers, so there must be some exponential growth at play here?

Now hold on to your hats because what I did next was to try to use #[tokio::test] async fn ... in tests. Diff: https://github.com/Ploppz/s3-upload/commit/71d2a668a4e2fb104428046f77a93a0ce0a19e9f (ignore the example).

Now I get:

error: reached the type-length limit while instantiating `std::pin::Pin::<&mut std::future...}[1]::poll[0]::{{closure}}[0])]>`
    |
    = note: consider adding a `#![type_length_limit="18960398"]` attribute to your crate

It almost doubled from 10 million to 19 million just because I now call the problematic async function from inside an async function. How is that possible? Is it a bug?

1 Like

The proposed fix is found here. Basically the issue is that when functions have a signature like this:

fn then<Fut, F>(self, f: F) -> Then<Self, Fut, F> where
    F: FnOnce(Self::Output) -> Fut,
    Fut: Future,

The return type Then includes the type of Fut in both F and Fut, thus the exponential growth. You can fight this by hiding types with Box<dyn Future>.

I think that writing your code with async await instead of combinators is also likely to decrease the type size.

3 Likes

That makes sense, thanks a ton!

2 Likes