Async download very large files

Hi guys,
I'm new to this forum and the language itself. I'm not an experienced programmer but I definitely like rust!
I was trying to asynchronously download two files as a POC of sequential vs asynchronous.
These 2 files are around 800 MB each and the challenge is not using RAM and then write to file, rather stream to file directly.

I tried this strategy but of course all gets loaded into memory:


use std::{time::{Instant}, io::{Cursor} };

use tokio::{
    io::{ self, AsyncWriteExt, AsyncReadExt },
    fs::{ File, remove_file },
    join,
};

use tokio_stream::StreamExt;

type Result<T> = std::result::Result<T, Box<dyn std::error::Error + Send + Sync>>;


async fn download_files(url: &str, path: &str) -> Result<()>  {
    let mut file = File::create(path).await?;
    println!("Downloading {}...", url);

    let mut stream = reqwest::get(url)
        .await?
        .bytes_stream();

        /* What now? :( */

    println!("Downloaded {}", url);
    Ok(())
}

For the reqwest blocking mode there's the copy_to(&mut file) method but I really don't know how to do this in async.

Thank you so much
Antonio

This should do it:

async fn download_files(url: &str, path: &str) -> Result<()>  {
    let mut file = File::create(path).await?;
    println!("Downloading {}...", url);

    let mut stream = reqwest::get(url)
        .await?
        .bytes_stream();

    while let Some(chunk_result) = stream.next().await {
        let chunk = chunk_result?;
        file.write_all(&chunk).await?;
    }

    file.flush().await?;

    println!("Downloaded {}", url);
    Ok(())
}
9 Likes

Thank you so much Alice!

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.