There are bunch of different crates that provides S3 interaction, like aws_sdk_s3, s3, minio
Result I wanted is to download large file from S3 by axum endpoint, for example "/file".
The core problem I met right now: when I try to download a file, rust service started to load data to RAM. I suppose it's not right behaviour..?
// Axum + rust-s3
pub async fn download_file_handler(Path(user): Path<Uuid>) -> impl IntoResponse {
match download_file(user).await {
Ok(stream) => {
let response = axum::response::Response::builder();
let body = axum::body::Body::from_stream(stream.bytes);
response.body(body).unwrap()
}
Err(err) => "File download failure".into_response(),
}
}
pub async fn download_file(user: Uuid) -> Result<ResponseDataStream> {
let region = s3::Region::Custom {
region: "".to_owned(),
endpoint: S3_ENDPOINT.to_owned(),
};
let creds = s3::creds::Credentials::new(
Some(S3_ACCESS_KEY),
Some(S3_SECRET_KEY),
None,
None,
None,
)?;
let bucket = s3::Bucket::new(CUSTOM_BUCKET, region, creds)?.with_path_style();
let stream = bucket
.get_object_stream(format!("{}/{}", user, CUSTOM_FILE_NAME))
.await?;
Ok(stream)
}
So, maybe someone have any ideas?
(I tried axum::response::Redirection, it works when I'm translating data directly from s3 with presigned request but it's interesting why previous way doesn't work)
What did you expect would happen when you get the blob from S3? Where is the data supposed to go if not into RAM? Or do you mean the whole file is loaded at once and not streamed?
The second one. Initially I assumed that I'm taking whole file into RAM, but memory jumps only on response sending.
No matter how obvious this can be, but rust-s3 crate provides whole bytes only for get_object fn, not for get_object_stream, while aws-sdk-s3 always provides ByteStream on response. So, in both cases it shouldn't go into RAM entirely.
Maybe im wrong, but in my understanding the data streaming means that data goes partially, not at once. Current situation shows that memory continuosly filling up by data, and as a result, when I'm trying to download 5Gb file, it looks like memory leak when process holds 11Gb..
Maybe you are processing multiple requests to that file concurrently? Have you thought about caching the file on the server, instead of streaming it from s3 upon every request?
well a stream is going to give you the data gradually yeah but where are you putting the stream data that you get? unless you use as you get it you will end up storing it all in ram before you use it
Nope, it was a single request. Caching is a good idea, but right now I think that such requests will not be too frequent. And right now Im in the same network as my S3.
The thing is I'm just puts it to the Axum body, not making any buffers to store whole data. Right now it looks like I simply upload it fully to the RAM. Maybe I should test this thing with local files, not S3, as example. Maybe streams are differ
Server and your client part have to have a parameter - max chunk size in the memory. I realized that after attempting to download multi gigabytes file. After introducing the parameter, memory usage went down to normal. I know the parameter for my server, but no for Axum yet.
Okay, I figured this out. The code above works fine. Also, I tested local file downloading with tokio + tokio-util ReaderStream. Everything is perfect, and there is no high memory usage.
Initially I got stuck while testing this with Postman and Swagger. It turns out the server couldn't write the data to disk and stored it in memory instead. As a result, swagger waits for the entire file to be loaded into RAM and then provides a link to it..
So, I just saw the problem from the API client perspective and didn't test the browser itself.
(I found only dead topics on how to bypass this swagger restriction, so.. nevermind )
)