Caching and Rotating Files in Rust for Downloading from S3 Storage

help
1

I'm building a Rust app that needs to download thousands of files from S3 storage, process them, and publish a report. I want to have a cache on the local system with a size cap and remove old files when the limit is reached. How can I accomplish this?

Even having an app outside of the Rust app

I tried to use rclone with local mount and s3 as remote, but the main problem is fetching new data when the s3 remote is updated the local stayed outdated until poll/cache time expires

2

Does this project do the job? It's written in Python.

The amount of code needed isn't large. You could reimplement it in Rust without too much problem.

1 Like
3

I can try

4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.