How to cache and rotate files in a Rust-based app?

Hello fellow Rust enthusiasts!

I have a Rust-based application that downloads images from an S3 bucket, processes them, and shares metadata about the image. However, every time a user requests to process a file, it needs to download the file from S3 and process it again, which can be time-consuming and resource-intensive.

I'm looking for a way to cache a certain amount of files on disk, say 200GB, and rotate least used files so that the app only downloads new files. Is there a Rust crate or library that can help me achieve this goal?

I'd appreciate any guidance or advice on this matter. Thank you in advance!

This is called an LRU cache. There are in-memory implementations of the pattern, e.g. lru. You can cheat by re-using such an in-memory cache for checking if a newly inserted item was evicted. Example.

However, this is not a very robust solution, as it requires multiple filesystem operations to implement correctly, which makes the higher-level cache operations non-atomic. This can cause problems if e.g. two processes try to use the same cache at the same time.

The better solution would be to use a real database with transactions and atomicity.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.