I have a Rust-based application that downloads images from an S3 bucket, processes them, and shares metadata about the image. However, every time a user requests to process a file, it needs to download the file from S3 and process it again, which can be time-consuming and resource-intensive.
I'm looking for a way to cache a certain amount of files on disk, say 200GB, and rotate least used files so that the app only downloads new files. Is there a Rust crate or library that can help me achieve this goal?
I'd appreciate any guidance or advice on this matter. Thank you in advance!
This is called an LRU cache. There are in-memory implementations of the pattern, e.g. lru. You can cheat by re-using such an in-memory cache for checking if a newly inserted item was evicted. Example.
However, this is not a very robust solution, as it requires multiple filesystem operations to implement correctly, which makes the higher-level cache operations non-atomic. This can cause problems if e.g. two processes try to use the same cache at the same time.
The better solution would be to use a real database with transactions and atomicity.