How to cache and rotate files in a Rust-based app?

help
1

Hello fellow Rust enthusiasts!

I have a Rust-based application that downloads images from an S3 bucket, processes them, and shares metadata about the image. However, every time a user requests to process a file, it needs to download the file from S3 and process it again, which can be time-consuming and resource-intensive.

I'm looking for a way to cache a certain amount of files on disk, say 200GB, and rotate least used files so that the app only downloads new files. Is there a Rust crate or library that can help me achieve this goal?

I'd appreciate any guidance or advice on this matter. Thank you in advance!

2

This is called an LRU cache. There are in-memory implementations of the pattern, e.g. lru. You can cheat by re-using such an in-memory cache for checking if a newly inserted item was evicted. Example.

However, this is not a very robust solution, as it requires multiple filesystem operations to implement correctly, which makes the higher-level cache operations non-atomic. This can cause problems if e.g. two processes try to use the same cache at the same time.

The better solution would be to use a real database with transactions and atomicity.

2 Likes