I am working on a scrapping project that scraps data from paginated APIs and Websites. I use Rayon to scrape it parallelly and store the scrapped data in a Sled database.
Here is the link to the playground: Rust Playground
I basically use a SledHandler struct to flush both the Sled Tree and Db when the variable is dropped and I do a sled_handler.clone()
to pass it on to the Rayon's scope.spawn
. It works as expected. But I somehow doubt that there is a better way of doing this.