Hi the rust community !
I have been trying to find such a solution for months and I seem to be the only one with an actual need to deserialize an
AsyncRead array of json into structs. The idea is to be able to download a huge json file and to process its individual elements one by one, as it is downloading. This is mostly to not need 4GB of RAM to take everything into memory before processing a
Vec of deserialized structs.
I'm trying to do the same thing as the csv_async crate, to create a stream from an array of jsons in this test repo. An example I found that reflects what I have in mind is this blog post : Efficient parsing of JSON record sets in Rust but it is synchronous so uses the Trait constraint
Read instead of
So, based on the blog post I'm trying to make it all async by replacing:
and by using
async_compat to be able to use
tokio_serde with my
And after hours of trying to add
AsyncSeek to my
AsyncRead reader so that I can peek for the next
char , I'm starting to be limited by my understanding of these types, Trait constraints & lifetimes.
In my test repo I created a Minimal, Reproducible Example in the hope that someone could help me out. Or maybe tell me that what I'm trying to do is simply not achievable in the way that I have tried?
I'm defining a type in my deser_json module, which surely could be hinted as a Generic Type when creating the AsyncReader.. But really I would be happy if I got something that works already as it is, because it's big enough of challenge without it.