Hi there, I'm looking for some help on transitioning over a Serde implementation to use zero allocation. I'm new to Rust and have been researching as much as possible but any advice in that regard would be welcome as well!
I have a small CSV parser where I read in a file, do some basic deserialization with Serde and then write back to a buffer to return some data. This is built using a combination of the CSV crate + Serde.
I have a basic working implementation right now that seems to use up extra memory. One of the files is around 80MB in size and by the end of the process uses around 300MB. I might see even larger files in the future so was interested in a zero allocation technique to try and make things a little more efficient.
I've tried following the CSV crate's documentation on this technique but have hit a snag when it comes to generics. One piece to this puzzle is that the CSV parser is built to handle different types of CSV files, so depending on the type of file that is uploaded, I pass a different type of struct to deserialize with. The main problem I'm running into is incorporating a lifetime in the appropriate fashion.
I've added a scaled down example of my code. I think I've got the lifetime implemented correctly on the
RecordA struct as well as the
read_and_write function signature. Right now I'm seeing two compiler complaints.
as mutable because it is also borrowed as immutable
raw_recorddoes not live long enough`
I think what's happening is that the
raw_record.deserialize call is consuming the raw_record variable and it's going out of scope at that point, but I'm not sure how to work around this. I've tried dereferencing and using
.copy(), etc... on
raw_record but these don't seem to be correct. I've spun my wheels on this for quite a while and haven't been able to figure it out.
Lifetimes are still pretty new to me also and I get the concepts behind them but I'm not even sure if I've gotten everything implemented correctly to use them with Serde. In particular, I'm having some trouble understanding the differences between
serde::DeserializeOwned trait bounds. (I'm new and can only add two links to my post but this part of the Serde docs is what I'm referring to: serde.rs/lifetimes.html#trait-bounds). Previously, I was using
DeserializeOwned to handle the deserialization, but when introducing zero allocation I believe I need to switch to
Deserialize so that I can just use references to the original
read_buffer. Is this assumption correct?
Anyways, thanks for reading. Any advice would be appreciated!