Serde deserialization from_borrowed_bytes + SeqAccess

Hi all,
I have another Serde question.

I'm trying to implement a serde Deserializer for a binary format, let's say, to keep things simple, that the format define two types:

bool: a byte that can be either 0 or 1
seq: first 3 bytes unsigned little endian int (is the size), following size bytes are bool

(line x) refer to lines of this Rust Playground

Requirement 1

In deserialize_newtype_struct (line 124) I need to call visitor.visit_borrowed_bytes because the data format primitive seq is implemented as pub struct Bytes<'b>(&'b [u8]); (line 363)

Requirement 2

I need also to derive Deserialize for generic struct for that I need to implement SeqAccess for my Deserializer (line 341) so I can call visitor.visit_seq in deserialize_struct (line 116)

In order to satisfy req 1 I have to implement Deserialize as impl<'de> de::Deserializer<'de> for &'de mut Deserializer<'de> Rust Playground

But in order to satisfy req 2 I have to implement Deserialize as impl<'de, 'a> de::Deserializer<'de> for &'a mut Deserializer<'de> Rust Playground

I'm not able to satisfy both req 1 and req 2. Where I'm wrong?

As reference the project for which I need to do it is this one https://github.com/stratum-mining/stratum/tree/main/protocols/v2/serde-sv2 for now it satisfy just req 2 but I would like to have also req 1 in order to not have to instantiate a new vector here https://github.com/stratum-mining/stratum/blob/main/protocols/v2/serde-sv2/src/sv2_primitives.rs#L154

Ty :slight_smile:

Here's a version that works.

Playground

I changed two things. The first thing I changed was the function signature on get_slice and parse_bytes from

fn get_slice(&'de self, len: usize) -> Result<&'de [u8]>
fn parse_bytes(&'de self) -> Result<&'de [u8]>

to

fn get_slice(&self, len: usize) -> Result<&'de [u8]>
fn parse_bytes(&self) -> Result<&'de [u8]>

Notice that I removed the 'de lifetime specifier on self. That is overly restrictive since it is saying that in order to use those functions the borrow of self must be for the exact same lifetime as the data inside of self (usually you want to allow for the outer borrow to be shorter than the inner one). To illustrate that a bit better expanded version of those function signatures would be

fn get_slice(self: &'de Deserializer<'de>, len: usize) -> Result<&'de [u8]>

and my change effectively made it

fn get_slice<'a>(self: &'a Deserializer<'de>, len: usize) -> Result<&'de [u8]>

Looking at it this way, the new version allows for the borrow of the Deserializer (what I've given lifetime 'a) to be for shorter than the borrow of the underlying data 'de

The second change, which may or may not work for you, is i changed the struct definition from

pub struct Deserializer<'de> {
    input: &'de mut [u8],
    cursor_: usize,
    cursor: *mut usize,
    len: u8,
}

to

pub struct Deserializer<'de> {
    input: &'de [u8],
    cursor_: usize,
    cursor: *mut usize,
    len: u8,
}

This has to do with borrowing rules as to why this is needed. Without the change, the compiler complains about lifetimes for the get_slice definition. Specifically it complains about

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter in function call due to conflicting requirements
  --> src/main.rs:76:20
   |
76 |         let res = &self.input[cursor..=cursor + l];
   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 64:5...
  --> src/main.rs:64:5
   |
64 |     fn get_slice(&self, len: usize) -> Result<&'de [u8]> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that reference does not outlive borrowed content
  --> src/main.rs:76:20
   |
76 |         let res = &self.input[cursor..=cursor + l];
   |                    ^^^^^^^^^^
note: but, the lifetime must be valid for the lifetime `'de` as defined on the impl at 63:6...
  --> src/main.rs:63:6
   |
63 | impl<'de> Deserializer<'de> {
   |      ^^^
note: ...so that the expression is assignable
  --> src/main.rs:77:9
   |
77 |         Ok(res)
   |         ^^^^^^^
   = note: expected `std::result::Result<&'de [u8], _>`
              found `std::result::Result<&[u8], _>`

The primary issue here is that the lifetime of res is not 'de. This comes out of the fact that since self has mutable access to input, you can't reborrow (which is what you're doing) with the same lifetime as the lifetime of the mutable borrow. (I always forget the technical reason for this, but someone may come in with the correct info for that). And the reason I thought it was fine to change it was that it didn't seem like you actually needed mut access to the data since you probably don't want to make any changes to the data anyway.

All that said, this code is not exactly idiomatic Rust. I took a quick pass at it and came up with this. The basic changes were to get rid of the pointer type which you don't really need and it introduces unsafe code into rust which I try to avoid completely. The other thing it changes is that instead of holding a reference to the entire original slice of data, it only stores the remaining data in the struct which simplifies things (and also in theory would allow you to add back in the mut into the struct definition if thats needed for some reason). In theory, you also probably don't want len in the struct and instead make a new custom struct to hold that len, but I'll leave that as an exercise for you if you want to try that. (I believe there's an example of that in the serde_json code).

Edit: Fixed last link

2 Likes

Thanks a lot! it works.

about:

I'm not seeing the changes maybe you put the wrong link?
Btw would like to store all the original slice cause the de-serialized struct are not copied from to the original data but just point to it. I'm still try to figure out if I can really have some advantage from that so if you have any advice will be very welcomed

about:

That is how I did it in the original code then I changed it in the example to simplify things.

Sorry you're right, here's the link to that code Playground Link

That's already being handled by the way Deserializer works. Basically, Deserializer is getting a view of some data and then as the data is parsed, it creates a bunch of smaller views into that data for use by the deserialized struct. And when the Deserializer struct is no longer needed (and is dropped), it drops its copy of the view into any of the remaining data, but the deserialized struct continues to maintain its views into the original data.

Perfect

1 Like

Clear, ty

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.