Serde serialization of arbitrary compound structures without hacking serde

This question is in reference to the question about access to a context during serialization & deserialization here: Serde question: Access to a shared context data within serialize and deserialize

Basically, I created parallel structs for all of the structs I want to serialize, with a reference to the shared context, and then I wrote serialize functions for each one.

Here's sample code that works, following the same pattern:

pub struct Context {
    name_table : Vec<String>,
}

struct NameIndex(usize);

struct NameIndexWithContext <'a> {
    the_context : &'a Context,
    name_index : usize,
}

impl <'a> serde::ser::Serialize for NameIndexWithContext <'a> {
  fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
  where S: serde::ser::Serializer {
    serializer.serialize_str(self.the_context.name_table[self.name_index].as_str())
  }
}

pub struct MyStruct {
  my_names : Vec<NameIndex>,
}

pub struct MyStructWithContext <'a> {
  the_context : &'a Context,
  my_names : Vec<NameIndex>,
}

impl <'a> serde::ser::Serialize for MyStructWithContext <'a> {
  fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
  where S: serde::ser::Serializer {
    let mut map = serializer.serialize_map(None)?;

    //Create a copy of the entire array!!
    let mut temp_vec = Vec::with_capacity(self.my_names.len());
    for the_name in self.my_names.iter() {
        temp_vec.push(NameIndexWithContext{
            the_context : self.the_context,
            name_index : the_name.0,
        });
    }
    map.serialize_entry("my_names", &temp_vec)?;

    map.end()
  }
}

fn main() {

  let my_context = Context{name_table: vec!["zero".to_string(), "one".to_string()]};

  let my_struct = MyStructWithContext {
    the_context : &my_context,
    my_names : vec![NameIndex(0), NameIndex(1)],
  };

  let my_json = json!(my_struct);
  println!("{}", serde_json::to_string_pretty(&my_json).unwrap());
}

While this works exactly as it should, it must duplicate the entire array in memory.

What I wish I could do is something that allows me to manufacture temporary objects, one at a time, as they're needed. For example, something like this, although this uses (probably misuses) a private serde interface:

impl <'a> serde::ser::Serialize for MyStructWithContext <'a> {
  fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
  where S: serde::ser::Serializer {

    let mut map = serializer.serialize_map(None)?;

    let nested_serializer = serde::private::ser::ContentSerializer(&mut map)?;

    //Only create temporary copies of the elements, one at a time.
    let seq = nested_serializer.serialize_seq(Some(self.my_names.len()))?;
    for the_name in self.my_names.iter() {
        seq.serialize_element(NameIndexWithContext{
            the_context : self.the_context,
            name_index : the_name.0,
        });
    }
    let nested_array = seq.end()?;

    map.serialize_entry("my_names", nested_array)?;

    map.end()
  }
}

This implementation also may well also unintentionally duplicate all of the elements in memory within serde.

What is the best way to accomplish the goal of serializing a compound structure within one function, while minimizing the required scratch memory?

Basically, is there a generalization of the serialize_struct() or serialize_struct_variant() function that could allow any combination of nested structs, maps, seqs, enums, etc?

Thank you all very much for putting up with my questions as I come up to speed on the intended design of serde.

Thanks again!

I would suggest doing something like this

I used a generic WithContext to allow me to seemlessly integrate a context onto whatever type I wanted (including foreign types like Vec<_>). This allows me to efficiently build out the exact impls I need using WithContext, without having to duplicate a lot of the work of making WithContext many many times.

The AddContext trait is pure syntactic sugar, if you don't like it, you can rip it out.

edit: updated with comments

4 Likes

You are awesome! I guess my brain was just too stuck in C++ to appreciate flexibility of traits and generics!

Thank you so much! You rock!

Yes, it's a bit mind boggling just how flexible Rust's type system is, especially given how restrictive it seems. Without seeing these sorts of patterns before, its hard to derive on your own.

Thanks for the example, very helpful for setting up an explicit architecture with Rust.

Workflow Pipelines for generic payloads provided by different delivery/persistence mechanisms (GraphQL, RestAPI Request payloads, Protobuf, DB rows, ...) transforming into your core domain becomes a breeze with this pattern: e.g.:

JSON/XML -> UnvalidatedDTO -> ValidatedModelState

Using results instead of panicking is also pretty neat for communicating errors back to the consumer.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.