Multiple variants of serde_derive outputs?

I suspect this isn't doable, and I should approach the idea another way, but...

I have various structs which describe some settings, something like

struct SomeSettings {
    selected: bool,
    thing: bool,
    another: i64,
    more: SubSettings,
}

struct SubSettings {
    selected: bool,
    more: i64,
    examples: String,
}

Using serde_derive I'm saving these to a file with serde_json, and all is great.

However I want to check if these settings have changed, and if so, "do something". I can quite easily serialize this and hash the output, and if the hash changes then the settings have changed. Also, this works great.. almost:

  1. Some of the struct members don't need to be saved at all, so I can annotate them with #[serde(skip)]. Good.
  2. However some of the struct members need to be saved, but don't need to be included in the hash output.

Essentially I want to skip the fields conditionally, based on how I'm invoking the serialization. I want to do something vaguely like this psuedo-code (where of course the condition stuff is made up):

use serde_derive; // 1.0.126
use serde_derive::Serialize;

#[derive(Default, Serialize)]
struct SomeSettings {
    #[serde(skip)]
    selected: bool,
    thing: bool,
    another: i64,
    more: SubSettings,
}

#[derive(Default, Serialize)]
struct SubSettings {
    #[serde(skip)]
    selected: bool,
    more: i64,
    #[serde(condition="write_to_file")]
    examples: String,
}

fn write_to_file(data: String) {
    dbg!(data);
}
fn compute_hash(data: String) -> i64 {
    use std::hash::Hasher as _;
    use std::collections::hash_map::DefaultHasher;
    let mut hasher = DefaultHasher::new();
    hasher.write(&data.as_bytes());
    hasher.finish()
}

fn main() {
    let mut settings = SomeSettings::default();

    // Write everything to file
    write_to_file(serde_json::to_string(&settings, conditions=["write_to_file"]).unwrap());

    // Compute hash
    let hash1 = compute_hash(serde_json::to_string(&settings).unwrap());

    // Change setting
    settings.more.examples = "Change not incorporated into hash".into();

    // The hash does not change because `SubSettings.examples` is marked in some way
    let hash2 = compute_hash(serde_json::to_string(&settings).unwrap());
    assert_eq!(hash1, hash2);
}

I could of course probably implement the Serialize trait manually, but using derive is very beneficial as it avoids lots of duplication and avoids the possibility of forgetting to add a field to the serialization output. Although even still, it's not clear how I would implement the conditional serialization

There is skip_serializing_if however it seems only suited to "static" methods - e.g I can't think of how I could use it in the above example (short of maybe having a method which looks at a global variable type thing)

To summarize:

  1. I want to dump a struct to a JSON file. serde_derive and serde_json do this excellently.
  2. I also want a way to compute a hash for "most" of the data in the struct, ignoring a few struct members
  3. I'd like to avoid having to manually implement things which do hash.write(self.field1); hash.write(self.field2) etc, the derive macro is good in this regard.

Hashing the serialized string seems a bit complicated if you want to compare them for equality. Have you considered to implement Hash or PartialEq on your Setting but simply skipping those fields? You can even derive that with crates, e.g., derivative.

Regarding a solution which only uses serde: There is no generic context to attach such information for Serialize. You only have the object to serialize, its Serialize implementation, and the Serializer.

  • You could add a boolean to the struct which indicates if all fields should be serialized. This will not work with skip_serializing_if so it would need a custom Serialize implementation.

  • Carry a generic type which indicates if the fields should be serialized. Something like this:

    #[derive(Default, Serialize)]
    struct SomeSettings<T: SkipField> {
        #[serde(skip)]
        selected: bool,
        thing: bool,
        another: i64,
        more: SubSettings<T>,
    }
    
    #[derive(Default, Serialize)]
    struct SubSettings<T: SkipField> {
        #[serde(skip)]
        selected: bool,
        more: i64,
        #[serde(skip_serializing_if = "T::skip_field")]
        examples: String,
        #[serde(skip)]
        _unused: PhantomData<T>,
    }
    
  • Use a Serialize implementation which accesses "global" data, e.g., thread locals.

  • Use a Serializer which understands which fields should be skipped (e.g., based on a "secret" prefix) and customize the Serializer as desired.

This is the solution I would pick, to read from a thread local variable. It is also the least intrusive way.

Interesting, derivative looks like it might be the neatest solution. Using #[derivative(Hash="ignore")] to skip the not-for-cache-calculation fields should do what I need, and is nicely disconnected from the serialization path

If that doesn't work out I'll investigate the thread-local approach

Thanks so much for the excellent response!
- Ben

Ah. I tried out derivative to implement Hash and quickly remembered why I originally went down the "hashing serde output" path: the structs contain a lot of float data, and f32/f64 don't implement Hash (as per this thread).

Using the serialized outputs avoids this as the floats gets turned into a series of bytes, glossing over all the complexities of NaN's etc (which for my use-case is unimportant)

This is roughtly what I ended up with:

use serde_derive; // 1.0.126
use serde_derive::Serialize;

use std::cell::RefCell;
std::thread_local! {
    /// Global variable to skip certain fields when computing hash
    static IS_CALC_HASH: RefCell<bool> = RefCell::new(false);
}


impl SomeSettings {
    fn is_calculating_hash<T>(_: T) -> bool {
        IS_CALC_HASH.with(|f| *f.borrow())
    }
}


#[derive(Default, Serialize)]
struct SomeSettings {
    #[serde(skip)]
    selected: bool,
    thing: bool,
    another: i64,
    #[serde(skip_serializing_if = "SomeSettings::is_calculating_hash")]
    more: SubSettings,
}



#[derive(Default, Serialize)]
struct SubSettings {
    #[serde(skip)]
    selected: bool,
    more: i64,
    examples: String,
}


fn main() {
    let settings = SomeSettings::default();
    IS_CALC_HASH.with(|f| *f.borrow_mut() = true);
    let data = serde_json::to_string(&settings);
    IS_CALC_HASH.with(|f| *f.borrow_mut() = false);

    dbg!(data.unwrap());
}

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.