Serde's serialize_with and deserialize_with for vectors and slices

[lovesh]

The structs in my crate are generic over associated types from another crate that don't implement serde. Eg

struct Foo<F: SomeExtTrait> {
     a: F,
     b: F,
}

Since F comes from another trait, i used attributes serialize_with and deserialize_with and i can check that both serialization and de-serialization work for

struct Foo<F: SomeExtTrait> {
     #[serde(serialize_with = "to_bytes", deserialize_with = "from_bytes")]
     a: F,
     #[serde(serialize_with = "to_bytes", deserialize_with = "from_bytes")]
     b: F,
}

where to_bytes and from_bytes can convert F to and from bytes respectively. I get this serialization
{"a":[190,8, ...], "b": [82,229, ...]} when using serde_json for above.

Now i want to serialize the following

struct Bar<F: SomeExtTrait> {
     a: F,
     b: [F; 2],
     c: Vec<F>
}

I intend to serialize each F into separate byte array so using serde-json for above, i want to get
{"a":[190,8, ...], "b": [[82,229, ...], [206,46]], "c": [[29,1,..], [143,90,..], [78,190..], ... ]}

But I don't see a way to serialize the above without implementing Serialize for the struct? Is there a way where I can use serialize_with attribute

I was looking at implementing a version of to_bytes that takes a vector and was thinking of using serialize_seq but suck on the following.

pub fn f_vec_to_bytes_vec<S, F>(elems: &Vec<F>, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer
{
    let mut seq = serializer.serialize_seq(Some(elems.len()))?;
    for i in 0..elems.len() {
        // The following doesn't work as I have not defined Serialize trait for F
        // seq.serialize_element(&elems[i])?;
    }
    seq.end();
}

[cole-miller]

I would suggest a generic newtype that implements Serialize and Deserialize:

#[derive(Serialize, Deserialize)]
#[serde(transparent, bound = "F: SomeExtTrait")]
struct Wrapper<F>(
    #[serde(serialize_with = "to_bytes", deserialize_with = "from_bytes")]
    F,
);

Now you can do

#[derive(Serialize, Deserialize)]
struct Foo<F> {
    a: Wrapper<F>,
    b: Wrapper<F>,
}

#[derive(Serialize, Deserialize)]
struct Bar<F> {
    a: Wrapper<F>,
    b: [Wrapper<F>; 2],
    c: Vec<Wrapper<F>>,
}

[lovesh]

Thanks @cole-miller. I was avoiding creating a wrapper as i have a lot of occurrences of F and I didn't want to replace all of those with this wrapper; I understand that there would be no performance cost of this. Secondly, i enable serde only when a specific feature is enabled, so having this wrapper would mean I need to have 2 declarations of structs Foo, one containing the wrapper type and enabled when serde feature is enabled and other declaration without the wrapper type.

[cole-miller]

@lovesh

Secondly, i enable serde only when a specific feature is enabled, so having this wrapper would mean I need to have 2 declarations of structs Foo , one containing the wrapper type and enabled when serde feature is enabled and other declaration without the wrapper type.

You don't need the wrapper-less versions since you can always get F from Wrapper<F>. You might run into problems conditionally deriving Serialize and Deserialize only if your use-serde feature is enabled; ideally you could just cfg_attr(feature = "use-serde", ...) everything, but I believe the compiler will still try unconditionally to parse the serde(...) attributes and fail if the dependency isn't there (people keep having exactly this problem with PyO3). So you might have to use a workaround like

#[cfg(feature = "use-serde")]
mod with_serde {
   // Wrapper declaration from before goes here
}

#[cfg(feature = "use-serde")]
pub use self::with_serde::Wrapper;

#[cfg(not(feature = "use-serde"))]
mod without_serde {
    // declaration without serde stuff
    pub struct Wrapper<F>(F);
}

#[cfg(not(feature = "use-serde"))]
pub use self::without_serde::Wrapper;

As for replacing all the fields, you should be able to do that pretty mechanically with your text editor or IDE. I don't know if there's a solution that doesn't involve a wrapper newtype.

[lovesh]

I can get F from Wrapper but I still need 2 versions as I was aiming for a code, that the caller, if was not using serde feature, will not see Wrapper

Say i have a function that returns Bar, i will have the following structs and the function

#[cfg(feature = "serde")]
#[derive(Serialize, Deserialize)]
struct Bar<F> {
    a: Wrapper<F>,
    b: [Wrapper<F>; 2],
    c: Vec<Wrapper<F>>,
}

#[cfg(not(feature = "serde"))]
struct Bar<F: SomeExtTrait> {
     a: F,
     b: [F; 2],
     c: Vec<F>
}

fn ret_bar() -> Bar<F> {
     // .....
     // a, b and c are of type F    

    #[cfg(feature = "serde")]
    Bar {
        a: Wrapper(a),
        b: Wrapper(b),
        c: Wrapper(c),
    }

    #[cfg(not(feature = "serde"))]
     Bar {
        a,
        b,
        c,
    }
}

Similarly, feature gates will become more pervasive when i have a function that takes Foo and returns Bar.

[jonasbb]

You could use the serde_with crate which makes it easier to apply the with attribute inside of container types. If you already have the to_bytes and from_bytes functions, the hard part is already done and you can just follow the steps from the link.
Additionally, the crate already implements a lot of conversions, such that you might get lucky and already find what you need.

@lovesh

Now i want to serialize the following

struct Bar<F: SomeExtTrait> {
     a: F,
     b: [F; 2],
     c: Vec<F>
}

You could turn this into something like:

struct Bytes;

impl SerializeAs for Bytes {
    // ...
}

#[serde_as]
struct Bar<F: SomeExtTrait> {
    #[serde_as(as = "Bytes")]
     a: F,
    #[serde_as(as = "[Bytes; 2]")]
     b: [F; 2],
    #[serde_as(as = "Vec<Bytes>")]
     c: Vec<F>
}

[lovesh]

Thanks @jonasbb This is exactly I was looking for. Is there a way to use the serde_as attribute it with newtype and tuple structs as well?

[jonasbb]

Yes, you can use serde_as also with newtypes, tuple structs and enums. The syntax is similar to the serde attribtues.

#[serde_as]
struct Newtype(#[serde_as(...)] i32);

[lovesh]

@jonasbb

#[serde_as(...)]

It works. Thanks again for your help.

[lovesh]

@jonasbb I have used serde_with to serialize most of the structures successfully but now when serializing a generic struct whose members i have already serialized with serde_as macro, i get errors like

^^^ the trait `Serialize` is not implemented for Type
note: required because of the requirements on the impl of `Serialize` for StructMember

Say I have a struct Foo that i have serialized as below

#[serde_as]
#[derive(
    Clone, PartialEq, Eq, Debug, Serialize, Deserialize,
)]
pub struct Foo<F: T> {
    #[serde_as(as = "TtoBytes")]
    a: F,
    #[serde_as(as = "TtoBytes")]
    b: F,
}

Now i have another struct Bar as below and that also compiles

#[serde_as]
#[derive(
      Clone, PartialEq, Eq, Debug, Serialize, Deserialize,
  )]
pub struct Bar<F: T>(pub Foo<F>);

But when i compile the struct Baz below, i get compile errors

#[serde_as]
#[derive(Clone, PartialEq, Eq, Debug, Serialize, Deserialize)]
pub struct Baz<E: SuperT> {
    pub x: Bar<E::T>,
}

Errors

pub x: Bar<E::T>,
     |     ^^^ the trait `Serialize` is not implemented for `<E as SuperT>::T`
     |
note: required because of the requirements on the impl of `Serialize` for `Bar<<E as SuperT>::T>`

I have tried to make the above example match my exact code, but here is my actual code, just in case. I get compile error for MembershipProof here.

326  |     pub randomized_witness: MembershipRandomizedWitness<E::G1Affine>,
     |     ^^^ the trait `Serialize` is not implemented for `<E as PairingEngine>::G1Affine`
     |
note: required because of the requirements on the impl of `Serialize` for `MembershipRandomizedWitness<<E as PairingEngine>::G1Affine>`
    --> vb_accumulator/src/proofs.rs:297:76
     |
297  |     Clone, PartialEq, Eq, Debug, CanonicalSerialize, CanonicalDeserialize, Serialize, Deserialize,
     |                                                                            ^^^^^^^^^
298  | )]
299  | pub struct MembershipRandomizedWitness<G: AffineCurve>(pub RandomizedWitness<G>);
     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: required by `batch_utils::_::_serde::ser::SerializeStruct::serialize_field`
    --> /home/lovesh/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/ser/mod.rs:1893:5
     |
1893 | /     fn serialize_field<T: ?Sized>(
1894 | |         &mut self,
1895 | |         key: &'static str,
1896 | |         value: &T,
1897 | |     ) -> Result<(), Self::Error>
1898 | |     where
1899 | |         T: Serialize;
     | |_____________________^
     = note: this error originates in the derive macro `Serialize` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `<E as PairingEngine>::Fr: Serialize` is not satisfied
    --> vb_accumulator/src/proofs.rs:328:5
     |
328  |     pub schnorr_response: MembershipSchnorrResponse<E::Fr>,
     |     ^^^ the trait `Serialize` is not implemented for `<E as PairingEngine>::Fr`
     |
note: required because of the requirements on the impl of `Serialize` for `MembershipSchnorrResponse<<E as PairingEngine>::Fr>`
    --> vb_accumulator/src/proofs.rs:318:76

Structs MembershipRandomizedWitness and RandomizedWitness compile (checked it by omitting serialization macros for MembershipProof).

But I have a similar situation where it works. PoKOfSignatureG1Protocol serializes and it contains SchnorrCommitment.

[cole-miller]

Sorry if I'm beating a dead horse here, but I realized that you can modify the "wrapper" approach so that callers who don't use the serde feature don't have to modify their code. Like this:

#[cfg(feature = "use-serde")]
mod with_serde {
    use serde::{Deserialize, Serialize};

    #[derive(Serialize, Deserialize)]
    #[serde(transparent, bound = "F: SomeExtTrait")]
    pub struct Wrapper<F>(#[serde(serialize_with = "to_bytes", deserialize_with = "from_bytes")] F);
}
#[cfg(feature = "use-serde")]
pub use self::with_serde::Wrapper;

#[cfg(not(feature = "use-serde"))]
pub type Wrapper<F> = F;

#[cfg(feature = "use-serde")]
use serde::{Deserialize, Serialize};

#[cfg_attr(feature = "use-serde", derive(Serialize, Deserialize))]
pub struct Foo<F: SomeExtTrait> {
    pub a: Wrapper<F>,
    pub b: Wrapper<F>,
}

#[cfg_attr(feature = "use-serde", derive(Serialize, Deserialize))]
pub struct Bar<F: SomeExtTrait> {
    pub a: Wrapper<F>,
    pub b: [Wrapper<F>; 2],
    pub c: Vec<Wrapper<F>>,
}

If use-serde isn't enabled, then Wrapper<F> is just a type alias for F, so callers can ignore it entirely. It still appears in signatures and generated documentation, but has no effect on the code itself.

[jonasbb]

@lovesh

But when i compile the struct Baz below, i get compile errors

#[serde_as]
#[derive(Clone, PartialEq, Eq, Debug, Serialize, Deserialize)]
pub struct Baz<E: SuperT> {
    pub x: Bar<E::T>,
}

Errors

pub x: Bar<E::T>,
     |     ^^^ the trait `Serialize` is not implemented for `<E as SuperT>::T`
     |
note: required because of the requirements on the impl of `Serialize` for `Bar<<E as SuperT>::T>`

This is a limitation in how the serde derive bounds trait implementation for generic types. The Serialize implementation looks something like:

impl Serialize for Baz
where
    E: Serialize
{ ... }

but what is needed here is either a Bar<E::T>: Serialize or a E::T: Serialize bound. serde has a bound attribute which you can use to force serde to use the correct trait bounds. It exists as a container attribute and as a field attribute.

[lovesh]

Thanks @cole-miller . This is good. I will go with it if I reach a dead end with my current approach