Why is serde skipping/ignoring attributes during tests?

[rainyday]

I'm working on a web application that, for legacy reasons, needs to accept timestamps in a couple different formats. I've accomplished this with the following:

#[derive(Default, Debug, Serialize, Deserialize, Validate)]
#[serde(rename_all = "camelCase", default)]
pub struct PacketFilter {
    #[serde(deserialize_with = "timestamp_from_map_or_number")]
    pub start: Timestamp,
    #[serde(deserialize_with = "timestamp_from_map_or_number")]
    pub end: Timestamp,
    // other optional fields that aren't important
}

#[derive(Debug, Default, Copy, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Timestamp {
    pub sec: u32,
    pub nsec: u32,
}
// ...
fn timestamp_from_map_or_number<'de, D>(deserializer: D) -> Result<Timestamp, D::Error>
where
    D: Deserializer<'de>,
{
    struct TimestampVisitor;

    fn invalid_timestamp<E>() -> E
    where
        E: de::Error,
    {
        E::custom("Timestamp invalid or out of range")
    }

    impl<'de> Visitor<'de> for TimestampVisitor {
        type Value = Timestamp;

        fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
            formatter.write_str("float, int or map")
        }

        fn visit_f64<E>(self, value: f64) -> Result<Self::Value, E>
        where
            E: de::Error,
        {
            let sec = value.trunc().to_u32().ok_or_else(invalid_timestamp)?;
            let nsec = (value.fract() * 1_000_000_000f64)
                .to_u32()
                .ok_or_else(invalid_timestamp)?;

            Ok(Self::Value { sec, nsec })
        }

        fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>
        where
            E: de::Error,
        {
            let sec = value
                .checked_div(1_000_000_000)
                .and_then(|s| s.to_u32())
                .ok_or_else(invalid_timestamp)?;
            let nsec = (value % 1_000_000_000).to_u32().ok_or_else(invalid_timestamp)?;
            Ok(Self::Value { sec, nsec })
        }

        fn visit_map<M>(self, map: M) -> Result<Timestamp, M::Error>
        where
            M: MapAccess<'de>,
        {
            Deserialize::deserialize(de::value::MapAccessDeserializer::new(map))
        }
    }

    deserializer.deserialize_any(TimestampVisitor)
}

This is working perfectly well in practice when running my application (rocket), but I want to have some tests to avoid breakage in the future:

mod tests {
    use super::*;

    #[test]
    fn filter_timestamp_deserialize() {
        let filter: PacketFilter = serde_json::from_str(
            r#"{
                "start": 1616114119.5192053,
                "end": 1616114119519205272
            }"#,
        )
        .unwrap();
    }
}

But this test fails with

failures:

---- filter::tests::filter_timestamp_deserialize stdout ----
thread 'filter::tests::filter_timestamp_deserialize' panicked at 'called `Result::unwrap()` on an `Err` value: Error("missing field `sec`", line: 1, column: 29)', lib/pcap-process/src/filter.rs:303:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This exact same input works totally fine when actually running the app. It seems that the call to serde_json::from_str is ignoring my deserialize_with attribute. What gives?

[mbrubeck]

This test passes for me: Playground

Can you provide a reproducible test case?

[rainyday]

Played around with some more and it seems to be caused by enabling the arbitrary_precision feature of serde_json. If I remove that it works. Still not sure why it causes the test to fail but not anything else.

[jonasbb]

The arbitrary_precision feature changes how numbers can be treated. You can basically treat any number as a newtype struct around a string. So what happens is that the visit_map function is being called and you get something like this, which does not have a sec or nsec field.

Map(
    {
        String(
            "$serde_json::private::Number",
        ): String(
            "1616114119.5192053",
        ),
    },
)

You need to change this line deserializer.deserialize_any(TimestampVisitor). Either use deserialize_f64 or deserialize_u64. Both seem to work, since they now favor a numeric type such that the code will not call visit_map anymore.