So I'm running into a really annoying, and probably incredibly stupid compile issue based on a lifetime error. The code I'm working with is a simple serialization test helper function that serialized, deserializes and then ensures no changes occurred:
fn validate<'de, E>(filename: &str, dto: E)
where E: Serialize + Deserialize<'de> + Debug + PartialEq
{
let ser = serde_json::to_string(&dto).unwrap().clone();
let des: E = serde_json::from_str(ser.as_str()).unwrap();
assert_eq!(des, dto);
}
And the error I see is:
39 | fn validate<'de, E>(filename: &str, dto: E)
| --- lifetime `'de` defined here
...
43 | let des: E = serde_json::from_str(ser.as_str()).unwrap();
| ---------------------^^^----------
| | |
| | borrowed value does not live long enough
| argument requires that `ser` is borrowed for `'de`
44 | assert_eq!(des, dto);
45 | }
| - `ser` dropped here while still borrowed
Okay, cool. The error makes sense. But how do I drop the use of des before reaching the end of the function? I assumed that des would be dropped before ser since they are initialized in that order.
I've seen this problem before and hacked a way around it but I'd like to understand it (as well as the correct fix) a bit better if anyone can offer some insight.
This is obviously a fundamental misunderstanding on my part about the implications of a function lifetime declaration. I realize this. Yet, I have no idea what that fundamental misunderstanding is.
= - (
fn validate<E>(filename: &str, dto: E)
where E: Serialize + for<'de> Deserialize<'de> + Debug + PartialEq
{
let ser = serde_json::to_string(&dto).unwrap().clone();
let des: E = serde_json::from_str(ser.as_str()).unwrap();
assert_eq!(des, dto);
}
Which is the same as
fn validate<E>(filename: &str, dto: E)
where E: Serialize + DeserializeOwned + Debug + PartialEq
{
let ser = serde_json::to_string(&dto).unwrap().clone();
let des: E = serde_json::from_str(ser.as_str()).unwrap();
assert_eq!(des, dto);
}
The issue with your code is as follows
You provided a caller chosen lifetime parameter 'de
You specify that the input of the deserialize will live for at least 'de with the Deserialize<'de> bound
You pass in an input that has a shorter lifetime than 'de (ser.as_str())
Things go boom
The fix is this magic looking for<'de> ... thing. This is a higher rank lifetime bound (HRLB), this is closely related to higher rank type bounds (HRTB). This says, for any choice lifetime parameter 'de, this bound holds. Now we are not bound to any particular lifetime, so we can use any input lifetime we want. DeserializeOwned is just a convenient alias for the HRLB
FWIW it's not entirely stupid that Rust doesn't allow that code to compile. It is preventing a real use-after-free memory safety violation here which would have caused a segfault or security vulnerability.
The code below shows how things go wrong with your original signature.
Bad implements Deserialize<'static> so we instantiate validate::<'static, Bad>
The Deserialize impl deserializes a &'static str and stashes it in a static
validate destroys ser when it goes out of scope
main prints the &'static str which is a dangling pointer into the original ser
use lazy_static::lazy_static;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use std::fmt::Debug;
use std::sync::Mutex;
fn validate<'de, E>(dto: E)
where
E: Serialize + Deserialize<'de> + Debug + PartialEq,
{
/* as above */
}
lazy_static! {
static ref HACK: Mutex<Option<&'static str>> = Mutex::new(None);
}
#[derive(Debug, PartialEq)]
struct Bad;
impl Serialize for Bad {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
serializer.serialize_str(".............................")
}
}
impl Deserialize<'static> for Bad {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'static>,
{
let s: &'static str = Deserialize::deserialize(deserializer)?;
*HACK.lock().unwrap() = Some(s);
Ok(Bad)
}
}
fn main() {
validate::<Bad>(Bad);
println!("{:?}", HACK.lock().unwrap().unwrap());
}
Now I'm back to my original issue which I ended up with after moving from a String to a &'a str inside my payload:
#[derive(Debug, Serialize, Deserialize, PartialEq)]
struct TestValidate<'a> {
pub name: &'a str,
}
#[test]
fn test() {
validate(TestValidate{name: "a name"});
}
fn validate<'a, E>(dto: E)
where for <'de> E: 'a + Serialize + Deserialize<'de> + Debug + PartialEq
{
let ser = serde_json::to_string(&dto).unwrap().clone();
let des: E = serde_json::from_str(ser.as_str()).unwrap();
assert_eq!(des, dto);
}
46 | validate(TestValidate{name: "a name"});
| ^^^^^^^^
|
= note: Due to a where-clause on serialization_tests::test_serialize::validate,
= note: serialization_tests::test_serialize::TestValidate<'_> must implement aggregate::_IMPL_DESERIALIZE_FOR_ProjectId::_serde::Deserialize<'0>, for any lifetime '0
= note: but serialization_tests::test_serialize::TestValidate<'_> actually implements aggregate::_IMPL_DESERIALIZE_FOR_ProjectId::_serde::Deserialize<'1>, for some specific lifetime '1
Again, I don't think that this should be difficult, but... there I am.
Change TestValidate to use a String instead of a &'a str, this way the DeserializeOwned bound is valid. Right now serde is trying a zero-copy solution because you asked for it, but that doesn't work in this case. There is now way to write the bounds to specify the lifetime of ser.as_str(), for<'de> .. is an overapproximation to get around this.
Take a &'de mut String input
fn validate<'de, E>(dto: E,temp: &'de mut String)
where E: Serialize + Deserialize<'de> + Debug + PartialEq
{
let ser = serde_json::to_string(&dto).unwrap().clone();
*temp = ser;
let des: E = serde_json::from_str(temp.as_str()).unwrap();
assert_eq!(des, dto);
}
this works because now the input lifetime for deserialize has the specified 'de lifetime. This does require you to construct a string on the outside, but you could call this like so validate(dto, &mut String::new()).
This solution is harder to use, because that 'de lifetime will infect almost every it touches. I would just use String instead of &'a str until you run into a perf issue.
Ahh, so I'm just prematurely optimizing. A lot of this is me trying to clear out a lot of clone() and to_string() calls from my code now that I'm a little more comfortable with the language.
Thanks for your help @RustyYato (especially the pointer to HRTBs which I've somehow completely missed up to this point)!
One way to clear out clone calls without going completely into references is to use Cow<'a, str>. You could use that instead of String or &'a str. Note that Cow will deserialize into it's owned variant unless you apply the #[serde(borrow)] attribute.