To_owned producing a non-assignable?

help
1

Here I have an struct with some references to borrowed items, and I want to put a copy of it in a HashMap, using insert. So I use to_owned, which ought to give me a clean ownable copy I can pass to HashMap. That should just work; I paid for an allocation and a copy. But the compiler complains the result of to_owned is not assignable. How can it not be? That's the whole point of to_owned, right? " The ToOwned trait generalizes Clone to construct owned data from any borrow of a given type."

The error message also complains that types don't match and yet lists "expected" and "found" as identical.

The HashMap involved is defined by

pub waiting_for_handle: HashMap<LLUUID, PendingHandle<'a>>,

and here's the error:

error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
   --> src/viewer/world.rs:96:15
    |
96  |         match ms {
    |               ^^
    |
note: first, the lifetime cannot outlive the anonymous lifetime #3 defined on the method body at 94:5...
   --> src/viewer/world.rs:94:5
    |
94  |     pub fn handlemessage(&self, ms: &AllMessages) -> Result<(), StrError> {                   // incoming message
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
   --> src/viewer/world.rs:96:15
    |
96  |         match ms {
    |               ^^
    = note: expected `&generateddefs::AllMessages<'_>`
               found `&generateddefs::AllMessages<'_>`
note: but, the lifetime must be valid for the lifetime `'_` as defined on the impl at 75:12...
   --> src/viewer/world.rs:75:12
    |
75  | impl World<'_> {
    |            ^^
note: ...so that the expression is assignable
   --> src/viewer/world.rs:138:68
    |
138 | ...   self.pending.waiting_for_handle.insert(id, PendingHandle{timestamp: timestamp(), handshake: ownedhandshake}.to_owned()); 
    |                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = note: expected `PendingHandle<'_>`
               found `PendingHandle<'_>`
2

The generic implementation of to_owned for a Clone type will simply clone the reference that you pass it, and return a new value of the same type.

This is useful for changing a borrowed reference (&PendingHandle<'a>) into a new struct (PendingHandle<'a>) but it doesn't change the lifetime of the reference inside the type: For example, it can't change a PendingHandle<'a> into a PendingHandle<'static>. You'll need to write your own method to do that.

3

How do you get a deep copy when you need one, then?

Clone is supposed to be a deep copy, except for reference counted item, which are shared. When you #[Derive(Clone)], every field of the struct involved must itself be cloneable. So the machinery is there to do that. A shallow to_owned is not very useful.

4

For a type that maybe owned or borrowed, typically you would use Cow to replace the reference. Cow implements to_owned, also a Cow that holds owned data can have a 'static lifetime. But pretty much all references with non 'static lifetime in your type would need to be wrapped with Cow including sub types, so it's Cows all the way down.

Alternatively if the types are read only you could use Rc or Arc instead of references. Which are generally cheaper to clone, they aren't deep copies per se but are often close enough.

5

I could do Cow all the way down, but would it help?

I'm generating a big tree on the stack as part of un-marshalling a complex foreign format. Usually I use the tree once and discard it, but sometimes I need to save a copy of it. So I need a deep copy to get it onto the heap and give its ownership to a collection. Copy-on-write might help, if that would make Clone or to_owner copy non-mutable references. Does it?

All that Clone generation machinery, and no deep copy?

Here's what I'm doing, in more detail I have lots of data items that look like this:

pub struct TransferPacket<'a> {
    pub transfer_id: LLUUID,
    pub channel_type: i32,
    pub packet: i32,
    pub status: i32,
    pub data: &'a [u8],
}

I have 470 different message formats like that. (That, and the decoding code, is all generated by a Rust program in "build", so I can change that if needed.)

These were un-marshalled from a byte array, and some of them have fields, like "data" above, which are references to slices of the input array. The variable length data hasn't been copied yet; it may be big and I want to avoid copying it unless necessary. Sometimes, though, I have to queue such structs for later processing. So that's when I want to copy them to the heap, including the "data" field, with ownership. I thought "Clone" would do that. But apparently not. I don't need to write them; they're immutable. So "Cow" may not help here.

I'm just amazed that deep copy isn't a standard primitive in Rust. That's what "to_owner" ought to do. If you don't get something with a longer lifetime, what's the point of to_owner?

6

It's hard to say what the best solution is without seeing the code. But clone typically gives you a owned copied of the same type. If that type has references so must the clone (since it's the same type). Either you need output a new type that doesn't have references which is what into_owned on Cow does (Sorry I confused that with to_owned which it doesn't have). Or instead of references you use something like Cow which can represent the owned form with 'static lifetimes. I think making a new type is probably simpler to reason about if the alternative is multiple layers of Cow.

7

What you're encountering here is familiar to me in the context of writing parsers, in which you're walking all over borrowed data, but if you want to return an error or result which outlives that data, you have to copy it elsewhere. The only thing I've found to work is to have Data and DataOwned variants of everything that might need to be sustained, and an into_owned method to convert it over.

If you don't get something with a longer lifetime, what's the point of to_owner?

A lifetime parameter on an object is a more general concept than a reference. It means the object is carrying around a validity constraint. A reference, on the other hand, while also only valid for some lifetime, does not constrain the range of validity of the object it points to, only itself. So you can convert &T to T without knowing anything about T (this is just dereferencing it), but you cannot do the same for T<'a> and get T<'static>: T may not be valid for that long. In other words, you can't generally copy T<'a> into T<'b> where 'b: 'a because you don't know why T has that constraint.

For a more practical example, imagine that the 'a is used to constrain a type which references executable data in a dynamically loaded library. What should a deep clone do? Copy the whole library into heap?

So the answer to this question:

How do you get a deep copy when you need one, then?

... is that you have to write code to specify how to do the copy yourself.

1 Like
8

Writing a deep copy for an object with references would not help, because the output would have the same references and borrow problems. It has to be copied to a different type to be truly cloneable. A derive macro to generate a "flattened" version of a type might be useful.

This should be documented better. The to_owner() and clone() methods appear to deal with this, and some sources indicate they do. But they can't. That's implicit in the type system, but non-obvious.

9

Even if you don't need to write them, you want to hold either a borrowed version or an owned version of them, and that's what Cow is usually used for.

That's because there's no way to deep copy a struct holding a reference. Who will own the deep copied data? The struct itself only holds a reference which implies it doesn't own that data! Cow solves this problem by beign either a reference or some owned data.

Mainly the point of ToOwned is to extend Clone to unsized types. This allows for example to have Cow<'_, str> and Cow<'_, [T]>.

Note that similar to your case, the ToOwned implementation for [T] can't and doesn't return a [T] or &[T] because that's not possible. Instead it returns a Vec<T>. This is another way you could solve your problem: create another struct that owns the data and a method to convert from borrowing one to the owning one.

Care to explain what those sources are? To me it's pretty clear that Clone goes from &T to T (not your case) and ToOwned extends Clone allowing to return a different type (i.e from &B to O, still not your case) , but nowhere it's said they change the fields properties. Implementing traits won't magically change your types.

10

It is, you are just holding it wrong.

Clone performs a deep clone by convention (and accordingly, when #[derive]d). Likewise, ToOwned converts from borrowed representations to owned representations, more precisely, it attempts to remove a level of indirection.

However, you need to make sure your data structures are owned in order for that to work in the first place. In Rust, you are expected to work with values as the source of truth, and then you provide temporary views into that owned data when needed. It seems that you are trying to do the opposite for some reason – defining references first, and then expecting owned data to come out of that.

If you are actively trying to program against the idioms of the language, it is no wonder you hit a wall sooner rather than later.

1 Like
11

That's an good point. So I've come up with an all-owned solution. Instead of using references back to the input data array, I'm using Smallvec. Once I have a reference to the desired bytes in v, I
use

    let sv: SmallVec<[u8;SMALLVECSIZE]> = v.to_smallvec();

to make it into an ownable item. It's still on stack, and there's no allocation unless v.len() > SMALLVECSIZE. Just a copy, of, usually, a few bytes. That beats doing millions of heap allocations of 5-10 bytes and discarding most of them within a millisecond.

A struct containing a SmallVec is cloneable, so if I need a copy, I can now make one.

12

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.