Generic mutability parameters

[tobz1000]

After some searching, I haven't found much discussion about the possibility of mutability parameters in Rust. By this, I mean the ability to make a structure/function/implementation generic over whether one of its inputs is mutable or not.

To me, it seems like something which could work similarly to lifetime parameters. The biggest benefit of this kind of genericity would be the avoidance of duplicate getter functions. For example:

impl MyStruct {
    fn get_n(&self) -> &i32 {
        &self.n
    }

    fn get_n_mut(&mut self) -> &mut i32 {
        &mut self.n
    }
}

Could be replaced with (syntax of course up for debate):

impl<mut m> MyStruct<mut m> {
    fn get_n(&m self) -> &m i32 {
        &m self.n
    }
}

With parameters determined by function arguments, like types and lifetimes:

let s_imm = MyStruct::new();
let mut s_mut = MyStruct::new();

let i: &i32 = s_mut.get_n(); // Ok
let i: &i32 = s_imm.get_n(); // Ok
let i: &mut i32 = s_mut.get_n(); // Ok
let i: &mut i32 = s_imm.get_n(); // Compilation error

Has this already been discussed/RFC'd? Is there another way to accomplish this (besides macros)?

The closest I've found is "Abstracting over mutability in Rust", which mentions the use of AsRef to specify the base type of a parameter, without specifying its reference type (and therefore its mutability). However it looks like this doesn't allow you to retain the mutability of the type, since .as_ref() will coerce mutable to immutable.

[vitalyd]

I'm not sure if there's been discussion about a parameter that indicates the mutability. Immutable and mutable refs are very different fundamentally, with knock-on effects on surrounding code/types, so this would need to be considered very carefully, not only for technical details but also ergonomics/maintainability of code.

AsRef/AsMut/Borrow/BorrowMut are mostly about abstracting over ownership, I don't think they abstract over mutability. For instance:

use std::borrow::BorrowMut;

fn foo<T: BorrowMut<String>>(mut t: T) {
    // can work with a mutable reference
    *t.borrow_mut() = "replaced".into();
    // can also work with an immutable reference (BorrowMut has Borrow as super trait)
    println!("{}", t.borrow());
}

fn main() {
    let mut s = "hi".to_string();
    // borrowed version
    foo(&mut s);
    // owned version
    foo("blah".to_string());
}

But this doesn't abstract over mutability, just ownership.

[tobz1000]

Sure - it would be a reasonably disruptive change, if just because so many traits already have foo() and foo_mut() variants. Seeing these migrated would be a long process.

Still, it seems like a strange omission, considering the similarities between mutability and lifetimes in terms of function signatures. Maybe no-one ever thought it was that big a deal to duplicate certain functions?

I'm considering writing an RFC to see whether the dev team thinks it is a worthwhile idea. Just wanted to check if it's already been considered.

[vitalyd]

@tobz1000

Still, it seems like a strange omission, considering the similarities between mutability and lifetimes in terms of function signatures. Maybe no-one ever thought it was that big a deal to duplicate certain functions?

Lifetimes and mutability aren’t really similar though. To wit, T, &T, and &mut T are 3 different types at the type system level. The type of reference isn’t some attribute of a type. In some sense, it’s like saying “I’d like get() to sometimes return a String and sometimes an i32”.

[tobz1000]

@vitalyd

T, &T, and &mut T are 3 different types at the type system level.

References and values, yes. But by my (limited) understanding, mutable and immutable references are only different for compilation checks. The generated byte code sees no difference (unless LLVM can use immutability guarantees for optimisations?). Therefore they are an abstract attribute of the reference, like lifetimes.

&'a i32 is a different type to &'b i32 in the same sense as mut and non-mut. If they're within the same function, one lifetime will be a superset of the other (or they'll be exactly the same). Similarly, two references with a hypothetical mutability parameter can be a subset and superset (i.e. &mut T is a superset of &T). The main difference is that there can be infinitely many lifetime values, but only two possible mutability values.

So, in terms of function signatures:

fn get_n<'a>(i: &'a Foo) -> &'a i32 { ... }
...
let n = get_n(&some_foo);

This implies that n's lifetime must be less or equal to 'a; i.e. it is a subset. Which has a requirement equivalence to:

fn get_n<mut m>(i: &m Foo) -> &m i32 { ... }
...
let n = get_n(&some_foo);
// or
let n = get_n(&mut some_foo);

In this second case, n's mutability must be a subset of mutability m (which either means they're the same, or 'm' is mutable and n is immutable).

The big difference is of course how the compiler chooses to validate them (how long the variable lives vs. whether it is mutated), given this implication.

In some sense, it’s like saying “I’d like get() to sometimes return a String and sometimes an i32”.

Isn't that what generics are for?

[vitalyd]

@tobz1000

References and values, yes. But by my (limited) understanding, mutable and immutable references are only different for compilation checks. The generated byte code sees no difference (unless LLVM can use immutability guarantees for optimisations?). Therefore they are an abstract attribute of the reference, like lifetimes.

No, all 3 are different types. For example, you may have noticed that you can impl a trait for a T, &T, and/or &mut T. You may also have noticed that &T is a Copy type, whereas &mut T is a move-only type. LLVM IR is a low level representation/mapping, which doesn’t necessarily carry the frontend language’s type system (it’s more of an abstract machine level IR, if anything).

@tobz1000

&'a i32 is a different type to &'b i32 in the same sense as mut and non-mut.

&'b i32 can be a subtype of &'a i32 if 'b outlives 'a, so these aren’t different types necessarily. For example, &'static i32 is a subtype (and thus substitutable) for any arbitrary/generic &'a i32.

If you haven’t yet, I recommend reading nomicon’s section on variance/subtyping and how mutability plays a role (eg makes things invariant).

@tobz1000

Isn’t that what generics are for?

Yes, if you can abstract over them which is what this thread is about with regards to mutability . My main point is they’re different types.

[tobz1000]

Good points.

@vitalyd

You may also have noticed that &T is a Copy type, whereas &mut T is a move-only type.

Yes - immutable references can be copied freely and mutable not so. But if you don't wish to actually mutate the referent, you can always choose to downcast a mutable ref to immutable, meaning that mutable is a superset (subtype) of immutable, at least in some regard. Rust's syntax doesn't necessarily frame it that way so obviously, but the fact that you can assign a &mut T to a &T-binding should be some indication of this.

If you haven’t yet, I recommend reading nomicon’s section on variance/subtyping and how mutability plays a role (eg makes things invariant).

Thanks, I will definitely check it out. Variance is always something of a brain-twister!

Yes, if you can abstract over them

Sure - my idea is only for the implementation of functions (/traits/structs) which would abstract over an input's mutability. This means that within the function, that type would have to be treated as immutable (the base type). But the caller would benefit from maintaining full type information of its input in the returned value, instead of it being downcast to immutable.

From the way mutability is treated in Rust, they are certainly more distinct than lifetimes. I imagine this is down to the fact that there are only two values, which appear everywhere, whereas lifetimes are an infinite number of "ephemeral" types, barely worth an individual definition.

While I'm not convinced that mutability is fundamentally "more" of a type-level distinction than lifetimes, I agree that each mutability-based type (i.e. mut and imm) is more well-defined, and the two attributes cannot be treated in the same way.

Maybe I'll feel differently after reading this chapter.

[vitalyd]

@tobz1000

But if you don’t wish to actually mutate the referent, you can always choose to downcast a mutable ref to immutable, meaning that mutable is a superset (subtype) of immutable, at least in some regard. Rust’s syntax doesn’t necessarily frame it that way so obviously, but the fact that you can assign a &mut T to a &T-binding should be some indication of this.

This is commonly referred to as “pointer weakening” - it’s a form of coercion to make it easier to work with Rust. There’s also something called reborrowing, which makes it look like &mut T is being copied but it’s not. Again, to make it easier to write normal code.

To give an idea of why mutable isn’t superset of immutable, despite the coercion making it look like that in some way, consider these two methods:

fn get(&mut self) -> &SomeType {...}

fn get(&self) -> &SomeType {...}

The mutable version, despite returning an immutable output, “locks” self - you cannot borrow it again until that returned reference is dead. The mutability “carries over”. This can get particularly interesting if you borrow self mutably for a lifetime parameter of the type - you get a value that cannot be borrowed again, period. So (in this example) &self and &mut self behave differently at a fundamental level.

Rust has other features that make it easier and more ergonomic, such as autoref and autoderef. But those things don’t change anything fundamental.

@tobz1000

While I’m not convinced that mutability is fundamentally “more” of a type-level distinction than lifetimes, I agree that each mutability-based type (i.e. mut and imm) is more well-defined, and the two attributes cannot be treated in the same way.

Lifetimes get erased after borrowck runs, but & vs &mut persists. You can’t impl a trait for &'a T and &'static T - they’re not different types at this level. So although (generic) lifetimes are part of the system, they’re sort of necessary appendages to the references but it’s the references that are the types.

[tobz1000]

@vitalyd

The mutable version, despite returning an immutable output, “locks” self - you cannot borrow it again until that returned reference is dead.

Yes - I should have been more clear when I said you can "choose to downcast a mutable ref to immutable". I meant as the receiver/owner of a new (or reborrowed?) reference, you can do this, regardless of the parent T/&T/&mut T. So if the source is a &mut T, then it will still be locked until the child &mut T is dead, but that child can be downcast to a &T. In fewer words:

fn get(&mut Foo) -> &mut Bar {...}
let mut foo: Foo = ...;
{
    let bar: &Bar = get(&mut foo); // Downcast on receive of new reference
    let baz = bar;
    println!("{} {}", baz, bar); // Ok
}
mutate(foo); // Unlocked

Your function signature fn get(&mut self) -> &SomeType {...} implies that this downcasting was already done somewhere inside the function.

The reason I believe this makes it a superset is because, with a &mut T, you have the choice of a) mutating/reborrowing once, or b) infinitely copying immutably (in the form of new &Ts). &T only lets you do b).

You can’t impl a trait for &'a T and &'static T

Because no lifetimes (other than 'static) exist at "design-time", this is true: lifetimes can only be mentioned in generic terms. However, I believe if a dynamic language (i.e. one where new functions are defined at runtime) had a concept of lifetimes, then they could be defined concretely within than runtime context.

All of the (two) enumerable values for mutability always exist, no matter the context. So we can mention them in concrete terms. However, I don't think this means they can't also be expressed generically.

I haven't quite finished the chapter on mutability/lifetime variance, but it's some good food for thought. I'll have to consider its consequences with respect to my proposal, especially for more complex use cases (e.g abstracting over the mutability of &[mut] &[mut] T...).

[jonh]

I do like intention of this topic. In terms of getting you thinking of what could be possible. My gut says it is wishful thinking. I also see it missing the Move case which often [but not always] comes with you have implementations for the other two.

@tobz1000

(unless LLVM can use immutability guarantees for optimisations?)

Yes (in theory). nomicon aliasing probably good explanation. (The are other not so good links to aliasing too else where.)

I personally see both lifetimes and mutability as metadata; this topic make me question whether I should be mentally splitting the data (i.e. pointer) as the entire data type from the metadata. (I still don't mentally like &T as Copy / &mut T not since any second &T lifetime is not tied to the first &T)

@tobz1000

Your function signature fn get(&mut self) -> &SomeType {...} implies that this downcasting was already done somewhere inside the function.

I think this is a case of bad terminology. The function will be dereferencing to generate the reference. Code maybe explains difference best;

    let mut i = 0;
    let m = &mut i;
    let _i1:&i32 = m;
    let _i2:&i32 = m;
    //let _i3 = &i; // does not work but would if m was immutable ref.

[vitalyd]

@jonh

I still don’t mentally like &T as Copy / &mut T not since any second &T lifetime is not tied to the first &T

The lifetime is not important here. What is important, like with any Copy type, is the original (source) is still live/valid/usable after a copy is taken.

[vitalyd]

Thinking about this some more, I think what @tobz1000 is proposing is not an abstraction over mutability. Not in the same sense that AsRef/Borrow and friends abstract over ownership - in their case, the user of these types truly doesn’t know whether the underlying type is borrowed or owned. But you can’t abstract over mutability because it’s a core concept and difference between the types. I’m not even sure what it would mean to abstract over mutability - either you need to mutate something or not, there’s no abstraction that I see.

Instead, what’s being proposed is closer to fn overloading and/or extensions to type inference where a single method can be used to obtain different mutability.

[comex]

CC @RalfJung, who in another thread proposed having a way to abstract over "reference modifiers" (i.e. mut and, in the future, potentially pin)...

[skade]

I wonder how prevalent this problem is outside of accessors? I generally have no issue with having double-accessors there.

By the way, the wanted pattern can be somewhat implemented with traits:

struct Target {
    attribute: i32
}

trait Attr {
    type Output;
    
    fn attribute(&self) -> &Self::Output;
}

trait AttrMut: Attr {
    fn attribute(&mut self) -> &mut Self::Output;
}

impl Attr for Target {
    type Output = i32;
    
    fn attribute(&self) -> &i32 {
        &self.attribute
    }
}

impl AttrMut for Target {
    fn attribute(&mut self) -> &mut i32 {
        &mut self.attribute
    }
}

fn one_way() {
    let mut target = Target { attribute: 0 };
    
    let x = (&mut target).attribute();
    
    *x = 42;
    
    let y = (&target).attribute(); // not okay
}

fn the_other() {
    let target = &mut Target { attribute: 0 };
    
    let x = target.attribute();
    
    *x = 42;
    
    let y : &i32 = target.attribute(); // not okay
}

[jonh]

@skade

wanted pattern

I was also thinking when I first read this post of a way that traits could do something similar.
The OP desire (in part*) is for code reuse so I see your code failing since you have written both bodies &self.attribute and &mut self.attribute
(* other part is not using a second _mut identifier.)

[tobz1000]

@jonh

I also see it missing the Move case which often [but not always] comes with you have implementations for the other two.

Do you mean as in the following?

fn get_move(self) -> T;

I hadn't thought about it until now, but it does make sense. If you provide an even less constrained input to such a function, you get an equivalently unconstrained output (full ownership). The resulting bytecode of the ownership variant of the function would be significantly different to the reference variant(s), but that isn't such a problem.

The main problem I see with this addition is that the generic function would be unable to take advantage of the moved variable, which might prevent some optimisations the programmer could otherwise make. For example:

struct Foo {
    bar: Bar;
    baz: Baz;
}

fn some_network_stuff(bar: Bar) { ... }

impl Foo {
    fn get(&self) -> &Baz {
        some_network_stuff(self.bar.clone()); // Have to clone here
        &self.baz
    }

    fn get_move(self) -> Baz {
        let Foo { bar, baz } = self;
        some_network_stuff(bar); // No clone necessary
        baz
    }
}

In a generic form of the function, the clone would occur even when the input Foo was being moved by the caller (at least without a very clever compiler optimisation).

---

@vitalyd

Thinking about this some more, I think what @tobz1000 is proposing is not an abstraction over mutability. Not in the same sense that AsRef/Borrow and friends abstract over ownership - in their case, the user of these types truly doesn’t know whether the underlying type is borrowed or owned.

It depends what you're referring to as "the user". If you mean the caller of a generic function, it will know the mutability state of the input/output.

However, the generic function itself will not know the mutability and therefore cannot mutate it. It can just say "you gave me this reference; I'll give you another reference derived from it. If you could mutate it before, you can still do that now. ¯\_(ツ)_/¯". This is abstraction over mutability from the point of view of the generic function.

@vitalyd

Instead, what’s being proposed is closer to fn overloading and/or extensions to type inference where a single method can be used to obtain different mutability.

It's not really overloading - the two versions of the generic function would behave the same. Overloaded functions can have arbitrarily different behaviour. (Generic functions are still turned into multiple real functions for each input type at compile time of course, but the programmer cannot affect their behaviours differently.)

[vitalyd]

@tobz1000

However, the generic function itself will not know the mutability and therefore cannot mutate it. It can just say “you gave me this reference; I’ll give you another reference derived from it. If you could mutate it before, you can still do that now. ¯_(ツ)_/¯”. This is abstraction over mutability from the point of view of the generic function.

I meant the generic function itself. I agree with @skade that your proposal only really helps the trivial case of simple accessors. A better example of where abstracting over it would be useful is this thread - it has two versions of a method, one for each reference type, but 99% of the code is the same between the two.

@tobz1000

It’s not really overloading - the two versions of the generic function would behave the same. Overloaded functions can have arbitrarily different behaviour. (Generic functions are still turned into multiple real functions for each input type at compile time of course, but the programmer cannot affect their behaviours differently.)

Right - it’s not really overloading but is closer to it than to abstraction from the fn implementation point of view. It’s actually closer to something like Into::into() where the target type is provided by the caller (explicitly or inferred). Your case is essentially that (at a high level) except mutability of the target is selected by caller.

As mentioned, the utility of this is very marginal if the fn itself cannot differ. What you’d really want is a way for the fn body to know the mutability and do things differently - caller sees a single function, they still get to pick resulting mutability but the fn itself also knows the context.

[tobz1000]

@skade

I wonder how prevalent this problem is outside of accessors? I generally have no issue with having double-accessors there.

For simple accessors it's not a problem, but still an unfortunate thorn to deal with - especially when polymorphism of other types is generally dealt with very well in Rust.

Ultimately, any function utilising generic mutability would be an accessor of some sort (possibly with side effects) - but some are more complex than others. A contrived example:

fn get_some_values(data: &Foo, iter: &mut Iterator<Item=SomeIndexer>) -> Vec<&Val> {
    let mut chosen_vals: Vec<&Val> = vec![];

    for i in iter { // Some sequence of index values
        let val = &data[i];

        // Add first-chosen value to `chosen_vals`, then insert subsequent vals
        // at a random place if some comparison between the currently-last val
        // succeeds
        if let Some(v) = data.last() {
            if some_comparison(v, val) {
                let insert_ind = rand(0, vals.len());
                vals.insert(insert_ind, val);
            }
        } else {
            vals.push(val);
        }
    }

    chosen_vals
}

Now imagine you need to also be able to receieve mutable output, where data is a &mut Foo.

You could minimise the repeated code by instead returning the chosen indexes from iter, instead of references directly. Then feed these index value into smaller getter functions:

fn get_some_values(data: &Foo, iter: &mut Iterator<Item=SomeIndexer>) -> Vec<&Val> {
    get_some_indices(data, iter).into_iter().map(|i| &data[i]).collect()
}

fn get_some_values_mut(data: &mut Foo, iter: &mut Iterator<Item=SomeIndexer>) -> Vec<&mut Val> {
    get_some_indices(data, iter).into_iter().map(|i| &mut data[i]).collect()
}

fn get_some_indices(data: &Foo, iter: &mut Iterator<Item=SomeIndexer>) -> Vec<Index> {
    let mut chosen_indices: Vec<&Val> = vec![];

    for i in iter {
        let val = &data[i];

        if let Some(v) = data.last() {
            if some_comparison(v, val) {
                let insert_ind = rand(0, vals.len());
                vals.insert(insert_ind, val);
            }
        } else {
            vals.push(i);
        }
    }

    chosen_indices
}

The final solution then:

• Allocates an intermediate Vec
• Performs indexing of data twice (which might be expensive)
• Involves some non-trivial new getter functions

With mutability generics, this would have been a one-line change with no performance impact.

@skade

By the way, the wanted pattern can be somewhat implemented with traits:

Good example. Yeah, I suppose traits can be used like this to achieve a kind of pseudo-overloading. @jonh's comment is correct however; my main reasoning behind this is to remove the need for the multitude of immut-then-mut traits and functions out there.

[vitalyd]

Vec<&Val> vs Vec<&mut Val> influences the types of Val references you can put into the Vec due to &mut changing variance. For instance, a certain Val type may not allow returning a &'static mut one and this influences code inside the fn as compiler would need to be conservative again.

[tobz1000]

@vitalyd

A better example of where abstracting over it would be useful is this thread - it has two versions of a method, one for each reference type, but 99% of the code is the same between the two.

This seems like a perfect example of when generic mutability would be helpful! It's not what I would call trivial, so I don't see how you conclude that it "only really helps the trivial case of simple accessors"?

Unless you mean that 1% difference is why overloading is necessary? The only difference is some return type/value modification (default value instead of Option) at the end of the function, which is nothing to do with dealing with mut/non-mut types; it's just an some extra functionality that the programmer added to one of the functions. With generics, the immutable-with-default-val case would just be:

self.get_wall(point, direction).unwrap_or(Wallyness::Wall)

@vitalyd

It’s actually closer to something like Into::into() where the target type is provided by the caller (explicitly or inferred). Your case is essentially that (at a high level) except mutability of the target is selected by caller.

The return type is determined by the input provided by the caller, and as such the caller must be able to receive a value with the matching mutability. This is just the same as a trait or lifetime parameter being included in the return type. But you could interpret that as the reverse: mutability determined by return type, and the caller must be able to provide a matching input.

@vitalyd

What you’d really want is a way for the fn body to know the mutability and do things differently - caller sees a single function, they still get to pick resulting mutability but the fn itself also knows the context.

This can be generally argued about trait/struct type generics too. It's a separate issue (the issue of overloading vs generics vs generics with reflection), and not specific to mutability.

@vitalyd

Vec<&Val> vs Vec<&mut Val> influences the types of Val references you can put into the Vec due to &mut changing variance. For instance, a certain Val type may not allow returning a &'static mut one and this influences code inside the fn as compiler would need to be conservative again.

Yes, a Vec<&mut Val> cannot take a reference to a subtype of Val, because that reference's data might be overwritten with an insufficient type. But as far as I can tell, this cannot happen with mutability generics because input data is supplied by the caller: In the mut case, input references will be to precisely the right type - suitable for a Vec<&mut T>. In the non-mut case, data might be a subtype - which is acceptable for the resultant Vec<&T>.

Maybe I've misunderstood the point. Could you provide an example where mutability generics cause a conflict with variance rules?