Question about `Pin::new()`

I've been studying Future which related to Pin. And I feel that the design of Pin::new is kind of counterintuitive. I mean, if the value can't be pinned, why would I pin it? In another word, does Pin::new() really have a usage scenario? Or such a design is only due to some historical issues?

1 Like

I think you need to pin values (using Pin::new) because a function may expect:

fn foo<T>(arg: Pin<T>)

instead of

fn foo<T: Unpin>(arg: T)

Both describe the same (but require different syntax), if I understand right.

If a type is Unpin, it means you can pin and unpin it in safe Rust. For types which are !Unpin, you can only do this in unsafe Rust (or by using some of the safe functions which pin it on the stack as done by tokio::pin! or the unstable std::pin::pin! or on the heap using Box::pin, for example).

IMO, Pin is designed to pin values, so the unpinned case should be excluded, we doesn't want a false pinned data, right? So if Pin::new() is more reasonable like this:

impl<P> Pin<P>
where
    P: Deref,
    <P as Deref>::Target: !Unpin,
{
    pub fn new(pointer: P) -> Pin<P>
}

Unpin really doesn't mean the type can't be pinned. It rather means: it doesn't matter if it's pinned or not (if I understand it right).

I think we need both Pin and Unpin because sometimes we want to described a wrapped type that is pinned (or where it doesn't matter whether it's pinned or not) in which case we use Pin<T>. And sometimes we want (or need to?) to express the same through a bound, in which case we use T: Unpin. Both are effectively the same, I guess. The second case (using T: Unpin) might be necessary in case of generic programming, I think.

According to docs, if the <P as Deref>::Target is Unpin, you can get it's mutable reference and call std::mem::swap or std::mem::replace to get it move. So it's couldn't be pinned obviously.

It does get pinned wrapped by Pin, but Pin::get_mut is "unpinning" unwrapping it (the method could have been named Pin::unpin, I guess). Pinning Pin-wrapping a &mut reference to an Unpin type can be undone, which is basically what's said in the beginning of the documentation of Unpin:

Edited: See this post of @steffahn regarding teminology.

Trait std::marker::Unpin

pub auto trait Unpin { }

Types that can be safely moved after being pinned.

Implementing the Unpin trait for T […] allows moving T out of Pin<P<T>> […].


P.S.: I feel like in this thread I have to use the prefix "un" a bit too often. :sweat_smile: unpin, undone, !Unpin, not pinned, not unpinned, … :face_with_spiral_eyes: It's certainly confusing.

1 Like

Pin::new is not about pinning a value, but about creating a value of type Pin<T>.

Why would you need to create a value of – say – type Pin<&mut Foo> even though Foo never actually needs pinning? The answer is: generic programming! Creating a general abstraction is what gave us this “weird/useless” type in concrete scenarios.

It’s perhaps similarly weird as how calling filter on a by-reference iterator will hand out &&Ts, or how looking up a value in a HashMap<Foo, u8> will give you back a &u8 return value, even though it could’ve copied it into a (smaller and more efficient) owned copy of type u8.

The main application of Pin is in the signature of the Future trait’s poll method.

fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;

Thus, if you want to poll a future of type Foo you need a value of type Pin<&mut Foo>. It doesn’t matter whether the actual concrete future implementation of Foo relies on pinning or not, the type signature requires Pin<&mut Foo> so you gotta provide Pin<&mut Foo>. To make this not so much of a hassle, you don’t actually need to pin a future type that doesn’t rely on pinning for its implementation. If it doesn’t rely on pinning, then it will implement Foo: Unpin. Then you can think of &mut Foo and Pin<&mut Foo> as being “essentially the same thing”, i.e. you can safely convert between the two; and in this case Pin::new is the way to convert these two types.

Why does Future involve the Pin<&mut Self> type? Because many futures do rely on pinning. It’s simply more general this way. In a generic setting, this is the better way. A trade-off. We support futures that rely on pinning, but we make the case of concrete future types that didn’t need the pinning in the first place slightly more tedious to call.

That’s all pretty tedious; why does the Future trait not have an additional method that doesn’t require me to use the Pin::new trick for such future types? Well, it exists, but not on the Future trait itself in the standard library which wants to be minimal. (And also it’s not actually designed for common manual usage, anyways. The convenient way to interact with futures is via async/await which you can do without learning about pinning at all in the first place.) The extension trait in the future crate will have the convenience method we might want, and using that, we can hide the (still extant) call to Pin::new behind a nicer abstraction.

8 Likes

Okay, I made a mistake with terminology. Converting a pointer &mut T into Pin<&mut T> isn't pinning T in memory. Instead it creates a wrapped pointer, which conditionally pins the value:

Struct std::pin::Pin

pub struct Pin<P> { /* private fields */ }

A pinned pointer.

This is a wrapper around a kind of pointer which makes that pointer “pin” its value in place, preventing the value referenced by that pointer from being moved unless it implements Unpin.

(Relevant part made bold by me.)

Right… the terminology is hard to work with. Any maybe not 100% clearly defined (or if it is, I wouldn’t know which way it is defined off the top of my head). Is a : Unpin type never “pinned”, or is “pinning” still possible but it’s just semantically irrelevant/void?

Note that converting &mut T to Pin<&mut T> (safely) will never actually do any semantically relevant “actual pinning”.

The only way to “actually” pin a value is to use methods like Box::into_pin (or the convenience function Box::pin which combines Box::into_pin with Box::new) or some stack-pinning macro like futures::pin_mut!. Or, if you aren’t doing the calls to Future API manually, the .await syntax will effectively do a stack-pinning under the hood for you, too.

1 Like

If it helps, IIRC one of the alternatives for Pin was adding another implicit bound (like Sized) called Move, and instead of Future::poll(self: Pin<&mut Self>,...) it would be something like Future::poll(&mut self,...) where Self: ?Move - or in other words, implementations of Future are not allowed to assume they can move from self, and many implementations of Future would be !Move

Now obviously there's some more questions here, the most obvious being how do you return a value you can't move, and the design that got picked was the current, where you explicitly state when it starts not being movable (if it matters) by constructing a Pin, and a type declares that it doesn't matter if it's pinned by implementing Unpin (or rather, not suppressing the automatic Unpin)

It's admittedly quite confusing: I'm pretty sure I've gotten several things wrong here!

3 Likes

The fact that Unpin is called Unpin somewhat encourages to speak of "unpinning" (which is what I was doing without thinking about it properly). After your post and after skimming through the documentation, I would say that a value never gets "unpinned".

Maybe it's possible to use the wording "unpinning" to describe unwrapping a pointer from Pin (using Pin::into_inner) or for obtaining a &mut T (using Pin::get_mut), but I guess that would not be consistent with Rust's documentation.

Perhaps the name Unpin is a bit unfortunate, and you could think of it more as WontPinEvenIfWrappedInPin (as a memory aid).

Reading through the full previous discussion now, I notice that this statement is wrong. Those two signatures are not the same. Though often there is an arbitrary choice of this style, commonly e.g. between (self: Pin<&mut Self>) and (&mut self) where Self: Unpin on Streams. But those two are only (somewhat) equivalent, because there’s a generic implementation of Stream for Pin<&mut S> where S: Stream, so that

  • if you have a &mut Foo where Foo: Unpin, you can create Pin<&mut Foo> via Pin::new, and
  • if you have a Pin<&mut Foo>, then &mut Pin<&mut Foo> will also be a &mut impl Stream, but also Pin<&mut Foo>: Unpin is always true

Note that switching between these two alternatives will result in more and more levels of indirection :sweat_smile:, so while the particular choice is not too relevant, some consistency is valuable. I’m not familiar enough with these things to give suggestions on best practices. Just to make things more complicated: There’s also a third alternative: an owned (self) (without extra Self: Unpin bounds). For that one, you could pass either of Pin<&mut Foo> or &mut Foo where Foo: Unbound since both of these implement Stream themself.

My current understanding is as follows:

Say P<T> is a pointer to T, e.g. P<T> could be &mut T. Then:

  • Creating a Pin<P<T>> only pins T in memory if T: !Unpin.
  • Pin::new (which is safe) can only accept a P<T> as argument where T: Unpin. That means Pin::new will never actually pin a value. In fact, it's generally not possible in safe Rust to pin a T in memory by consuming a pointer to T. That is because the pointer may be short lived (which is explained in the docs of Pin::new_unchecked).
  • Implementing Unpin for a type T means
    • that it never matters whether a value of type T is pinned in memory,
    • and that a value of type T will never be pinned.

Getting back to the OP:

The answer is: Pin::new doesn't "pin" a value in memory, because:

1 Like

If you want to get even more subtle. Even an !Unpin type might not actually really pin anything. The documentation of std::pin is conservatively talking about the general case. I.e. what you are allowed or not allowed to do with types someone else defined. If it says you “must” or “must not” do something, then that’s usually talking about type you didn’t define yourself.

E.g. in Pin::get_unchecked_mut docs

You must guarantee that you will never move the data out of the mutable reference you receive when you call this function, so that the invariants on the Pin type can be upheld.

or Pin:new_unchecked

A value, once pinned, must remain pinned forever (unless its type implements Unpin).

are safety requirements that only apply in the general case.

The picture becomes clearer when structural pinning is considered, which demonstrates that every type/struct can essentially free decide on its own which fields it considers pinned and which once it doesn’t. A type with some fields structurally pinned will be !Unpin, but this does not mean that none of the fields’ memory can move, the not structurally pinned ones can move freely. And if none of your struct’s fields are structurally pinned, but you still decide not to implement Unpin, than you as the struct’s author are still free to write unsafe code that acts as if the type was implementing Unpin; after all you could’ve (safely) implemented Unpin anyways. In this context, playing around with safe structural pinning using the pin-project crate can also be educational.

I would say that Pin::<P<T>>::new_unchecked(p) does pin a T: !Unpin in memory insofar as safe Rust is concerned. Using safe/sound Rust, it's impossible to move the pointed-to value of type T afterwards.

With unsafe Rust, it's always possible to do anything (and even cause UB).


I guess you can provide projections or other safe (and sound!) methods which allow you to move some (or all) of the data, particularly those where it doesn't cause problems. So if I claim that Pin::<P<T>>::new_unchecked(p) does pin a T: !Unpin, then it would mean it's also possible to reverse this using unsafe Rust (or projections).

(But I'm not sure at all if I get things right now.)

You can define a type safely (using helper crates) and soundly that is !Unpin but doesn’t really get pinned anyways. Here’s a demo

use bytemuck::TransparentWrapper;
use pin_project::pin_project;
use core::marker::PhantomPinned;

use std::marker::PhantomData;
use std::pin::Pin;

#[derive(TransparentWrapper)]
#[repr(transparent)]
#[pin_project]
#[transparent(T)]
struct Weird<T>(T, #[pin] PhantomPinned);

fn it_really_aint_unpin<T: Unpin /* even with T: Unpin */>() {
    fn test_unpin(_: impl Unpin) {}
    // uncomment to “do” the test
    // test_unpin(PhantomData::<Weird<T>>); // compilation failure
}

fn this_works<T>(x: Pin<&mut Weird<T>>) -> &mut Weird<T> {
    Weird::wrap_mut(x.project().0)
}

run online


But all this example is showing is that when you define your own type and make it unnecessarily a !Unpin type, it can be soundly possible for you, the implementor to act as-if the type was Unpin after all.

I don't fully understand that example, and I don't know where wrap_mut comes from, but I guess it's only possible because the PhantomPinned value can be created/copied/doesn't really exist.

Right. There are crates that allow the casting that #[repr(transparent)] promises is safe, in this case from &mut T to &mut Weird<T>. Probably the best crate for that is ref_cast - Rust; I just used the first I could think of (bytemuck) instead, but the result should be equivalent.