Existential types: Can you break this API? (soundness review)

TL;DR, please try to find potential soundness holes in this API, or confirm if you think that it should be sound

use std::{
    marker::PhantomData,
    mem::{self, ManuallyDrop},
};

pub trait TyConFor<'a> {
    type Applied;
}

pub type Apply<'a, C> = <C as TyConFor<'a>>::Applied;

pub trait TyCon: for<'a> TyConFor<'a> {}
impl<C: ?Sized> TyCon for C where C: for<'a> TyConFor<'a> {}

pub struct Existential<'lower_bound, C>
where
    C: TyCon,
{
    marker: PhantomData<&'lower_bound ()>,
    inner: Apply<'static, C>,
}

impl<'lower_bound, C> Existential<'lower_bound, C>
where
    C: TyCon,
{
    pub fn new<'a: 'lower_bound>(inner: Apply<'a, C>) -> Existential<'lower_bound, C> {
        let inner = ManuallyDrop::new(inner);
        unsafe {
            Self {
                marker: PhantomData,
                inner: mem::transmute_copy::<Apply<'a, C>, Apply<'static, C>>(&inner),
            }
        }
    }
    pub fn with<'s, F, O>(&'s self, f: F) -> O
    where
        F: for<'a> FnOnce(&'s Apply<'a, C>, PhantomData<&'lower_bound &'a ()>) -> O,
    {
        f(&self.inner, PhantomData)
    }
    pub fn with_mut<'s, F, O>(&'s mut self, f: F) -> O
    where
        F: for<'a> FnOnce(&'s mut Apply<'a, C>, PhantomData<&'lower_bound &'a ()>) -> O,
    {
        f(&mut self.inner, PhantomData)
    }
    pub fn with_owned<F, O>(self, f: F) -> O
    where
        F: for<'a> FnOnce(Apply<'a, C>, PhantomData<&'lower_bound &'a ()>) -> O,
    {
        f(self.inner, PhantomData)
    }
}

(code above contains one unsafe block that uses mem::transmute_copy)
(the same code in the playground, in case you don’t like copying stuff yourself)

By the way, I haven’t seen something like this anywhere else. If you have, please tell me :slight_smile:


Context / example use case

I managed to use this abstraction in an arena-based self-referential struct. If you have something like

#[ouroboros::self_referencing]
struct Tree<T: 'static> {
    dummy: (),
    #[borrows(dummy)]
    #[not_covariant]
    arena: typed_arena::Arena<Node<'this, T>>,
    #[borrows(arena)]
    #[not_covariant]
    root: &'this Node<'this, T>,
}

struct Node<'this, T> {
    element: T,
    children: std::cell::RefCell<Vec<&'this Node<'this, T>>>,
}

(this is somewhat simplified from a more practical use-case)

then it’s hard to write a by-reference iterator over all the T’s. The problem is that you cannot store any copies of the &'this Node<'this, T> references outside of Tree, because such a type would need to mention the internal “'this” lifetime of the self-referential struct. In particular, Node is not covariant (otherwise you could get rid of the 'this lifetime for such an iterator by subtype coercion). However, if you can create an existential type using the API above, it becomes possible:

impl<'s, T: 'static> IntoIterator for &'s Tree<T> {
    type Item = &'s T;

    type IntoIter = Iter<'s, T>;

    fn into_iter(self) -> Self::IntoIter {
        fn help_out_type_inference<'s, T: 'static, F, O>(f: F) -> F
        where
            F: for<'this> FnOnce(&'s &'this Node<'this, T>) -> O,
        {
            f
        }
        self.with_root(help_out_type_inference::<'s, T, _, _>(|root| Iter {
            marker: PhantomData,
            inner: Existential::new(IterInner { stack: vec![*root] }),
        }))
    }
}

struct IterInner<'this, T> {
    stack: Vec<&'this Node<'this, T>>,
}
struct IterInnerCon<T>(PhantomData<T>);
impl<'this, T: 'static> TyConFor<'this> for IterInnerCon<T> {
    type Applied = IterInner<'this, T>;
}
pub struct Iter<'s, T: 'static> {
    marker: PhantomData<&'s T>,
    inner: Existential<'s, IterInnerCon<T>>,
}

impl<'s, T: 'static> Iterator for Iter<'s, T> {
    type Item = &'s T;

    fn next<'a>(&'a mut self) -> Option<Self::Item> {
        fn help_out_type_inference<'s, T: 'static, F, O>(f: F) -> F
        where
            F: for<'this> FnOnce(&mut IterInner<'this, T>, PhantomData<&'s &'this ()>) -> O,
        {
            f
        }
        self.inner
            .with_mut(help_out_type_inference::<'s, T, _, _>(|inner, _| {
                inner.stack.pop().map(|i| {
                    inner.stack.extend(i.children.borrow().iter().rev());
                    &i.element
                })
            }))
    }
}

(only compiles on 1.56 or newer!)

dependencies of the example use case:

[dependencies]
ouroboros = "0.10.1"
typed-arena = "2.0.1"
2 Likes

FWIW, it looks sound to me as well:

  • : 'lt checks: the entry point is : 'lower_bound, and the resulting entity is infected with the 'lower_bound lifetime, effectively not being allowed to be used beyond it; (the &'s-bounded APIs are, among other things, caged inside 's, which is itself at most as big as 'lower_bound, and the owned API is higher-order-lifetime bounded, thus caged within the call to with_owned which must itself be contained within 'lower_bound by well-formedness of Self).

  • Thinking about ways to "escape" the higher-order scopes: there can't be such. To see why, let's consider a generalization of your Existential, one with an extra lifetime parameter, the one used for inner: Existential<'lower_bound, 'inner, C>. Then one can construct an Existential<'lower_bound, 'a, C> out of a Apply<'a, C> without any unsafe code whatsoever, and feature the with... APIs: this means these APIs themselves / their signatures, in your case, are not granting extra capabilities w.r.t. the safe case.

The object thus yields the same (reduced) API as a non-unsafe using equivalent, and itself, respects the : 'lower_bound outliving relationship (the same that makes dyn Trait + 'lt work), I thus fail to see how this could be unsound.

  • This is under the assumption that specializing over lifetimes will never be doable in Rust, since otherwise the parametricity of lifetimes would be broken and all hell would break loose :grinning_face_with_smiling_eyes:

Note that another question would be whether this abstraction is sound in the presence of ouroboros' API, but in the unlikely case where the would be a problem there, I'd blame ouroboros rather than your Existential setup.

2 Likes

Here is another way to express my thoughts:

pub struct Existential_<'lower_bound, 'inner : 'lower_bound, C>
where
    C: TyCon,
{
    marker: PhantomData<&'lower_bound ()>,
    inner: Apply<'inner, C>,
}

type Existential<'lower_bound, C> = Existential_<'lower_bound, 'static, C>;

trait IsExistential<'lb, C>
where
    C : TyCon,
{
    fn with<'s> (
        &'s self, f: &mut dyn for<'a> FnMut(&'s Apply<'a, C>, [&'lb &'a (); 0]),
    );
    fn with_mut<'s> (
        &'s mut self, f: &mut dyn for<'a> FnMut(&'s mut Apply<'a, C>, [&'lb &'a (); 0]),
    );
    fn with_owned (
        self: Box<Self>, f: &mut dyn for<'a> FnMut(Box<Apply<'a, C>>, [&'lb &'a (); 0]),
    );
}

impl<'lower_bound, 'inner, C> IsExistential<'lower_bound, C>
    for Existential_<'lower_bound, 'inner, C>
where
    'inner : 'lower_bound,
    C: TyCon,
{
    fn with<'s> (
        &'s self, f: &mut dyn for<'a> FnMut(&'s Apply<'a, C>, [&'lower_bound &'a (); 0]),
    )
    {
        f(&self.inner, [])
    }
    fn with_mut<'s> (
        &'s mut self, f: &mut dyn for<'a> FnMut(&'s mut Apply<'a, C>, [&'lower_bound &'a (); 0]),
    )
    {
        f(&mut self.inner, [])
    }
    fn with_owned (
        self: Box<Self>, f: &mut dyn for<'a> FnMut(Box<Apply<'a, C>>, [&'lower_bound &'a (); 0]),
    )
    {
        f(Box::new(self.inner), [])
    }
}
impl<'lb, C> dyn 'lb + IsExistential<'lb, C>
where
    C : TyCon,
{
    fn new<'inner : 'lb> (e: Existential_<'lb, 'inner, C>)
      -> Box<dyn 'lb + IsExistential<'lb, C>>
    {
        Box::new(e)
    }
}

My claim is that, API-wise, Existential<'lb, C> and Box<dyn IsExistential<'lb, C> + 'lb> are identical: both have performed erasure of the 'inner lifetime since it plays no role whatsoever in its API; the former through a transmute from Existential_<..., 'inner, ...> to Existential_<..., 'static, ...>, and the latter through dyn erasure :slightly_smiling_face:

1 Like

Thanks for the answer. I’ve totally forgotten about the fact that dyn Trait is Rust’s “existential types” feature, and it makes sense that it could be used to achieve something similar, yet less efficiently so.

I’ve put that code into a crate for now :slight_smile:

1 Like

Now that I’m seeing the docs I feel like the auto-trait impls here aren’t quite correct

impl<'lower_bound, C> RefUnwindSafe for Existential<'lower_bound, C>
where
    <C as TyConFor<'static>>::Applied: RefUnwindSafe,
impl<'lower_bound, C> Send for Existential<'lower_bound, C>
where
    <C as TyConFor<'static>>::Applied: Send,
impl<'lower_bound, C> Sync for Existential<'lower_bound, C>
where
    <C as TyConFor<'static>>::Applied: Sync,
impl<'lower_bound, C> Unpin for Existential<'lower_bound, C>
where
    <C as TyConFor<'static>>::Applied: Unpin,
impl<'lower_bound, C> UnwindSafe for Existential<'lower_bound, C>
where
    <C as TyConFor<'static>>::Applied: UnwindSafe,

I don’t know of any particular type that would implement Send/Sync only with a 'static lifetime, but it’s possible. I cannot imagine any reason why sending / sharing a value between threads would ever only be safe with 'static, but I guess technically one would need to fix this.

Slightly annoying, that replacing a Send/Sync implementation with a strictly weaker one (i.e. one with strictly stronger where clauses as condition) will still require another use of the unsafe keyword.

Yeah, that happens sometimes :upside_down_face:

  • :thinking: What if internally you used the 'inner = 'lower_bound substitution (rather than 'inner = 'static); would that keep the API sound while leading to better auto-trait impls?
1 Like

Great idea!

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.