Circular reference over FFI

help
1

Hello,

I'm working on a Rust binding for an existing C++ library, and fall into a situation where there is circular reference over objects, as following

struct Foo {
  // attach a listener 
  void attach(Bar *bar) {
    /* push bar into a list of listeners */
  }
  /* other methods */
};

struct Bar {
  Bar(Foo *foo) {
    if (foo) foo->attach(this);
  }
  /* other methods */
}

I believe that this implementation is erroneous, because when a listener Bar is deleted, there is no method to notify Foo; e.g:

Foo foo();
{
  Bar bar(&foo);
}
// bar is deleted here but foo doesn't know that and uses bar

In short, any correct usage requires that foo and bar have the same lifetime.

But my job is not to blame the C++ implementation, rather write a binding for that such a situation can be compile-time checked, e.g. some Foo and Bar in Rust for which:

let foo = Foo::new();
{
   let bar = Bar::new(&mut foo);
}

is refused by the compiler.

So I've done, first a C wrapper

// a simple C wrapper with type-erasure

extern "C" {
void *foo_create() {
  auto *f = new Foo();
  return f;
}

void *foo_attach(void* foo, void *bar) {
  auto *f = static_cast<Foo*>(foo);
  auto *b = static_cast<Bar*>(bar);
  f->attach(b);
}

void *bar_create(void *foo) {
  auto *f = static_cast<Foo*>(bar);
  auto *b = new Bar(f);
  return b;
}

then a Rust binding

// Rust binding

pub struct Foo {
  obj: NonNull<c_void>;        // opaque pointer to a Foo
}

pub struct Bar<'a> {
  obj: NonNull<c_void>;        // opaque pointer to a Bar
  _p: PhantomData<&'a ()>;     // store lifetime of some Foo
}

impl Foo {
  pub fn new() -> Self {
    Foo {
      obj: unsafe { NonNull::new_unchecked(foo_create()) },
    }
  }

  pub fn attach(&mut self, bar: &mut Bar) {
    unsafe { foo_attach(self.obj, bar.obj); }
  }
}

impl<'a> Bar<'a> {
  pub fn new(f: &'a mut Foo) {
    Bar {
       obj: unsafe { NonNull::new_unchecked(bar_create()) },
       _p: PhantomData, // store lifetime of f
    }
  }
}

But this simple binding doesn't work, the compilers only checks that the lifetime of foo is larger than (or equal to) bar, but not vice-versa, i.e.

let foo = Foo::new();
{
   let bar = Bar::new(&mut foo);
}

still type-check.

Adding a phantom field into Foo is non-sense, because we cannot predict the lifetime of a possibly attached bar: only when it's explicitly attached via attach.

Is there any method which helps check the lifetime of foo and bar be identical?

2

In general, when you're using FFI you want to prefer runtime checks like reference counting over compile-time checks like lifetimes.

For one, there's no guarantee data Rust receives from C++ will follow your constraints (making use of unsafe unsound). Also, fitting a lifetime system onto C tend to result in unusable APIs and problems like self-referential structs.

I'd recommend dropping the lifetime and adding your own reference-counted pointer abstraction. Otherwise, what do consumers of the normal C++ library do? Even if C++ is an unsafe language, they still need to use the API correctly to prevent memory problems.

1 Like
3

Here is a playground with a pattern that manages to set up what you want, at the cost of explicitly requiring to attach (which could be solved with a macro for easier sugar), using, under the hood, a form of stack pinning due to the &'a mut ...'a pattern.

As @Michael-F-Bryan said, it'll most probably hinder the ergonomics, but maybe it suffcies for your use case, and it is always interesting to see some lifetime hacks. For instance, having a struct carry a seemingly unrelated lifetime can be useful:

Example of usage:

fn main ()
{
    let foo = Foo::new();

    let mut bar = Bar::detached();
    let mut baz = Bar::detached();
    let mut foo = foo; // has to be after the different `Bar`s
    // let mut bar = bar; // Error (bar would be dropped before foo)
    { // You can have inner scopes since the attached objects have no destructor
        // bar.do_stuff(); // Error
        let bar = bar.attach_to(&mut foo);
        let baz = baz.attach_to(&mut foo);
        bar.do_stuff();
        baz.do_stuff();
    } /*
foo.drop(); // Dropped before the others
baz.drop();
bar.drop(); */
}
3 Likes
4

Thank you @Yandros for an excellent answer, as always. I must admit that it takes time to me to understand your code, sorry for my novice :frowning:.

As far as I understand, the crucial point is the attach_to method:

pub fn attach_to<'a>(&'a mut self, foo: &'_ mut Foo<'a>) -> Bar<'a> {
  ...
}

when applying, for example:

let mut bar: 'x = Bar::detached();
let mut foo: 'y = Foo::new();                  // so 'x :> 'y
{
  let bar: 'a = bar.attach_to(&mut foo);       // is 'a = 'y ?
}

The self in attach_to will be borrowed with some lifetime 'a so that:

  • &'a mut DetachedBar is shorter than (or equal to) &'x mut DetachedBar, while
  • &'_ mut Foo<'a> is shorter than (or equal to) &'y mut Foo<'?> (I'm not sure which lifetime should be filled into ?, it's some 'b in _lifetime: PhantomInvariant<'b> of Foo, so I suppose that it's 'y)

The former, we have 'a <= 'x because &'a mut T is covariant over 'a. The latter, PhantomInvariant<'a> is invariant over 'a, so 'a = 'y. In summary, the condition 'a = 'y <= 'x must be satisfied, that's why foo has to be after bar.

IMHO the returned Bar of attach_to doesn't need to be Bar<'a>, any Bar<'_> should work too. And the phantom lifetime in

pub struct Bar<'foo> {
    obj: ptr::NonNull<c_void>,         // opaque (borrowing) pointer to a bar
    _lifetime: PhantomInvariant<'foo>, // store lifetime of some Bar
}

can be covariant, i.e. we can use

pub struct Bar<'foo> {
    obj: ptr::NonNull<c_void>,         // opaque (borrowing) pointer to a bar
    _lifetime: PhantomData<&'foo ()>,  // store lifetime of some Bar
}

Is this true?

Another point which still seems magic to me is with

pub struct Foo<'a> {
    obj: ptr::NonNull<c_void>,         // opaque pointer to a Foo
    _lifetime: PhantomInvariant<'a>,   // store lifetime of some Foo
}

then in

let foo: 'y = Foo::new();

which is the value of 'a in _lifetime: PhantomInvariant<'a>: it seems coming from nowhere?

Another problem which is implied by @Michael-F-Bryan and you is code ergonomic, there seems no way to keep the original FFI constructor of Bar:

extern "C" bar_create(foo: NonNull<c_void>);

indeed, there exists always a DetachedBar which must be created first (with lifetime larger than one of Foo), later a Bar is created by attaching the detached bar into foo.

5

Indeed it is

  • :white_check_mark:'x : 'y

  • :grey_question: Quid of 'foo the lifetime in foo's type? This is actually a parameter that Rust can set as it wishes, except that 'foo : 'y needs to hold (you cannot have a variable outlive its own type, that is just silly / ill-defined).

  • let bar: 'a :x: Again, same as with 'y and 'foo, one must distinguish the (invisible) lifetime of a binding with the lifetime within its type. So it is let bar : 'bar, where 'a : 'bar (like with 'foo : 'y)

  • is 'a = 'y ? :x: Instead, we have 'a = 'foo

    • Indeed, &'_ mut Foo<'foo> : &'_ mut Foo<'a> if and only if 'a = 'foo, due to invariance of the Foo<_> lifetime parameter, thus 'a = 'foo

      • This is the key thing that makes the whole thing be sound.

    • :white_check_mark: &'a mut DetachedBar is shorter than (or equal to) &'x mut DetachedBar
      'x : 'a i.e., 'x : 'foo

Conclusion:

'x : 'foo : 'y, so Rust does not have that much freedom for the 'foo parameter. Note that 'x :< 'y (taking your notation) due to the order of the let declarations of foo and bar, meaning that 'x : 'y and 'x != 'y. If that order had been inverted, we would have had 'y :< 'x on one hand, and 'x : 'foo : 'y on the other, which is not possible.

If the lifetime in the returned Bar<'_> is truly free (imagine having <'c>(...) -> Bar<'c>), then the returned value would be able to outlive both the borrow on bar and the one on 'foo ('c : 'bar would not imply that 'foo : 'bar). Outliving the mut borrow on foo is fine (it is even necessary to be able to reborrow foo for baz, but outliving the borrow on bar /self (and thus being able to outlive DetachedBar is not). So we do need it to be Bar<'a>.

That, on the other hand, seems true. I admit I didn't think that one through, since, when in doubt, invariance is never wrong, it can just hinder some usages (Imagine wanting a function to compare / use two Bars coming from two different foos. If this is an acceptable operation of your API, then the lifetime will need to be covariant indeed). So covariance does not look wrong here indeed :+1:

Yes, their warning was justified: my solution was more of a theoretical exercise than anything else, it will prevent many use cases. Having runtime-based checks would give more flexibility (e.g., imagine wrapping Foo in an Rc, and then having each wrapped Bar also hold a clone of this Rc, thus guaranteeing that no matter the code pattern, no Bar gets to be alive while its Foo is dead (since an alive Bar keeps Foo alive).

1 Like
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.