Curious bororow checker problem getting image buffers from two camera streams

This is a cross-post from the internals forum as I was suggested to take this issue here :slight_smile: Curious bororow checker problem getting image buffers from two camera streams - compiler - Rust Internals

I have an application, where I have two cameras connected using the v4l API.
I am getting images from the cameras, and the methods return &[u8] buffers which are owned by the v4l driver.

I need to synchronize the two cameras, so that their images are in sync, for that the procedure is the following:

  1. Get left_img: &[u8] = left_stream.next(); and right_img: &[u8] = right_stream.next(); from the two camera streams
  2. Given the two buffers, extract and compare their timestamps
  3. If the left camera is behind, we "refresh" left_img = left_stream.next();
  4. Else if the right camera is behind, we get the next right_img = right_stream.next();
  5. Return (left_img, right_img)

This synchronization logic is verbose and messy, and I want to abstract it out, but the Rust borrow checker does not allow me to.
I have been able to make a case where I do almost exactly the thing I want to, but only for a single camera, but the minute I try to do it for two cameras, it breaks down. Could any one help me cracking this nut?

A greatly simplified example showing the issue is available on this link: Rust Playground

Any help and advice would be greatly appreciated!

Streams are buffered generally for the convenience of accessing their buffers multiple times before advancing. I think the same could apply to the DummyStream in your example. Instead of gettings its buffer only via next is there a method for getting the current frame without advancing? That would solve the borrowing problems completely. Get a hold of the buffer a second time when you actually return, having advanced both sufficiently.

In this case the buffers are streamed directly from the MIPI (low-level electrical camera interface) hardware, and there is no direct way of accessing them without calling next given the current API surface of libv4l-rs.

I believe I need to find a way to make this work, without changing the semantics of DummyStream.

The borrow checker complains here, but it is easy to see that the behavior is safe, as no invalid memory is ever read/written. I really feel there should be a way to do this.

To convince myself that I am not completely off, I have made a full crate that happily compiles as long as it is compiled with RUSTFLAGS="-Zpolonius".

The crate is here: GitHub - molysgaard/rustc-polonius-lifetime-challenge

If someone is able to convert this crate to something that works on stable rust I would be very grateful!

Does this count? Rust Playground

(Sorry, I don't know a better solution.)

1 Like

There's more to the borrow checker than that, but you're not wrong that this case is fine. And the plan is to make the compiler smarter so that it accepts this kind of code. (OK, from your post while I was writing this, I see you understand.)

In the meanwhile, given the API restrictions, I think your question boils down to

  • Can polonius_the_crab handle this?[1]
  • If not, is my unsafe sound?

If unsafe is acceptable...[2]

        match DeltaRes::from_delta(delta) {
            DeltaRes::Synchronized => {
                // Cameras are synchronized, and we do not need to update any of them
                // SAFETY: We're working around the conditional return
                // of borrow limitations of the NLL borrow checker.
                // These casts disassociate the borrow of our fields to
                // allow the below calls to `next` to compile.  There
                // are no accesses to the borrowed fields between the
                // casts and the return.
                // FIXME: Remove `unsafe` once possible
                let l_buf = unsafe { &*(l_buf as *const _) };
                let r_buf = unsafe { &*(r_buf as *const _) };
                (l_buf, r_buf)
            }
            DeltaRes::LeftLagging => {
                // Left is lagging behind, update it
                let l_buf = self.left_stream.next();
                // SAFETY: See above
                // FIXME: See above
                let r_buf = unsafe { &*(r_buf as *const _) };
                (l_buf, r_buf)
            }
            DeltaRes::RightLagging => {
                // SAFETY: See above
                // FIXME: See above
                let l_buf = unsafe { &*(l_buf as *const _) };
                // Right is lagging behind, update it
                let r_buf = self.right_stream.next();
                (l_buf, r_buf)
            }
        }

  1. Sorry for not having an answer to this ↩ī¸Ž

  2. n.b. I didn't try to compile, may need tuning ↩ī¸Ž

1 Like

Hmm, the doubly mutually-reborrowing pattern was indeed challenging for ::polonius-the-crab; but that crate is nonetheless capable of yielding a compiling solution that requires no unsafe from the caller:

fn next_frame<'r>(&'r mut self) -> (&'r [u8], &'r [u8]) {
  use ::polonius_the_crab::*;

  // outer polonius, we assume the `r_buf` side has been sorted out.
  match polonius::<_, _, ForLt![<'left> = (&'left [u8], &'r [u8])]>(
      &mut self.left_stream,
      // inner polonius; we have to sort out the `r_buf` stuff.      'left
      |left_stream| match polonius::<_, _, ForLt![<'right> = (Option<&'_ [u8]>, &'right [u8])]>(
          &mut self.right_stream,
          |right_stream| {
              let l_buf = left_stream.next();
              let r_buf = right_stream.next();

              // Sometimes, we want to skip this frame based on some condition that
              // can only be calculated when we have access to the buffer.
              // In the camera case, this will be when a frame is too old compared to the other camera.
              let delta = l_buf[0] - r_buf[0];

              match DeltaRes::from_delta(delta)
              // polonius logic: we are within the *inner* `polonius()` call, to determine
              // whether we are outputting anything `right_stream`-dependent, such as `r_buf`,
              // in which case we have to be `PoloniusResult::Borrowing` (and allowed to use
              // `r_buf`); otherwise we return `PoloniusResult::Owned` with everything but
              // `r_buf`/`right_stream`-related stuff, and the *caller* will receive the
              // original `right_stream` in the `input_borrow` spot.
              {
                  DeltaRes::Synchronized => {
                      // Cameras are synchronized, and we do not need to update any of them
                      // (l_buf, r_buf)
                      // \-> mentions r_buf: `Borrowing`
                      PoloniusResult::Borrowing((Some(l_buf), r_buf))
                  }
                  DeltaRes::LeftLagging => {
                      // Left is lagging behind, update it
                      // (self.left_stream.next(), r_buf)
                      // \-> mentions r_buf: `Borrowing`
                      //   + but we'll also need to convey, to the caller, that
                      //     we're not interested in `l_buf` anymore.
                      //     Hence the `Option`.
                      PoloniusResult::Borrowing((None, r_buf))
                  }
                  DeltaRes::RightLagging => {
                      // Right is lagging behind, update it
                      // (l_buf, self.right_stream.next())
                      // \-> we want to use `right_stream`, so we skip any further mentions
                      //     of `r_buf`, and return something `Owned`.
                      PoloniusResult::Owned {
                          input_borrow: Placeholder, // <- please give me back `right_stream`
                          value: l_buf, // "pass-through" companion payload.
                      }
                  }
              }
          }
      )
      // polonius logic: we are within the *outer* `polonius()` call, receiving
      // the output of the inner `polonius`: we'll either have a `return`-able `r_buf`
      // (friendly lifetime in it), or access to the `input_borrow` to the `right_stream`,
      // so as to query again.
      // We still have to determine whether we are outputting anything `left_stream`-dependent,
      // such as `l_buf`;
      /* .match */ {
          /*Inner*/PoloniusResult::Borrowing((Some(l_buf), r_buf)) => {
              /*Outer*/PoloniusResult::Borrowing(
                  (l_buf, r_buf)
              )
          },
          // (self.left_stream.next(), r_buf)
          /*Inner*/PoloniusResult::Borrowing((None, r_buf)) => {
              /*Outer*/PoloniusResult::Owned {
                  input_borrow: Placeholder, // <- please give me back `left_stream`
                  value: r_buf,
              }
          },
          // (l_buf, self.right_stream.next())
          /*Inner*/PoloniusResult::Owned {
              input_borrow: right_stream, // <- we got `right_stream` back!
              value: l_buf,
          } => /*Outer*/PoloniusResult::Borrowing(
              (l_buf, right_stream.next())
          ),
      }
  )
  // Outside the outer polonius: time to just inject `left_stream.next()` wherever we
  // receive the `input_borrow: left_stream` back:
  /* .match */ {
      PoloniusResult::Borrowing((l_buf, r_buf)) => (
          l_buf, r_buf,
      ),

      // (self.left_stream.next(), r_buf)
      PoloniusResult::Owned { value: r_buf, input_borrow: left_stream } => (
          left_stream.next(), r_buf,
      ),
  }
}
3 Likes

Hi @Yandros and @quinedot. Thank you for your help and time.

This has unblocked me and I am now able to abstract this into a working pattern that lets me hide the ugly details.

Wish you a nice day!

1 Like

FWIW, it will be possible for ::polonius-the-crab to expose a new helper function, polonius2, that will be aware of these more advanced disjunctions, so as to let your function be "reduced" to:

fn next_frame(&mut self) -> (&[u8], &[u8]) {
    match
    polonius2::<ForLt!(&[u8]), ForLt!(&[u8]), _, _, _, _, _, _>(
        &mut self.left_stream,
        &mut self.right_stream,
        |left_stream, right_stream| {
            let l_buf = left_stream.next();
            let r_buf = right_stream.next();
    
            // Sometimes, we want to skip this frame based on some condition that
            // can only be calculated when we have access to the buffer.
            // In the camera case, this will be when a frame is too old compared to the other camera.
            let delta = l_buf[0] - r_buf[0];
    
            match DeltaRes::from_delta(delta) {
                DeltaRes::Synchronized => {
                    // Cameras are synchronized, and we do not need to update any of them
                    PoloniusEither::BorrowingBoth(l_buf, r_buf, ())
                }
                DeltaRes::LeftLagging => {
                    // Left is lagging behind, update it
                    PoloniusEither::BorrowingRight(Placeholder, r_buf, ())
                }
                DeltaRes::RightLagging => {
                    // Right is lagging behind, update it
                    PoloniusEither::BorrowingLeft(l_buf, Placeholder, ())
                }
            }
            
        },
    )
    /* .match */ {
        PoloniusEither::BorrowingBoth(l_buf, r_buf, ()) => (
            l_buf,
            r_buf,
        ),
        PoloniusEither::BorrowingLeft(l_buf, right_stream, ()) => (
            l_buf,
            right_stream.next(),
        ),
        PoloniusEither::BorrowingRight(left_stream, r_buf, ()) => (
            left_stream.next(),
            r_buf,
        ),
        PoloniusEither::BorrowingNone(_, _, never) => {
            let _: Never = never;
            match never {}
            enum Never {}
        },
    }
}
2 Likes