How to reference data without copying and lifetime

help
1

My program involves in reading data from data,
storing the data in a vec,
and use a struct to store the parsed result.
I want to avoid the copy the original data.

let file_data: Vec<u8> = ... // Read from file

use core::ffi::CStr;
struct Foo<'a> {
   name: &'a CStr,
   value: &'a [u8]
}

However, now I need to bind Foo to Python,
so I need to use the pyo3 crate.

It requires that python class cannot have lifetime,
because python uses garbage collection and referencing counting to handle lifetime

struct Foo {
   ...
}

So what is the recommended approach to the remove 'a lifetime?

I know that one way is to use self referencing type

/*
[dependencies]
ouroboros = "0.16"
*/

use ouroboros::self_referencing;
use core::ffi::CStr;
use std::sync::Arc;

#[self_referencing]
struct Foo {
    file_data: Arc<Vec<u8>>,
    #[borrows(file_data)]
    // the 'this lifetime is created by the #[self_referencing] macro
    // and should be used on all references marked by the #[borrows] macro
    name: &'this CStr,
    #[borrows(file_data)]
    value: &'this [u8],
}

But this makes API very complicated,
and many extra code needed to construct the struct,
and parsing code need to be redesigned

and I have roughly 10 similar structs

Any suggestions?

2

If you use ouroboros, one thing that you could do to limit the amount of refactoring is to instead of changing Foo itself, pack it up into another struct

#[self_referencing]
struct FooOwned {
    file_data: Arc<Vec<u8>>,
    #[borrows(file_data)]
    #[covariant]
    foo: Foo<'this>,
}

This means that your parsing code, assuming it essentially does some &'a [u8] -> Foo<'a> transformation, could stay unchanged.

Given you have multiple structs, it’d be interesting to learn more about how they interact together. Is there many structs sharing the same data? Is it perhaps even a hierarchy? What kind of API do they have, for themselves or between each other, and what are the effects you described as “makes API very complicated” looking like when you tried ouroboros?

For situations where you want to keep the ability to easily clone the owned version of the struct, and/or if you want to be able to do some FooOwned -> BarOwned transformations, based on former Foo<'a> -> Bar<'a> transformations, then I believe the yoke crate might offer even more powerful (whilst also simpler) APIs.

/*
[dependencies]
yoke = { version = "0.7", features = ["derive"] }
*/

use yoke::{Yoke, Yokeable};
use std::sync::Arc;

use core::ffi::CStr;
#[derive(Yokeable, Copy, Clone)]
struct Foo<'a> {
    name: &'a CStr,
    value: &'a [u8],
}

#[derive(Clone)]
struct FooOwned(Yoke<Foo<'static>, Arc<Vec<u8>>>);

and some demo of having a second type and wrapping a parsing function and a conversion function:

use std::error::Error;

impl<'a> Foo<'a> {
    fn parse(data: &'a [u8]) -> Result<Self, Box<dyn Error>> {
        todo!()
    }
}

impl FooOwned {
    fn parse(data: Arc<Vec<u8>>) -> Result<Self, Box<dyn Error>> {
        Ok(Self(Yoke::try_attach_to_cart(data, |data| Foo::parse(data))?))
    }
}

#[derive(Yokeable, Copy, Clone)]
struct Bar<'a> {
    value_part1: &'a [u8],
    value_part2: &'a [u8],
}

impl<'a> Bar<'a> {
    fn from_foo(foo: Foo<'a>) -> Self {
        Self {
            value_part1: &foo.value[0..10],
            value_part2: &foo.value[10..],
        }
    }
}

#[derive(Clone)]
struct BarOwned(Yoke<Bar<'static>, Arc<Vec<u8>>>);

impl BarOwned {
    fn from_foo(foo: &FooOwned) -> Self {
        Self(foo.0.map_project_cloned(|foo, _| Bar::from_foo(*foo)))
    }
}
3

Thanks.
I decide to refactor the code,
and try to make every parsed struct lifetime-less


struct FileData {
   data: Vec<u8>
}

use core::ffi::CStr;
#[derive(Yokeable, Copy, Clone)]
struct FooBorrow<'a> {
    name: &'a CStr,
    value: &'a [u8],
}

#[derive(Clone)]
struct Foo(Yoke<FooBorrow<'static>, Arc<FileData>>);