Making a value of a type `Undroppable` at compile time

This is a follow-up question on Creation of Non-linear types in Rust.

Suppose I have a non-linear type (defined as a type whose value can be used at most N: usize times), let's call it, with an example, Secret<T, const MEC: usize, const EC: usize>; MEC is the maximum number of times the value can be used and EC is the current number of times the value is used.

With const NOT_MAX: () = assert!(EC < MEC), I can make compile-time assertions that Secret is used at most MEC times.

Now, I want to make a SecretRef<'secret, T, MEC, EC>, returned via a method expose_secret<'secret>(self) -> SecretRef<'secret, T, MEC, { EC + 1 }> of Secret type.

However, expose_secret(...) again cannot be called since I already consumed it to get a SecretRef. Hence, I should make sure SecretRef has a consuming Self method that returns back Secret. Yet, how do I force the user of SecretRef to call say, a method call into_secret(Self) -> Secret<...>?

Finally, the basic idea is: Secret -> SecretRef -> Secret -> SecretRef -> ... -> Secret {(MEC × 2) + 1 times}.

I have seen this pattern in the wild with blockchain applications but I do not know how to implement it.

I have thought of two ideas.

The first is not satisfying because I can always have a flag that is switched on, then off when SecretRef is created and dropped respectively. But this is a runtime check.

The second is to embed the flag in the const generics. However, if I were to implement const assertion in the implementation of Drop in SecretRef, the method of SecretRef, into_secret(self) will consume SecretRef and raises a compile time panic because of the const assertion which I do not want.

use wrapper struct?

const MEC: usize = 10;

#[derive(Clone)]
struct Secret;

struct SecretPointer {
    secret: Secret,
    mec: usize,
    ec: usize,
}

impl SecretPointer {
    pub fn new(secret: Secret) -> Self {
       SecretPointer { secret, mec: MEC, ec: 0 }
    }

    pub fn secret_ref(&mut self) -> Option<Secret> {
       if self.ec < self.mec {
           self.ec += 1;
           Some(self.secret.clone()) // whatever
       } else { None }
    }
}

Sorry but thank you for your response, I need this to be done at compile time, preferably, so I can avoid Result and Option.

AFAIK, this isn't possible, as was discovered during the leakpocalypse. There's no way to prevent the user mem::forgetting the value, bypassing the destructor that's supposed to ensure this behavior.

2 Likes

Disclaimer: in the remainder of my post, I'll be using the currently nightly-only / unstable feature(generic_const_exprs), to keep things simple and intuitive. On stable Rust, there are two ways to polyfill them:

  • the preferred method, here, imho, would be to be using ::typenum generic types as const GENERIC: usize replacements, since by virtue of being types within which basic arithmetic operations such as + 1 have been encoded in it, it lets us write generic_const_exprs-looking statements in stable Rust. Really good fit here.

  • in other scenarios, using const within which assert!()ions are made is also possible (called a post monomorphization error). This is, however, to be used as a last resource, since:

    • the errors when these assertions fail are not always displayed to the user. Most infamously, a simple cargo check is unable to trigger post monomorphization errors (since such a command never makes it to codegen let alone monomorphization).

    • there are certain cases where Rust will allow a failing/panicking constant without causing a compilation error. Beyond the aforementioned cargo check case, consts in generic context will not cause this either, and up until the most recent versions of Rust, a failing const in dead_code/unreachable_code, such as if false { function_with_failing_const() }, would not trigger this either.

      All this, in turn, means that using these failing constants as a safeguard mechanism can be rather error-prone or flaky-ish.


You seem to have, currently, an API along the lines of:

fn demo<T>(s: Secret<T, 3, 0>)
{
    let exposed = s.expose_secret();
    let s = exposed.into_secret();
    let exposed = s.expose_secret();
    let s = exposed.into_secret();
    let exposed = s.expose_secret();
    let s = exposed.into_secret();
    
    // let exposed = s.expose_secret(); // <- Error!
}

Thanks to:

pub
fn expose_secret(self: Secret<T, MEC, EC>)
  -> ExposedSecret<T, MEC, EC>
where
    generic_const_predicate!(MEC > EC) :,
{
    ExposedSecret(self.0)
}

and:

pub
fn into_secret(self: ExposedSecret<T, MEC, EC>)
  -> Secret<T, MEC, { EC + 1 }>
{
    Secret(self.0)
}

Imho, to achieve the MEC > EC enforcement, you do not need to require .into_secret() to be called back: the user could technically get rid of the exposed secret, and nobody else gets access to it. Their loss.


But, if you really want to do it, or are just curious about how would one do it should they want to, the only solution we have at the moment, in Rust, to "guarantee" these things is by a combination of two things:

  • a scoped / callback-based API, so as to put a very specific constraint on the return type of such a closure;
  • a builder pattern / typestate pattern wherein this specific return type can only be producing by having consumed one of the closure inputs in a very specific manner.

Note that this does not guarantee lack of panics, infinite loops, or aborts etc.. What it guarantees is that if the callback returns without unwinding, then the expected function calls have taken place.

source.with(|transformed| {
    /* stuff... */
    
    // eventually, required by the callback return type
    transformed.undo() // <- required unless we `!`-diverge (e.g., panic).
});

Given the above remark, it would technically be possible to use an internal panic to avoid the requirement, coupled with a catch_unwind() to recover from it:

let mut smuggling_channel = None;
catch_unwind(|| {
  source.with(|transformed| {
    smuggling_channel = Some(transformed);

    panic!();
  })
}).ok();
let transformed = smuggling_channel.unwrap();
// It escaped!

Which is why, optionally, but customarily, the tranformed callback input / value yielded by the API is infected with a scope-emprisoned / anonymous / higher-order lifetime parameter, much like that of ::std::thread::scope().

That way, it would be illegal to smuggle out the transformed value through some out variable such as my smuggling_channel example.

Complementarily, one can also set up a abort-on-unwind guard around that callback, which is yet another advantage of these scoped APIs.

The API in question would boil down to:

  1. Add a lifetime-infecting, and invariant (effectively lifetime branding) parameter to our type:

    pub
    struct ExposedSecret<'brand, T, const MEC: usize, const EC: usize>(
        T,
        ::core::marker::PhantomData<fn(&'brand ()) -> &'brand ()>,
    );
    
  2. write our scoped API:

    pub
    fn expose_secret(
        self: Secret<T, MEC, EC>,
        scope: impl FnOnce(ExposedSecret<'_, T, MEC, EC>)
                        -> ExposedSecret<'_, T, MEC, EC>
        ,
    ) -> Secret<T, MEC, { EC + 1 }>
    where
        generic_const_predicate!(MEC > EC) :,
        Secret<T, MEC, { EC + 1 }> :,
    {
        let yield_ = scope;
        let witness = yield_(
            ExposedSecret(self.0, <_>::default())
        );
        Secret::new(witness.0)
    }
    
  3. Usage:

    fn demo<T>(s: Secret<T, 2, 0>)
    {
        let s = s.expose_secret(|exposed| {
            // ...
            exposed // <- Give it back
        });
        let s = s.expose_secret(|exposed| {
            // ...
            exposed // <- Necessary!
        });
        // let s = s.expose_secret(|exposed| { // <- Error!
        //     // ...
        //     exposed.into_secret()
        // });
    }
    

It effectively boils down to that } callback-scope-closing brace acting as an "implicit" .expose_secret(), which cannot be skipped.


Note that depending on your use case, you could then simply expose a &T in that closure, and it would be just as simple:

pub
fn expose_secret<Ret>(
    self: Secret<T, MEC, EC>,
    scope: impl FnOnce(&T) -> Ret,
) -> (Secret<T, MEC, { EC + 1 }>, Ret)
where
    generic_const_predicate!(MEC > EC) :,
{
    let yield_ = scope;
    let ret = yield_(
        &self.0
    );
    (Secret::new(self.0), ret)
}

in this design, there would simply be no ExposedSecret at all, since the scoped API already conveys all the necessary info, alone.

7 Likes

Potential alternative approach, but completely untested, using an additional "key" type:

struct Secret<T, const EC: usize> { … }
struct Key<T, const EC: usize> { _priv: (), }

impl<T, const EC: usize> Secret<T, EC> {
    fn expose(self) -> (Secret<T, {EC+1}>, Key<T, EC>) { … }
    fn peek(&self, _: Key<T, {EC-1}>) -> &T { … }
}

The limitation of this approach as written is that any Key<T, N> can be used to open any Secret<T, {N+1}>, potentially multiple times if you have multiple keys. (Carrying EC into Key is to minimize this somewhat.) The solution to that is again to apply some sort of brand, so that the generated key is tied to the secret it came from.

For more on lifetime brands, see the crates indexing (the original), generativity (my crate, and an attempt to document the pattern, alongside a way to create branded lifetimes with a macro instead of a closure), and qcell (compile time RwLocks using various type level tricks, including generative lifetimes).


The simplest approach would be to just create the appropriate number of Keys when creating the Secret.

2 Likes

Good day to you @Yandros, thank you for your extremely well-thought and brilliant response.

To be honest, I thought of having expose_secret(self, impl FnOnce(&T)) -> Secret<T, MEC, { EC + 1}> {...}. The problem I faced back then was I didn't know how to make use of &T within the closure. My naive/inexperienced resolution (or at least that was how I imagined how my API could be used) was to capture a &mut Something (Something could be a REST API client that needs to establish a connection using a secret API key etc.) and within the closure, to dereference the mutable reference of Something and 'copy' or 'clone' T into Something. This then requires Something to be default-constructible (I cannot leave a variable declared but uninitialized) and T: Clone minimally, which is an unwieldy API. Which is why I then wanted to return the tuple, i.e. (Secret<T, MEC, { EC + 1 }>, SecretGuard<'secret, T, { EC + 1} so that I maintain the type invariance (EC < MEC) but I face with the lifetime issue and hence this post.

Your second approach of making expose_secret generic over Ret and returning the tuple with value of type Ret is a way to allow the caller to make use of the &T without the constraints of say Something (e.g. an API client that requires a secret API key etc.) must be default constructible etc.

My naive approach with callback definitely didn't take into account of internal panic within the closure and subsequent recover from it to retrieve the inner value of type T, which is a great 'vulnerability'.

You have definitely helped me with avoiding this vulnerability.

Just to make sure if I understand this correctly, how does adding a branded lifetime, i.e., adding a PhantomData with a 'brand lifetime, helps with avoiding the panic-recover vulnerability?

My thought is that it is avoided with two mechanisms, the first is with the lifetime annotation and second with the requirement to return a ExposeSecret<'brand, ...>. Let me know if the following point is accurate.

  1. Since the ExposeSecret<'brand, ...> lives at most 'brand, the closure itself is scoped for lifetime at most 'brand, and any panic-recovery will bring a 'transformed' value out of the scope of the closure, outliving 'brand and hence is rejected by the compiler.

My second question with regards to your first approach with ExposedSecret<'brand, ...> is although ExposedSecret<...> implements std::ops::Deref, how does the caller of the expose_secret(...) method uses the value of type ExposedSecret<...> to obtain a value of another type, e.g. an API client where the underlying secret is an API key etc.?

Lastly, both of approaches are very clever because you don't have to implement std::ops::Drop as the inner secret value of type T is moved from Secret to ExposedSecret, then to Secret and so on.

FINALLY, on your comment with respect to using ::typenum, that is in my plan; it is only for the ease of development that I am using the unstable feature for now.

Another question (sorry for the long post): will using of typenum makes the eventual API slightly unwieldy? Since caller will now have to use e.g. type U5 instead of just a const usize 5. What would you recommend to circumvent this slight inconvenience?

Final question: In your kind implementation,

    pub fn expose_secret<ReturnType>(
        self,
        scope: impl FnOnce(ExposedSecret<'_, T, MEC, EC>) -> (ExposedSecret<'_, T, MEC, EC>, ReturnType),
    ) -> (Secret<T, MEC, { EC + 1 }>, ReturnType)
    where
        generic_const_predicate!(MEC > EC):,
        Secret<T, MEC, { EC + 1 }>:, // why specify this bound?
    {
        let yield_ = scope;
        let (witness, returned_value) = yield_(ExposedSecret(self.0, <_>::default()));
        (Secret::new(witness.0), returned_value)
    }

Why did you specify this bound: Secret<T, MEC, { EC + 1 }>:,?

I now have a semi-working implementation of Secret using the typenum crate.

use typenum::{
    assert_type,
    consts::{U0, U1},
    op,
    type_operators::IsLess,
    Bit, IsGreater, Same, True, Unsigned, B0, B1,
};

pub type AddU1<A> = <A as core::ops::Add<U1>>::Output;

pub struct Secret<
    T,
    MEC: Unsigned,
    EC: core::ops::Add<typenum::U1> + typenum::IsLess<MEC> + Unsigned = U0,
>(
    T,
    core::marker::PhantomData<MEC>,
    core::marker::PhantomData<EC>,
);

pub struct ExposedSecret<'brand, T, MEC: Unsigned, EC: Unsigned>(
    T,
    ::core::marker::PhantomData<fn(&'brand ()) -> &'brand ()>,
    ::core::marker::PhantomData<MEC>,
    ::core::marker::PhantomData<EC>,
);

impl<T, MEC: Unsigned> Secret<T, MEC, U0>
where
    U0: IsLess<MEC>,
{
    #[inline(always)]
    pub fn new(value: T) -> Self {
        Self(value, <_>::default(), <_>::default())
    }
}

impl<T, MEC: Unsigned, EC: core::ops::Add<typenum::U1> + Unsigned + typenum::IsLess<MEC>>
    Secret<T, MEC, EC>
{
    const ASSERT_EC_LESS_THAN_MEC: () = assert!(<<EC as IsLess<MEC>>::Output as Bit>::BOOL);
    #[inline(always)]
    pub fn expose_secret<ReturnType>(
        self,
        scope: impl FnOnce(ExposedSecret<'_, T, MEC, EC>) -> (ExposedSecret<'_, T, MEC, EC>, ReturnType),
    ) -> (Secret<T, MEC, AddU1<EC>>, ReturnType)
    where
        AddU1<EC>: core::ops::Add<typenum::U1> + Unsigned + typenum::IsLess<MEC>,
    {
        Self::ASSERT_EC_LESS_THAN_MEC;
        let (witness, returned_value) = scope(ExposedSecret(
            self.0,
            <_>::default(),
            <_>::default(),
            <_>::default(),
        ));
        (
            Secret(witness.0, <_>::default(), <_>::default()),
            returned_value,
        )
    }
}

impl<T, MEC: Unsigned, EC: Unsigned> ::core::ops::Deref for ExposedSecret<'_, T, MEC, EC> {
    type Target = T;

    #[inline(always)]
    fn deref(&self) -> &T {
        &self.0
    }
}

Cargo.toml

[dependencies]
typenum = "1.17.0"
zeroize = "1.6.0"

I have to use this const ASSERT_EC_LESS_THAN_MEC: () = assert!(<<EC as IsLess<MEC>>::Output as Bit>::BOOL); even though it is ugly as const operations on generic type parameter is not on stable Rust.

If you're using typenum, you don't need the associated const assert, you want a bound of EC: IsLess<MEC, Output=True>. If you're going to use an assert, use assert_type!(op!(EC < MEC)).

I tried using assert_type!(op!(EC < MEC)) but it doesn't exist within the scope of expose_secret(...).

Will try IsLess<MEC, Output=True>.

Right now, I am trying to translate:

impl<
        T: Zeroize,
        MEC: Unsigned,
        EC: core::ops::Add<typenum::U1> + Unsigned + typenum::IsLess<MEC>,
    > ExposeSecret<'_, T, MEC, EC> for Secret<T, MEC, EC>
{
    type Exposed<'brand> = ExposedSecret<'brand, T, MEC, EC>;

    #[inline(always)]
    fn expose_secret<ReturnType>(
        self,
        scope: impl FnOnce(ExposedSecret<'_, T, MEC, EC>) -> (ExposedSecret<'_, T, MEC, EC>, ReturnType),
    ) -> (Secret<T, MEC, AddU1<EC>>, ReturnType)
    where
        AddU1<EC>: core::ops::Add<typenum::U1> + Unsigned + typenum::IsLess<MEC>,
    {
        Self::ASSERT_EC_LESS_THAN_MEC;
        let (witness, returned_value) = scope(ExposedSecret(
            self.0,
            <_>::default(),
            <_>::default(),
            <_>::default(),
        ));
        (
            Secret(witness.0, <_>::default(), <_>::default()),
            returned_value,
        )
    }
}

Into an ExposeSecret trait method, I am writing the trait which is very difficult to write:

use crate::secret::{AddU1, Secret};
use typenum::{consts::U1, Bit, IsLess, Unsigned};
use zeroize::Zeroize;

pub trait ExposeSecret<
    'lt,
    T: Zeroize,
    MEC: Unsigned,
    EC: core::ops::Add<typenum::U1> + Unsigned + typenum::IsLess<MEC>,
>
{
    const ASSERT_EC_LESS_THAN_MEC: () = assert!(<<EC as IsLess<MEC>>::Output as Bit>::BOOL);
    type Exposed<'a>;

    fn expose_secret<ReturnType>(
        self,
        scope: impl for<'a> FnOnce(Self::Exposed<'a>) -> (Self::Exposed<'a>, ReturnType),
    ) -> (Self, ReturnType)
    where
        AddU1<EC>: core::ops::Add<typenum::U1> + Unsigned + typenum::IsLess<MEC>;
}

This is giving me the error of

error[E0582]: binding for associated type `Output` references lifetime `'a`, which does not appear in the trait input types
  --> src\traits.rs:17:58
   |
17 |         scope: impl for<'a> FnOnce(Self::Exposed<'a>) -> (Self::Exposed<'a>, ReturnType),
   |                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

GitHub Repo

My bad; these kind of bounds used to be necessary, IIRC (or at least necessary in somewhat similar situations; probably when the type in question appears in the fn body but not the signature), so I added it (overly) preëmptively. Later on, in other parts of the code, I reälized they weren't needed, as they seem to be inferred by the very existence of the function signature naming these types, so I skipped them; and I think I even tried to go back and edit these bounds out in remaining places, but it is likely I forgot about the one you pointed out :smile:

Ah yeah, assert_type! won't capture generics. To bridge from real const generics, typenum provides the generic_const_mappings module that allows you to write U<N> to go from const generic to typenum, but note that you're still not allowed to project from a const generic N in type bounds. You can sometimes manage to get things to work with just bounds directly on the generics, but the sooner you get everything into the typenum world the easier it is.

I was (a bit nerd sniped and thus) able to get to an I think working solution: [playground]

pub trait ExposeSecret<'max, T, MEC: Unsigned, EC: Unsigned>: Sized {
    type Exposed<'brand>
    where
        'max: 'brand;

    type Next: ExposeSecret<'max, T, MEC, op![EC + U1]>
    where
        EC: Add<U1>,
        op![EC + U1]: Unsigned;

    fn expose_secret<F, R>(self, scope: F) -> (Self::Next, R)
    where
        EC: Add<U1> + IsLess<MEC, Output = True>,
        op![EC + U1]: Unsigned,
        F: for<'brand> FnOnce(Self::Exposed<'brand>) -> R;
}

impl<'a, T: Zeroize, MEC: Unsigned, EC: Unsigned> ExposeSecret<'a, &'a T, MEC, EC>
    for Secret<T, MEC, EC>
{
    type Exposed<'brand> = Exposed<'brand, &'brand T>
    where
        'a: 'brand;

    type Next = Secret<T, MEC, op![EC + U1]>
    where
        EC: Add<U1>,
        op![EC + U1]: Unsigned;

    fn expose_secret<F, R>(self, scope: F) -> (Self::Next, R)
    where
        EC: Add<U1> + IsLess<MEC, Output = True>,
        op![EC + U1]: Unsigned,
        F: for<'brand> FnOnce(Exposed<'brand, &'brand T>) -> R,
    {
        let result = scope(Exposed(&self.0, PhantomData));
        (Secret(self.0, PhantomData), result)
    }
}

It's tricky to write because there's a lot of subtle interactions of things like implied bounds involved. E.g. if I switch the impl to use FnOnce(Self::Exposed<'brand>) instead, which you might think would be equal, we get the unhelpful error of just:

error[E0478]: lifetime bound not satisfied
  --> src/lib.rs:39:8
   |
39 |     fn expose_secret<F, R>(self, scope: F) -> (Self::Next, R)
   |        ^^^^^^^^^^^^^
   |
note: lifetime parameter instantiated with the lifetime `'a` as defined here
  --> src/lib.rs:27:6
   |
27 | impl<'a, T: Zeroize, MEC: Unsigned, EC: Unsigned> ExposeSecret<'a, &'a T, MEC, EC>
   |      ^^

When running into "bleeding edge" features with subpar errors, the only (and necessary) trick is to simplify. Thus not using impl Trait here. E.g. while getting to this impl, I hit that error and entirely commented out the expose_secret function as a reduced test, and then got shown a different error which needed to be fixed first for any hope of fixing the method to actually work.

But with that written, I need to ask the question: do you need to make this a trait? Unless you're a) going to have more than one implementation of the trait and b) going to write code generic over the trait to work on all of such types, you shouldn't be using a trait; using traits where they aren't necessary just serves to make things more complicated for no benefit. See also matklad on concrete abstraction as a code smell.

1 Like

First and foremost, thank you for spending your time in writing these codes for me, it is beautiful and brilliant!

Second, not withstanding my admiration for you, I hope to become like you one day! I would really love to know how did you get to your current coding competency, and especially in Rust. It is beautiful in how you're able to write the trait ExposeSecret, I didn't know you can have where clauses tied to a type alias. And the type alias Next is just pure brilliance again.

1 Like

Past a point, it really does just come from experience and pattern matching "I want to do this" against a corpus of "I've done that before."

Associated type on the trait, not a type alias. If you put bounds on an actual type alias, it'll compile but warn you that the bounds are ignored. Associated types are similar to but a distinct concept to type aliases.

Basically: using an alias is exactly[1] equivalent to directly using the aliased item. An associated type is transparent like an alias on known types, but is opaque when projected from a type parameter (or other opaque type, such as impl Trait), only permitting usage of the provided bounds (which must be fulfilled by implementors).


  1. some caveats apply, e.g. only some usage locations are aliased ↩︎

1 Like

Hi @CAD97, thank you for your response.

Would you also mind elaborating how did you thought of 'max: 'brand, like how did it cross your mind that the trait itself should have a lifetime annotation 'max that lives longer than that of the associated type itself, i.e. 'brand?

Thank you.

Hi @CAD97, I have noticed that for your trait bound in the F closure type,

F: for<'brand> FnOnce(Self::Exposed<'brand>) -> R;

Now the closure does not return Self::Exposed<'brand>, this is different from the original implementation Link to original impl. Will that mean that the exposed value will still not be transported out illegally via a panic-recovery attack?

I have also implemented Drop on Secret and wrapped T around ManuallyDrop because I would want T to be only dropped when EC == MEC and the zeroize crate's .zeroize trait method on trait Zeroize to be called prior to T's dropping.

Does the SAFETY notes on L101 and L60 make good sense to you?

Also added your name on the Credits section of the README.md because, lol, you basically wrote the entire crate... :smiley:

If you would like, I don't mind giving you full access to the repo.

Hi @CAD97,

Thank you for providing an implementation of ExposeSecret trait some time ago. I realise for branded invariant types, there is still a way to 'steal' the references out of the scope by making the value, whose reference is taken from, static.

Consider this:

use sosecrets_rs::{prelude::*, traits::ExposeSecret};
use std::sync::OnceLock;
use typenum::consts::U2;

struct AStruct {
    _inner: Vec<i32>,
}

struct ReferenceWrapper<'a> {
    _inner: Option<&'a AStruct>,
}

fn make_return_secret<'a>(a_struct_: &'a AStruct) -> Secret<&'a AStruct, U2> {
    Secret::new(&a_struct_)
}

fn give_69_vec() -> Vec<i32> {
    vec![69]
}

fn main() {
    static A_STRUCT: OnceLock<AStruct> = OnceLock::new();

    let secret_astruct = make_return_secret(&A_STRUCT.get_or_init(|| AStruct {
        _inner: give_69_vec(),
    }));

    let mut ref_wrapper = ReferenceWrapper { _inner: None };

    let (_, _) = secret_astruct.expose_secret(|exposed_secret| {
        let _ = ref_wrapper._inner.insert(*exposed_secret);
    });

    assert_eq!(ref_wrapper._inner.take().unwrap()._inner, vec![69]);
}

The above actually compiles. Effectively stealing the secret out of the scope with invariant lifetime.