Extracting Fn's args

I'm trying to port some C++ libraries I use to Rust however I'm having a number of issues, the one hitting me today now is that I, in short, have a function that needs to be able to accept something 'callable' (in C++ this was often a function object or lambda) and the function via templates built up what is passed in to the function based on what the function accepts. In trying to translate this to rust (vastly simplified of course) I'm trying to do something like this (pretend that ... is variadic like variadic templates in C++, although the C++ version of this doesn't even use variadic templates as we can extract the argument types from the function type directly):

fn run<F: FnMut(Args...) -> Ret, Ret, Args: (...)>(&self, cb: F) -> Ret {
  cb(self.id, self.extract<Args>...)
}

And could thus be called like (the C++ version auto acquires things as const and all as necessary based on the function types and the lifetime of a reference return should be set accordingly to self):

let result = context.run(|id: usize, a_ref: &Thing, a_mut_ref: &mut AnotherThing| {
  a_mut_ref.do_something();
  (id, a_ref)
});

If I can solve this problem generically then it would work for the actual case (which is acquiring a RwLock set of types and iterating over the entire set).

I've tried a number of things including passing in the needed types as a cons list as a generic argument and all but I'm having issues at some step or another no matter which pattern I try.... Any thoughts as to something that could work that I can try pursuing next?

One very simple answer is to just always give the closure everything. Then the closure can toss what it doesn't need. If this doesn't involve any locks before calling the closure, this should fairly easily optimize to not passing the extra arguments (since generic functions are monomorphic).

One very complex answer is to (ab)use frunk and its type level hlist sculpting to get the behavior you want. It definitely won't be pretty, but I think it should be possible to set it up so you can accept a callback consisting of an HList of any (sub)permutation of a set of types, and generically populate an HList with the correct types to invoke the callback.

So, for fun / challenge more than anything else, I've tried to implement this:

  • it requires unstable features related to closure traits (+ a never type feature now that we are at it, but that one isn't essential),

  • an upper bound over the number of parameters you want to support (including the usize), so that we can use macros to handle the finite number of cases.

  • A way to implement your .extract() logic: this can be as challenging as your question to begin with. I have circumvented the issue by restraining my self to tuples that implement Default, and used that trait to thus forge such tuple out of thin air.

#![feature(
    fn_traits, unboxed_closures,
)]

struct Foo {
    id: usize,
}

impl Foo {
    fn extract<Args : Tuple> (&mut self)
      -> Args
    where
        Args : Default, // my work around for this impl
    {
        Args::default()
    }

    fn run<F, Ret, Args> (self: &'_ mut Self, mut cb: F) -> Ret
    where
        Args : Tuple,
        F : FnMut<Args, Output = Ret>,
        Args::Rest : Tuple<WithPrependedUsize = Args>,
        Args::Rest : Default, // needed for my extract impl
    {
        let id = self.id;
        cb.call_mut(
            self.extract::<Args::Rest>()
                .prepend_usize(id)
        )
    }
}

fn main ()
{
    let mut foo = Foo { id: 42 };
    foo.run(|id: usize, s: String| {
        dbg!(id, s)
    });
}

trait Tuple {
    type Head;
    type Rest : Tuple;
    fn cons (self: Self) -> (Self::Head, Self::Rest)
    ;
    
    type WithPrependedUsize;
    fn prepend_usize (self: Self, _: usize) -> Self::WithPrependedUsize
    ;
}

// implemented with a macro for (T0, T1, ...) tuples of up to 10 elements.

Actually I have managed to simplify it to work on stable:

struct Foo {
    id: usize,
}

trait FnMutHelper<FirstArg, OtherArgs> {
    type Ret;
    
    fn call_mut (
        self: &'_ mut Self,
        _: FirstArg,
        _: OtherArgs,
    ) -> Self::Ret;
}

impl Foo {
    fn extract<Args : Default> (&mut self)
      -> Args
    {
        Args::default()
    }

    fn run<F, OtherArgs> (self: &'_ mut Self, mut cb: F)
      -> F::Ret
    where
        F : FnMutHelper<usize, OtherArgs>,
        OtherArgs : Default,
    {
        cb.call_mut(
            self.id,
            self.extract::<OtherArgs>()
        )
    }
}

fn main ()
{
    let mut foo = Foo { id: 42 };
    foo.run(|id: usize, s: String| {
        dbg!(id, s)
    });
}

// use macros to derive FnMutHelper from FnMut(Args...)

But more seriously, I think it is better if you just require the users to call your foo with a macro:

run!(foo, |id, s: String, b: bool| ...)

that will just convert the given closure to

foo.run(|id, (s, b): (String, bool)| ...)

Since the .run() method for the latter is "trivial" to implement:

struct Foo {
    id: usize,
}

impl Foo {
    fn extract<Args : Default> (&mut self)
      -> Args
    {
        Args::default()
    }

    fn run<F, Ret, OtherArgs> (self: &'_ mut Self, mut cb: F)
      -> Ret
    where
        F : FnMut(usize, OtherArgs) -> Ret,
        OtherArgs : Default,
    {
        cb(self.id, self.extract::<OtherArgs>())
    }
}

#[macro_export]
macro_rules! run {
    (
        $foo:expr, |$id:tt $(: usize)? $(, $pat:tt $(: $T:ty)?)* $(,)?|
            $($body:tt)*
    ) => (
        $foo.run(
            |
                $id : usize,
                ($($pat,)*) : ($($crate::run!(@first $($T)? _),)*),
            | $($body)*
        )
    );
    
    (@first
        $first:tt $($snd:tt)?
    ) => (
        $first
    );
}

fn main ()
{
    let mut foo = Foo { id: 42 };
    run!(foo, |id, s: String, b: bool| {
        dbg!(id, s, b)
    });
}
1 Like

The issue I'm having is that each thing to pass in is expensive to acquire (involving locks and a double indexed array lookup) and this is a very hot zone (hence why the actual code will call the callback many many times until it exhausts the storage).

That's one of the methods I've tried though in iterator form (though manually made it before I learned about frunk, modeled on my old pre-C++11 recursive template horrors), however I kept having lifetime issues, specifically the next function in the Iterator:

	fn next(&mut self) -> Option<Self::Item> {
		if self.remaining == 0 {
			None
		} else {
			let ret = Some((
				self.entities.direct[self.entities.direct.len() - self.remaining],
				self.pools.get_ret_value_from_end(self.remaining),
			));
			self.remaining -= 1;
			ret
		}
	}

Kept having this error:

error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting requirements
   --> src/entity/registries/group.rs:479:16
    |
479 |                 self.pools.get_ret_value_from_end(self.remaining),
    |                            ^^^^^^^^^^^^^^^^^^^^^^
    |
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 472:2...
   --> src/entity/registries/group.rs:472:2
    |
472 |       fn next(&mut self) -> Option<Self::Item> {
    |  _____^
473 | |         if self.remaining == 0 {
474 | |             None
475 | |         } else {
...   |
486 | |         }
487 | |     }
    | |_____^
note: ...so that reference does not outlive borrowed content
   --> src/entity/registries/group.rs:479:5
    |
479 |                 self.pools.get_ret_value_from_end(self.remaining),
    |                 ^^^^^^^^^^
note: but, the lifetime must be valid for the lifetime `'s` as defined on the impl at 453:3...
   --> src/entity/registries/group.rs:453:3
    |
453 |         's,
    |         ^^
note: ...so that the types are compatible
   --> src/entity/registries/group.rs:479:16
    |
479 |                 self.pools.get_ret_value_from_end(self.remaining),
    |                            ^^^^^^^^^^^^^^^^^^^^^^
    = note: expected  `entity::registries::group::ComponentListLockedPools<'_, EntityType>`
               found  `entity::registries::group::ComponentListLockedPools<'s, EntityType>`

And I don't know where that expected entity::registries::group::ComponentListLockedPools<'_, EntityType> is coming from as the get_ret_value_from_end return value. The get_ret_value_from_end function is on a 'HList' style trait, which is what self.pools is (pools: ComponentTypes::AsLockedComponentPools, where ComponentTypes: ComponentListAsLockedComponentPools<'s, EntityType> and the 's is the lifetime from the main struct that creates this iterator via iter).

Basically every place anywhere in the 'hlist' style system (in reality it's just a Cons list of () as the nil and (Head, Tail) where Head is any type and Tail is the type level Cons list) where ComponentListLockedPools is defined, it is defined with 's as it's lifetime, passed in throughout the whole structure. get_ret_value_from_end is:

pub trait ComponentListLockedPools<'s, EntityType: Entity + 'static>: ComponentList {
	type AsRetValue: 's;
	fn get_ret_value_from_end(&'s self, remaining: usize) -> Self::AsRetValue;
}

impl<'s, EntityType: Entity + 'static> ComponentListLockedPools<'s, EntityType> for () {
	type AsRetValue = ();

	fn get_ret_value_from_end(&'s self, remaining: usize) -> Self::AsRetValue {
		()
	}
}

impl<
		's,
		EntityType: Entity + 'static,
		StorageData: CoLockType<'s> + 'static,
		TAIL: ComponentListLockedPools<'s, EntityType>,
	> ComponentListLockedPools<'s, EntityType> for (StorageData, TAIL)
{
	type AsRetValue = (StorageData::CoRetType, TAIL::AsRetValue);

	fn get_ret_value_from_end(&'s self, remaining: usize) -> Self::AsRetValue {
		unimplemented!()
	}
}

(It used to have code instead of unimplemented but I'd been pruning out stuff in an attempt to find out where the lifetime is '_ from...)

I'm trying to restrict myself to stable and #![forbid(unsafe_code)] for this library, the C++ version of it is like 40% template magic so I considered it a good learning opportunity to convert to Rust. ^.^

An upper bound is definitely a no-go for me, otherwise I'd just use flat tuples. I don't want to restrict the API further from what the C++ can unless the C++ version of the API is unsafe in some way and I can't really guess just how many possible things will be passed in as that's entirely user controlled.

Optimally I'd prefer to return an iterator, I'm only falling back to a function being passed in (which is how the C++ version actually does it) because I couldn't figure out the above lifetime issues. I've thought of a dozen ways to do it with both macro calls and even better proc-macro's to generate the optimal code as a hard-wired set, but all of that is very noisy scaffolding in comparison and makes it far more difficult to handle some internals (which are actually able to generate it all dynamically to optimize the access but then you can't access the actual types of things that way, it's for other purposes rather than this main interface).


In short I've gathered that extracting Fn's arguments from the type's is a no-go. ^.^;

Any thoughts about the iterator interface then? Excepting that above lifetime issue, basically I'm optimally wanting to be able to call it like:

group.iter::<(&SomeType, &mut AnotherType, ...)>().for_each(|(id: EID, some: &SomeType, other: &mut AnotherType, ...)| {
});

However so far the best I've managed to get it so far is this (with the above lifetime issue but otherwise works if I compile out that line, just it returns an empty tuple then):

group.iter::<CL!(CoRead<SomeType>, CoWrite<AnotherType>, ...)>().for_each(|(id: EID, cl!(some, other, ...): CL!(RwLockReadGuard<SomeType>, RwLockWriteGuard<AnotherType>, ...))| { // Simplified the `CL` expression for brevity here, usually can be elided anyway
});

Which I don't find too bad, the CL and cl macro's just make my above mentioned Cons list of tuples as either types or data respectively (so CL!(CoRead<SomeType>, CoWrite<AnotherType>) becomes (CoRead<SomeType>, (CoWrite<AnotherType>, ())) and so forth), however I'm paranoid something else is wrong if I could get the main code path to work... >.>

I may have just had a realization, I think the call of get_ret_value_from_end itself is expecting the :_ in it's own self even though self.pools is 's... but how... o.O

It looks like the self coming from the Iterator's next call is somehow erasing the 's from the self.pools?!

Adding this to the Iterator implementation doesn't do anything either:

where
	Self: 's,

You might want to take a look at how specs and other ecs libraries handle this, as iiuc that is similar to what you're doing.

1 Like

Already checked, specs is via macro's to generate a predefined set of tuples, and legion is via a lot of unsafe.

I haven't looked in detail into your iterator suggestion, but you may be hitting the limitations of Rust's Iterator API which was designed to yield elements to be collected and thus mutable exclusive references from two .next() calls cannot overlap.

If a more "lazy" iterator (i.e., one that cannot yield a new element until you dispose of the previous one) matches your need, maybe the streaming-iterator crate can help you.

Wise person :slightly_smiling_face:

1 Like

I've been leaning to this same thought so I think you're right. The addition of GAT support in Rust would also be intensely useful, plus of course generic-length tuples. The loss of itertools on the streaming iterator crate would be saddening though, lol.

This is a learning project, it may never actually see the light of day, but it has been exceptional at helping me translate my decades of C++ knowledge to Safe Rust. ^.^

1 Like

Yep, definitely need GAT's for this bit to do it correctly. I'm just trying a simplified test of making an iterator for only one pool rather than 'multiple-user-chosen' and I need to be able to do type Blah<'a>: ...;, I keep getting stopped at so many places by lack of GAT's... >.>

It might be time to break my 0-dependency streak, Streaming Iterator might be needed, unless I remake the parts of it I need in it but eh.... No for or itertools or so on it either...

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.