Type of closure

Hi all,

I have faced with the issue that when I try to save in static variable object that type depend on lambda it is not possible save it:

thread_local! {
    static STRATEGY: Cell<Strategy<impl FnMut(&Vec<&'static OtherObject>) -> Option<JoinHandle<()>>>> = {
        Cell::new(Strategy::new(|obj| {
                None
        }))
    };
}

It does not work, because I cannot use impl in variable declaration ...
Bu I do not know type of closure !!

How to fix this code ... ?

I have found the #![feature(impl_trait_in_bindings)] ... But than I got the following error:

expected struct `std::cell::Cell<gc::Strategy<impl for<'r> std::ops::FnMut<(&'r std::vec::Vec<&'static gc::OtherObject>,)>>>` (opaque type at <src\gc.rs:605:36>)
   found struct `std::cell::Cell<gc::Strategy<impl for<'r> std::ops::FnMut<(&'r std::vec::Vec<&'static gc::OtherObject>,)>>>` (opaque type at <src\gc.rs:605:36>)

The closure does not have a name you can write down, and note that every closure has a different type, so even if you could specify the type, you couldn't replace it with another closure, because it would have a different type.

What are you trying to do? Perhaps a function pointer or trait object would do the job?

1 Like

I am trying to save functional object in Strategy and than call it ...
Functional object has concrete signature, but I do not what to use memory allocation for it ...
It would be nice if it'll be possible to save functional object without references

If you want to store the function, you need to describe its type somehow. Here are three approaches you might be able to use:

// Function pointer.
fn(&[&'static OtherObject]) -> Option<JoinHandle<()>>

// Trait object.
Box<dyn FnMut(&[&'static OtherObject]) -> Option<JoinHandle<()>>>

// Make custom trait.
trait MyFnTrait { ... }
struct MyFnStruct { ... } // you can now name the struct
impl MyFnTrait for MyFnStruct { ... }

I changed them to take slices instead of vectors. That's a minor nit, but slices are always better than immutable references to vectors.

I don't think there are other options than these.

1 Like

If you are storing it in a thread_local your closure can't really capture any non-'static state, so if that's the case you can just use a function pointer (e.g. fn(&Vec<&'static OtherObject>)) instead of impl FnMut(&Vec<&'static OtherObject>).

Closures which don't capture anything will also coerce to function pointers, so by just changing the type to Cell<Strategy<fn(&Vec<&'static OtherObject>)>> it should compile.

You don't need to worry about allocations because we aren't using a Box<dyn FnMut(...)>. I also wouldn't be surprised if LLVM's constant propagation pass was smart enough to "see through" the function pointer and allow the closure to be inlined, avoiding the overhead of dynamic dispatch.

2 Likes

Pointer to function helped, but what if I want to put in Strategy closure with capture ?

In that case I got the following error:

note: expected fn pointer `for<'r> fn(&'r std::vec::Vec<&'static gc::OtherObject>) -> std::option::Option<std::thread::JoinHandle<()>>`
                  found closure `[closure@src\gc.rs:603:33: 606:10 gr:_]`

I want something like std:function<...> in C++ ...
It could be done by using impl in variable declaration, but it does not work ...

You have to use a box to do that. The closure can have many different sizes if you capture environment, but the thread local has a fixed size.

Can you give me an example of such a closure?

A thread-local variable behaves like a normal static for the purposes of lifetime checking, so the when it is constructed the only things it can refer to are are other statics... Which you'd do directly by name instead of trying to carry it along as a closed over variable.

1 Like

std::function in C++ does use dynamic allocation and dynamic dispatch in the general case, because it is type erased (i.e. it's roughly equivalent to Box<dyn Fn*>).

3 Likes

How I usually model a strategy is using an enum to allow for selection, and I put the actually executable code not in a closure, but a method (which may accept an enum variant as an argument, or own it in a field, depending on use case) that performs a match on the strategy enum variant, and executes the corresponding code.

Then, by deriving Clone for the strategy enum, we can make it cheap and easy to pass around the information on which strategy to invoke, without having to pass around the executable code all the time which is bound to be more difficult due to borrow checker rules.

Let's go step by step over your request.

Trait bound

For starters, let's look at the function signature you requested:

  1. &Vec<T>
    offers no useful API w.r.t, the slice it can refer to, &[T]; so let's use that and be more general (for instance, we may avoid the heap-allocation that creating a non-empty Vec requires).

  2. &'_ ... &'static T
    is, similarly, not better than the more general &'_ ... &'_ T, unless you intend to store those &'static T somewhere during the call (unlikely). So let's use the latter.

  3. static ... FnMut
    is an oxymoron: static implies sharing (& access), FnMut requires lack thereof (&mut exclusive access).

  4. static ... + 'static
    A static, even a "thread-local one", may live arbitrarily long, thus the values it holds must be safe to hold that long. Such property is expressed as that of having a 'static bound.

    For instance:

    • &'static _ : 'static
      An infinitely-long borrow of a value can be held infinitely long.

    • String : 'static
      Although not all String live infinitely long (quite the opposite, actually), one that you hold will live until you drop it. Thus, a String can be held infinitely long.

    • ditto for all non-generic types.

    Counter examples:

    • &'a _
      for some lifetime 'a which is unknown, universal, or referring to a local variable, cannot be held infinitely long.
      Indeed, past the end of the lifetime, the variable the reference refers to may no longer exist, thus leading to a dangling pointer.

      • let s = String::from("Hello!"); // some local
        let at_s: &'_ String = &s; // refers to a local => not 'static
        

    So, in our case, we need that the closure (environment / captures) be 'static.

This leads us to the now fixed closure signature / API / interface / trait bound:

Fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
+ 'static

Making a type out of a trait bound

We have, up until now, decided what the trait bound / the contract of the held type should be.

Now remains the question of what should be the type of some implementor of the contract we wish to hold.

We have, for instance, a very simple implementor of closure API: good ol' function( pointer)s, with their lack of captured environment:

// the type of function pointers / capture-less closures...
fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
// implements...
:
// our required API:
Fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
+ 'static

And now we have our first working solution!

thread_local! {
    static STRATEGY
        : fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
        = |objs| {
            println!("{}", objs.len());
            None
        }
    ;
}

fn main ()
{
    STRATEGY.with(|strategy| {
        strategy(&[]);
        strategy(&[&OtherObject,]);
    });
}

From this point, there are three possibilities:

  • You are satisfied with the situation, since you do not need the closures to capture/use an environment :white_check_mark:

  • You need to capture an environment, but realise that a global function can use its own private globals as an environment: Example :white_check_mark:

  • Since using global variables isn't great, you are thinking about refactoring the code into not using globals anymore, so you need a more general solution, while still having the aforementioned trait bounds constraints :arrow_right: keep reading

Other implementors of the trait bound

So, we have seen that function pointers are a concrete / nameable type, easy to construct, and which implements the given trait bounds; all thanks to its not requiring any environment.

But what happens when we need some environment / state for our closure?

In that case, we have two possibilities:

  • "static" (compile-time) / exact / fixed knowledge of the type (and thus size and layout) of our implementor.

  • dynamic closure type (i.e., the exact type (and thus size and layout) of the closure is "not known" at compile-time, so as to be able to, for instance, swap an instance of one type with another of another type, etc.)

dyn-amic closure type

This the (in)famous dyn <(object-safe) trait bounds> type.

So, in our case, the type is:

dyn 'static + Fn(&'_ [&'_ OtherObject]) -> Option<...>

such a type, as mentioned above, may have instances of different sizes and layouts (we say the type is not Sized). Thus, using them requires (pointer) indirection:

  • &'_ mut (dyn ...)
    Indirection by exclusive reference / borrow,

    • Too restrictive in our case, since the Fn trait only requires shared &self access :x:
  • &'_ (dyn ...)
    Indirection by shared reference / borrow,

    • Given our 'static requirement, this implies the concrete &'static (dyn ...), which, realistically, isn't gonna be happening often (except for the fn pointer-like closures, for which there is the simpler solution of using the fn pointer directly). :x:
  • Box<dyn ...>, Rc<dyn ...>, Arc<dyn ...>
    Indirection by an owning pointer (Box for an exclusive such pointer, Rc for a shared thread-local one, and Arc otherwise).

    • In our case, all three pointers could be used; but given that we do not need the reference counting, we are gonna go and use a Box :white_check_mark:
thread_local! {
    static STRATEGY
        : Box<dyn 'static +
            Fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
        >
        = Box::new({
            use ::core::cell::Cell;
            let counter = Cell::new(0);
            move |objs| {
                let counter = counter.replace(counter.get() + 1);
                dbg!((counter, objs.len()));
                None
            }
        })
    ;
}

Inlineable closure type

As you may have noticed, the Box<dyn ...> requires an allocation, much like c++'s std::function (implicitly) does.

To avoid the heap-allocation, one needs a compile-time fixed type, to be able to inline the closure within the (global) memory of the program, whose layout must be known at compile-time (≠ heap memory).

The classic way to do this in Rust is to define your own struct, and ensure it meets the trait bounds by implementing the required traits.

The problem in this case, is that one cannot implement the Fn trait(s) in stable Rust!

To work around this, one must use a classic traits rather than the special Fn ones, thus having to drop the () call sugar when using an implementor of the trait:

trait MyFn : 'static {
    fn call (self: &'_ Self, objs: &'_ [&'_ OtherObject])
      -> Option<JoinHandle<()>>
    ;
}
/// Optional: Make closures implement `MyFn`
impl<F> MyFn for F
where
    F : 'static + Fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
{
    fn call (self: &'_ Self, objs: &'_ [&'_ OtherObject])
      -> Option<JoinHandle<()>>
    {
        self(objs)
    }
}

use ::core::cell::Cell;

struct MyImplFnState {
    counter: Cell<usize>,
}

thread_local! {
    static STRATEGY: MyImplFnState = {
        let counter = Cell::new(0);
        return MyImplFnState {
            counter,
        };
        // where
        impl MyFn for MyImplFnState {
            fn call (self: &'_ MyImplFnState, objs: &'_ [&'_ OtherObject])
              -> Option<JoinHandle<()>>
            {
                let Self { counter } = self;
        
                let counter = counter.replace(counter.get() + 1);
                dbg!((counter, objs.len()));
                None
            }
        }
    };
}

fn main ()
{
    STRATEGY.with(|strategy| {
        strategy.call(&[]);
        strategy.call(&[]);
        strategy.call(&[]);
        strategy.call(&[&OtherObject,]);
    });
}

The classic way to do this in Rust is to define your own struct, and ensure it meets the trait bounds by implementing the required traits.

But, in nightly, there is another way: that of using impl ... type-erasure anywhere. Which you seem to have attempted to use. To be able to use impl ... anywhere, one must use:

  • A type SomeHiddenImplementor = impl ... type alias;

  • Have a function definition that returns SomeHiddenImplementor, to tie the hidden type to the actual type used in the implementation of that function.

#![feature(type_alias_impl_trait)]

type MyImplFn =
    impl 'static + Fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
;

thread_local! {
    static STRATEGY
        : MyImplFn
        = {
            return new_MyImplFn();
            // where
            #[allow(nonstandard_style)]
            fn new_MyImplFn () -> MyImplFn
            {
                use ::core::cell::Cell;
                let counter = Cell::new(0);
                move |objs| {
                    let counter = counter.replace(counter.get() + 1);
                    dbg!((counter, objs.len()));
                    None
                }
            }
        }
    ;
}
12 Likes

And similarly, Box<dyn Fn()> in Rust does not allocate when it is constructed from a zero-sized type, such as a fn item or a closure with zero captures. So if some or all of your closures are zero-sized, you won't pay any allocation cost for them.

3 Likes

@Yandros
Thank you for such precice replay ...

Do you know how to call the following code properly ?:

        LOCAL_STRATEGY.with(|strategy| unsafe {
            if strategy.borrow().is_active() {
                // Could not get how to call this method
                // (&*strategy.borrow()).start();
            }
        });

where start is:

fn start(&'static self) {
    // ...
}

I got the following error:

first, the lifetime cannot outlive the anonymous lifetime #2 defined on the body

I have figured out by myself how to do it:

        LOCAL_STRATEGY.with(|strategy| unsafe {
            if strategy.borrow().is_active() {
                let strategy = unsafe { &mut *strategy.as_ptr() };
                strategy.start();
            }
        });

Ouch, please don't do that. unsafe should not be used for hacking around lifetime errors. Lifetime errors are emitted for a very good reason, and it's overwhelmingly likely that if you do this, you will end up with unsound, buggy code that contains undefined behavior.

In this case, your local strategy parameter and the Ref you borrowed from it isn't 'static, so it's invalid to call a method on it that requires that self: &'static. What are you actually trying to do?

4 Likes

The LLVM backend of the rustc compiler places significant semantic constraints on the intermediate-representation (LLIR) code that the rustc compiler front-end produces. unsafe does not remove LLVM's input constraints; it merely instructs the rustc frontend that YOU are assuming some of the responsibility of meeting those constraints.

The term "Undefined Behavior" (UB) refers to the situation where you have violated LLVM's input constraints, giving it permission to do whatever it chooses with your code in its quest for hyper-optimization. If your use of unsafe leads to UB, you have given LLVM permission to miscompile your program. There's an old phrase for this: "Garbage in, garbage out".

Edit: Corrected "Unspecified" to "Undefined". Thanks, @lxrec

3 Likes

I'm guessing you meant Undefined Behavior?

(unfortunately both terms are in use and have extremely different meanings so we are required to nitpick this)

(Sorry for my late response)

It seems we have an XY problem here: as @H2CO3 said, you shouldn't be having a self: &'static Self method to begin with (the only reason I think you might have ended up with it is because of some other lifetime constraint in your code). Try to modify start to take self: Self instead (with a potential where Self : 'static bound if you need it), so as to return ownership out of the ownership you take in input. If, for some reason, you need to both be : 'static (i.e., not infected by some lifetime parameter) and have pointer indirection (e.g., you are dealing with a trait object (dyn)), then Boxing the self value is the way to go:

type MyDynFn =
    dyn Fn(&'_ [&'_ OtherObject]) -> Option<JoinHandle<()>>
        + 'static
;

impl Strategy {
-   fn start (self: &'static Self) -> &'static MyDynFn
+   fn start (self: Self) -> Box<MyDynFn>
    {
        // extract the `Fn...` closure out of the struct
-       &self.some_field_name
+       Box::new(self.some_field_name)
    }
}
  • (I can only speculate about the contents of your code: if, on the contrary, your start method does not return anything nor write to any global, then the solution is to simply replace the 'static lifetime with a (free) lifetime parameter, such as '_: fn start (self: &'_ Self))
1 Like

Thanks all for support !!

If you interested what is it for ...

I've been just rewriting from scratch the rust-gc library, just to try strength... )

Here is my implementation:

And also I have the similar project written for C++ with Python :wink:

But writing it in Rust is much more fun :wink:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.