Function which takes an Iterator<Item = Box<dyn Any>>

help
#3

I'm mapping each element too (wrapping it in this example). But in my later code, the wrap function will perform other tasks too. This is my real-life code, but I wanted to keep the example simple:

unsafe fn open_or_opt_create_dbs<'a, K, V, C>(
    self: &Arc<Self>,
    options: impl IntoIterator<
        IntoIter = impl Iterator<Item = &'a DbOptions<K, V, C, Option<CString>>> + ExactSizeIterator,
    >,
    create: bool,
) -> Result<Vec<Db<K, V, C>>, io::Error>
where
    K: ?Sized + Storable,
    V: ?Sized + Storable,
    C: Constraint,
{
    let options = options.into_iter();
    let mut dbs = Vec::with_capacity(options.len());
    let db_guard = self.db_mutex.lock().unwrap();
    let mut txn_inner = MaybeUninit::<*mut lmdb::MDB_txn>::uninit();
    check_err_code(lmdb::mdb_txn_begin(
        self.inner,
        null_mut(),
        if create {
            0 as LmdbFlags
        } else {
            lmdb::MDB_RDONLY as LmdbFlags
        },
        txn_inner.as_mut_ptr(),
    ))?;
    let txn_inner = txn_inner.assume_init();
    let txn = TxnBackend {
        env_backend: self,
        inner: txn_inner,
        cursors: Default::default(),
    };
    for option in options {
        let mut db_inner = MaybeUninit::<lmdb::MDB_dbi>::uninit();
        let mut flags = option.cleansed_lmdb_flags();
        if create {
            flags |= lmdb::MDB_CREATE as LmdbFlags;
        }
        check_err_code(lmdb::mdb_dbi_open(
            txn.inner,
            option.name.as_ref().map_or(null(), |x| x.as_ptr()),
            flags,
            db_inner.as_mut_ptr(),
        ))?;
        let db_inner = db_inner.assume_init();
        let db_backend = DbBackend {
            env_backend: Arc::downgrade(self),
            inner: db_inner,
        };
        if !K::TRIVIAL_CMP && !K::OPTIMIZE_INT {
            check_err_code(lmdb::mdb_set_compare(
                txn.inner,
                db_backend.inner,
                Some(compare_function::<K>),
            ))?;
        }
        if !V::TRIVIAL_CMP && !V::OPTIMIZE_INT && C::DUPLICATE_KEYS {
            check_err_code(lmdb::mdb_set_dupsort(
                txn.inner,
                db_backend.inner,
                Some(compare_function::<V>),
            ))?;
        }
        dbs.push(Db {
            key: PhantomData,
            value: PhantomData,
            constraint: PhantomData,
            backend: ArcByAddr::new(db_backend),
        });
    }
    txn.commit()?;
    drop(db_guard);
    Ok(dbs)
}

P.S.: I wanted to design the interface such that the caller can't hold the internal db_mutex lock for a long time.

#4

What is the signature of your actual dyn_wrap function?

#5

It would be something like:

unsafe fn open_or_opt_create_dbs_dyn<'a>(
    self: &Arc<Self>,
    options: impl IntoIterator<
        IntoIter = impl Iterator<Item = &'a Box<dyn Any>> + ExactSizeIterator,
    >,
    create: bool,
) -> Result<Vec<Box<dyn Any>>, io::Error>
where
    K: ?Sized + Storable,
    V: ?Sized + Storable,
    C: Constraint;
#6

I think you can do it with downcast_ref. The module docs for Any has some examples.

#7

But which type do I downcast to? I won't know (I think). Hence my question if anyone knows a way how to fix the todo!("How to implement this?") in my OP.

#8

If you don't know the type at all, you won't be able to call things on the iterator items (like you have called let mut flags = option.cleansed_lmdb_flags();). But, you can still iterate all right and simply accumulate.

#9

The caller of dyn_wrap would know the type, but the dyn_wrap function itself will only be able to use trait methods, of course.

#10

But you have "objects" of trait Any - which means you cannot access any methods without a downcast.

#11

Yes, I'll later have Any + SomeTrait.

I tried to modify my original example to expect an additional trait (which has an associated type that reflects the wrapped type), but couldn't get the original example working.

#12

You should probably use just dyn SomeTrait then, without Any?

#13

Maybe. Butā€¦ to even understand if it's possible what I try to achieve, I would like to get the above example working (or understand why it's not possible to get that example working).

I don't need particular methods in that example case. I need to be able to wrap the value in S. But not sure how to express that with a trait (whether Any or SomeOtherTrait).

#14

Maybe you can describe your situation agin, please?

Here is my understanding
1.You have a collection of trait objects
2.You have a transformation, which acts element-wise
3.You want to call: IntoIter,Map with transformer, collect

So, as far as I can tell, the only problem is step 2.
Is this close to your problem? (Maybe not)
If it is close, can you describe your transformation?

Regarding your wrap vs dyn_wrap: It is often much easier to write a transformer forba specific type thrn for a trait object, so I can "feel" your struggles.

For me, it is often helpful to start with an enum, with a handful of real-world variants. Then, once I manage write the transformer for this one, I tackle the trait object case (which is often much harder)

#15

Note also that it's impossible to consume dyn Trait via the method on Trait. You can either consume the Box<dyn Trait>, treating it as a "black box" (pun unintended), i.e. in your initial example S would be not generic, but just a wrapper around Box<dyn Trait>; or you can treat it as a reference and build the output from scratch (with possibly partially cloned data, if the trait provides such capabilities).

#16

What's missing here?

fn dyn_wrap(
    elements: impl IntoIterator<
        IntoIter = impl Iterator<Item = Box<dyn Any>>,
    >,
) -> Vec<Box<dyn Any>> {
    elements
        .into_iter()
        .map(|field| -> Box<dyn Any> { Box::new(S { field }) })
        .collect()
}
#17

The problem is that the contained values are of type S::<dyn Any> instead of S<i32> and S<&'static str>. Thus:

/* ā€¦ */
fn main() {
    /* ā€¦ */
    let c = dyn_wrap(vec![
        Box::new(7) as Box<dyn Any>,
        Box::new("seven") as Box<dyn Any>,
    ]);
    let c1: Option<&S<i32>> = c[0].downcast_ref();
    let c2: Option<&S<&'static str>> = c[1].downcast_ref();
    println!("{c1:?}");
    println!("{c2:?}");
}

(Playground)

Output:

ā€¦
None
None

Instead, I would like c to be something like this:

#![allow(dead_code, unused_variables)]

use std::any::Any;

#[derive(Debug)]
struct S<T> {
    field: T,
}

fn main() {
    let c: Vec<Box<dyn Any>> = vec![
        Box::new(S { field: 7 }),
        Box::new(S { field: "seven" }),
    ];
    let c1: Option<&S<i32>> = c[0].downcast_ref();
    let c2: Option<&S<&'static str>> = c[1].downcast_ref();
    println!("{c1:?}");
    println!("{c2:?}");
}

(Playground)

Output:

Some(S { field: 7 })
Some(S { field: "seven" })

I hope that clarifies my problem.

P.S.: In my real-life code, I want to be able to open or create several databases at once using open_or_opt_create_dbs even if those databases have different type arguments K, V, and C. That's why I guess I need some dyn type. But I also need the result to be downcast-able.

#18

I think my problem is (and I didn't clarify it yet, sorry about that):

  • I want to perform the transformation of a list of values of different type in a single method call.
  • I want to downcast the results such that I recover concrete types afterwards (because I know which types I gave as input into the function and which types are to be expected as output of the function).

The enum won't help me (I think), because I need to be generic over T (or K, V, C in my real-world example).

I hope to be able to publish my full source soon, so maybe I can also get back to this issue with the full real-life example.

#19

One more question:
Can you solve your Problem for a single item?

I expect something like

  • apply transformer to trait object
  • receiving a new trait object
  • downcast it to your given type
    This should work also for collections

I'm interested to learn more about your case

#20

Ah, I see. So you're going to need to move the boxed object, even though you don't know what it is [1]. To me that sounds like you need a Box<Self>-taking method on a trait; the trait implementation will know the underlying type and can move it into an S for you.

trait Burrito: Debug {
    fn wrap_in_s<'s>(self: Box<Self>) -> Box<dyn Any + 's> where Self: 's;
}

impl<T: Debug> Burrito for T {
    fn wrap_in_s<'s>(self: Box<Self>) -> Box<dyn Any + 's> where Self: 's {
        let field = *self;
        Box::new(S { field })
    }
}

fn dyn_wrap(
    elements: impl IntoIterator<
        IntoIter = impl Iterator<Item = Box<dyn Burrito>>,
    >,
) -> Vec<Box<dyn Any>> {
    elements
        .into_iter()
        .map(<dyn Burrito as Burrito>::wrap_in_s)
        .collect()
}

Playground.

  1. unless S is transparent and can be transmuted perhaps ā†©ļøŽ

2 Likes
#21

Thanks very much for this example. It's the first time I see / have to use Box<Self>.

It's exactly what I wanted to achieve in my original example.

But after trying some things based on your code, I'm not sure whether (or how) this can help me with my real-world problem.

After some struggles with crates.io and figuring out a publication process, I was finally able to get things published! Here's the full real-life example:

mmtkvdb/src/lib.rs

(Documentation doesn't build yet, I'll ask another question in the forum read into this thread on how to fix that.)

My goal is to make this function work with databases that use different types (K, V) and constraints (C):

    /// Open databases in environment
    ///
    /// SAFETY: If a database exists already, it must have been created with
    /// compatible options.
    unsafe fn open_dbs<'a, K, V, C>(
        &self,
        metas: impl IntoIterator<
            IntoIter = impl Iterator<Item = &'a DbOptions<K, V, C, Option<CString>>> + ExactSizeIterator,
        >,
    ) -> Result<Vec<Db<K, V, C>>, io::Error>
    where
        K: ?Sized + Storable,
        V: ?Sized + Storable,
        C: Constraint;

I tried to add a trait akin to @quinedot's proposal but ran into some problems when doing so (I think it had to do with having to specify associated types in the functions signature).

If I find time, I'll try to address the issue again and provide more information on where I stumbled.

#22

Finally the package built properly :hot_face:, so there is documentation on my project on docs.rs available :smiley:.

Consider the method mmtkvdb::Env::open_dbs (source code available behind the link) which opens several databases:

unsafe fn open_dbs<'a, K, V, C>(
    &self,
    metas: impl IntoIterator<
        IntoIter = impl Iterator<Item = &'a DbOptions<K, V, C, Option<CString>>> + ExactSizeIterator
    >
) -> Result<Vec<Db<K, V, C>>, Error> where
    K: ?Sized + Storable,
    V: ?Sized + Storable,
    C: Constraint;

Each database may use different types for their keys and values. Thus, the DbOptions builder uses the typestate pattern and has type arguments K and V for the key and value type.

The open_dbs method opens several databases at once (it will have to acquire a mutex for it and do a few other things, so when multiple databases are opened, it may make sense to only do that once, which is why I allow passing multiple DbOptions through an iterator). However, while the signature of open_dbs is generic over different K and V types, it does not support opening databases with different keys (or values) in one call, because K and V are particular types for each invocation of open_dbs.

To overcome the limitation of only being able to open multiple databases with the same key (and value) types, I wanted to use dyn objects. The caller of open_dbs could then downcast each element in the result according to the (known) types K and V.

In practice, the code quickly gets very complex though. I might just leave the interface as is and accept the overhead of opening differently-typed databases in multiple calls.

I wonder, however, if similar problems occurred in other contexts where there are lists of structs with type arguments passed to a function that returns a list of different structs with the same type arguments (in my example above: K, V, C appear both in the arguments as well as in the result of the function).

1 Like
Marker types: Unit-like structs or zero-variant enums? PhantomData or not?