Defining a storage type and using it at runtime

Hello, I’m trying to find a way to define custom homogeneous containers. i.e. I’d like to have an array of ImageComponent so that the definition would be Vec<ImageComponent> but I want that to be part of the ImageComponent itself.

Something along those lines:

trait Component {
}

pub struct ImageComponent {
    x: f32,
    y: f32,
}

impl Component for ImageComponent {
    type Storage = Vec<Self>;
}

and then I’d like to have an HashMap that I can use to add new named containers to, like:

struct World {
    component_types: HashMap<String, Storage>,
}

impl World {
    pub fn add_storage_to_container<C: Component>(&mut self, name: String, component: C) {
        self.component_types.insert(name, component::Storage::new());
    }
}

And furthermore do something like:

fn main() {
    let world = World {
        component_types: HashMap::new(), 
    }

   world.add_storage_to_container(ImageComponent::Storage);
}

The code is just a mockup, I know it’s not working and I’m not even sure you can pass around types instead of variables. I’ve been trying to set this up on the Playground, but I’m really not grasping how to do it.

I could really use some help to point me in the right direction.

Thanks in advance!

You can specify generic type parameters via the turbofish:

world.add_storage_to_container::<ImageComponent::Storage>()

But these only exist at compile time, not at runtime. For what you’re trying to do, it’s not so straightforward, but I might try starting with

struct World {
    component_types: HashMap<String, Box<dyn Any>>,
}

This isn’t quite the same thing as “passing around a type” at runtime (that’s impossible!)… but if you look in the map with a specific string, and you know what type the storage must be at that string, you can downcast the Any into that type.


Am I correct in guessing that you’re trying to write an Entity Component System? There are many, many crates on crates.io that at least claim to implement an ECS (some probably better than others). Maybe you might want to browse around for inspiration?

I didn’t know about the turbofish, awesome name btw :wink:

Compile time could be a problem, though. I would like to register and create the storages before initializing the system… I thought generics might serve that purpose.
I don’t know what a string actually maps to. I’m using a HashMap just to make things simpler here, but in a real case scenario I will use a Vec and each component time will be represented by the index in that Vec.
Anyway I guess I can live with having to remind that a certain id maps to a certain type, that’s something I can go with.

Yes, you are correct and I think that specs does something similar to what I’m trying to do but it’s a lot more complicated compared to what I need.

I thought I could get away with something like this, but I’m still missing something:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=6d9b0392fbd0c1d1e03306353bda32d2

Hi, I’ve got a “working” playground, it uses very simplified parts of specs.
Storage is like specs’s UnprotectedStorage and the world is using a HashMap<TypeId, Box<dyn Any>> like you could find here.
But as soon as you add an entity without all the components present in the world it will break.

Do you mean as a user? Or someone trying to make an ECS?

Ooh, if it’s untyped data storage we’re talking about, I just finished a crate (restor) that does just that! I presume you could say that it is similar to an ECS, but I didn’t intend for it to be a replacement.

1 Like

Thanks! That’s definitely helpful!

That’s because you’re storing in World.components the actual component I guess.
What I’ve been thinking about is to store in World.components the actual Storage and then in the Storage I actually store the single instances of the struct implementing Component.

I mean as a user, specs has a lot of features that I do not need and that are actually making code more complex, like all the async processing of systems. My requirements are much simpler, I just want some plain arrays of each component type that I can walk over, and then systems have to subscribe to the component types they’re interested in and receive them as argument and that’s pretty much all I really need. I don’t need parallelism or anything fancier :slight_smile:

I’ll have a look at it, I guess I could either use it or steal some ideas from your crate :wink:

@leudz I’ve elaborated your Playground and came up with this: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1e65c58a29b99f53bffa7916504fc02f

It looks like it’s working even if I’m not using all the components for every entity. I do not like that I have to go through the whole array at line 50. I guess this could be solved with a bitset or something like that but I don’t even know if there’s something like that out of the box in Rust.

If you just need Vecs you can get rid of the Component and GenericStorage traits, playground.

You can keep it ordered, but a bitset is definitely better.

There are a few crates ready to be used like bit-set.

But I would still recommend using an existing ECS, with specs you can run your systems one by one if you don’t want to use the scheduler, you can remove the parallel feature and won’t have any parallelism. Pyro is really simple.

@leudz I agree that using an existing ECS might bring more good than bad, but I’m on a learning path here and I just want to get the bare minimum working as it will be enough for the time being :slight_smile:

I completely forgot about destroying entities… I tried to put something together, but I don’t know how to convert the TypeId to the actual struct so that I could remove it from the components storage.

I tried to put something together at this playground: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=fb760591e63f4f407b2e3798a1500d79

I don’t know any easy fix, what you want is reflection but it’s really hard to get right so currently you can’t go from TypeId to T.
You can use unsafe and hack something that might be sound, you could leave the components inside their storage or do something like specs. If you follow the trail you want delete_entity from there you get to any_storages, that uses MetaTable from shred and that’s where it gets tricky, from what I understand it’s making Frankenstein monsters out of trait objects. It takes vtable apart, store them and reassemble the trait object to be able to call drop.
It’s a really good solution to a complicated problem but if you want a safe and simple fix I don’t know any, maybe someone will step in with something (I’m interested too =).

You also have to do something to prevent two entities from having the same id or at least make it very difficult. A pretty easy fix is to use a generational index, basically a usize or (u32, u32) with an index and a generation, the generation is incremented each time you delete the entity.
There is a big conversation talking about a lot of things including what an entity will be in specs at this issue, it can be an interesting read.

I guessed that would be pretty much impossible or very hard to do in a straightforward way.

What if I store a component_type_id as usize instead of the TypeID? That way I might be able to work my way back to the component through indexing eventually.

That’s the unsafe hacky solution I mentioned, you store the TypeId of all component (you already do), their size (you can put it with the container to just have it once) and you also need to store the entity id for each component (your memory won’t like it) or you can add a level of indirection.
From here since you don’t have the real type you just cast containers to Vec<u8> and you can swap remove bytes based on the size. The last component’s entity has to be updated too (that’s why you need the reverse lookup).
If the component implements Drop it won’t be called but

Rust’s safety guarantees do not include a guarantee that destructors will always run

so it’s technically within safe Rust’s rules but not pretty. If for example you store Vecs it will just remove the “stack part” and the heap allocated memory will only be reclaimed at the end of the program.

I just noticed your component remove doesn’t work, since you use Vec's remove all indices after the one you remove will be shifted. But the entities still have the old index, so they’ll steal components from others like in this playground.

I haven’t yet solved removal but I wanted to take a stab at implementing systems, so I went forward and did something that is mostly working but I’ve got a big issue.

The problem is that I’d like to pass to process a user-defined data type. In C I’d usually just pass a void * and then cast it to whatever I want to, but in Rust how can I achieve something like that?

There are a few ways to do it, it depends on how you made process.

This is pretty much what I had in mind, but since SystemData is a trait, I cannot use it as a custom type to pass around: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0bbab872d6b40385d2e20057580c1011

If you simply get SystemData by reference, it compiles. playground
You shouldn’t have too many lifetime issue as long as it’s not multi threaded.

True, but what if I want to pass some custom data in SystemData? What I’ll be passing isn’t known beforehand, it’s something that depends on the implementation of the System.

i.e. I might have something like:

pub struct Context {
    something: bool
}

impl SystemData for Context {
}

pub struct SpriteSystem {
    base: BaseSystem
}

impl System for SpriteSystem {
    fn process(&self, entity: Entity, dt: f32, user_data: &SystemData) {
        // How do I pass my Context in SystemData and read it back here?
    }
}

You can’t get the concrete type back this way, you’d have to add methods on the trait and do something with them.

I just thought of it, why not make it a &mut Any and cast it inside process, as long as your type is 'static this is exactly like your void * solution.