Is it possible to convert a vec<u8> to a vec<&T> (ECS implementation)?

help
1

Hello, I'm working on a little ECS for learning purpose.
I'm actually stuck on the system part :

This example work :


    fn move_system((p, p1, p2, p3, p4) : (Position, Position1, Position2, Position3, Position4))
    {
        //Do some stuff
    }
    
    fn main()
    {
        let entity_capacity = 1383333;
        let mut world = World::new(entity_capacity);
        for n in 0..entity_capacity {
            let f = n as f32;
            //Spawn and register component if they are new
            world.spawn((
                Position { x: f, y: f, z: f }, Position1 { x: f, y: f, z: f }, Position2 { x: f, y: f, z: f },
                Position3 { x: f, y: f, z: f }, Position4 { x: f, y: f, z: f },
            ));
        }
      world.add_system(move_system); //Register a function or a closure as a new system
      world.run_once(); //Run all the registered system once, Working well
    }

This one doesn't

    
    fn move_system((p, p1, p2, p3, p4) : (&Position, Position1, Position2, Position3, Position4))
    {
        //Do some stuff
    }
    
    fn main()
    {
        let entity_capacity = 1383333;
        let mut world = World::new(entity_capacity);
        for n in 0..entity_capacity {
            let f = n as f32;
            //Spawn and register component if they are new
            world.spawn((
                Position { x: f, y: f, z: f }, Position1 { x: f, y: f, z: f }, Position2 { x: f, y: f, z: f },
                Position3 { x: f, y: f, z: f }, Position4 { x: f, y: f, z: f },
            ));
        }
      world.add_system(move_system); //Register a function or a closure as a new system
      world.run_once(); //Runtime error => STATUS_ACCESS_VIOLATION
   }

After digging into my implementation, I understand why that STATUS_ACCESS_VIOLATION happend,
This is because my underlying function to get the data of each components look like this :


    fn get_components_internal<T: Any + Component>(&self) -> Result<ManuallyDrop<Vec<T>>, String> {
        let component_type = TypeId::of::<T>();
        match self.component_types.get(&component_type) {
            Some(t) => {
                if let Some(data) = self.component_data.get(t) { // data is a Vec<u8> that contain all the data of this component type
                    let (ptr, len, cap) = (data.as_ptr(), data.len(), data.capacity());
                    let component_size = self.component_sizes.get(t).unwrap();
                    let rebuilt = unsafe {
                        Vec::from_raw_parts(ptr as *mut T, len / component_size, cap / component_size) //This line seems to be the root of the issue
                    };
                    return Ok(ManuallyDrop::new(rebuilt));
                }
            },
            None => {}
        }
        return Err("This component type isn't register in this world".to_string());
    }

So, this function work perfectly fine for value types :


    if T == Position
       Vec::from_raw_parts(ptr as *mut Position, len / component_size, cap / component_size)
    //This is valid (vec<u8> to vec<T>)
    but
    if T == &Position
       Vec::from_raw_parts(ptr as *mut &Position, len / component_size, cap / component_size)
   //This is not valid, because I'm creating a vector of ref,
   //onto a vector of u8 values which of course doesn't make sens (vec<u8> to vec<&T>).
   //The runtime error occur later: vec[index] => STATUS_ACCESS_VIOLATION

So, I guess my question is:
Is it possible to create a vec<&T> from a vec<u8> ?
Maybe I should create first a vec<&T> of size len / component_size, but then how I'm suppose to assign these ref to the actual vec<u8> data?

I also wrote another implementation of get_components_internal that is able to know at runtime if T, is a Value, a Ref or a MutRef, but since I don't know how to convert a vec<u8> to a vec<&T>... This didn't solve my issue (but I could write a custom code for Ref and MutRef, and keep the actual code for values).

Hope my question make sens :sweat_smile:, this is my first try to implement an ECS, so I maybe not approching the subject correctly.

2

Please format your code correctly, it's not readable like this. See the pinned post for more.

1 Like
3

I know, sorry about that, that is my first post.

4

That should be more readable, I didn't saw the preview at first glance because of the forum tips hiding it :upside_down_face:.

5

No, it's not possible to transmute Vec<u8> to a Vec with a type that has a different alignment requirements or capacity that isn't even multiple of element's size. The memory allocator used by Vec can expect that alignment won't change, and that freed size will be even multiple of element size. Transmuting Vec is a huge minefield full of gotchas.

It's slightly less terribly crashy-dangerous to transmute slices. It is possible to transmute between &[u8] to other types using slice::align_to, but only if the data is actually valid for that type and properly aligned (i.e. it had to actually be that data originally). However, transmuting &[u8] to a slice of temporary borrows &[&T] is very unlikely to work, and is incredibly dangerous, because it's most likely losing original lifetime of the temporary borrows, and you end up with a slice of dangling pointers.

You might be able to store Vec<T> and deref it to &[T] which you can cast to &[u8] and back to &[T], but only if T is #[repr(C)] and doesn't contain any padding or atomic types or mutexes (UnsafeCell). I think you could use &[MaybeUninit<u8>] for types with padding.

You could look into moving instead of casting. ptr::write_unaligned/ptr::read_unaligned lets you put type's data in an arbitrary location. When it's moved to location you control, then you don't get all of the headaches about alignment, allocators or UB from sharing/exposing data you shouldn't have.

3 Likes
6

Hum well I see, so it seems that having a HashMap<TypeId, vec<u8>> to store all my components may not be the right way to store my components in a continious memory location.
This way I was sure that all the Position where stored in HashMap<TypeId::of::<Position>, vec<u8>>, since I was casting Position to u8 to store them.
So my whole implementation, is based on a wrong assumption.
That was working pretty well with a non reference type though :slightly_frowning_face:.

So, I need to find another way to store all my components :sweat_smile:

7

It sounds like you're actually looking for some sort of HashMap<TypeId, Box<dyn Any>>. Check out the anymap crate for ideas on how it can be implemented.

Edit: Here's a quick'n'dirty implementation to help you get started: playground.

3 Likes
8

It's more something like this HashMap<TypeId, Vec<Box<dyn Any>>>, but as far as I understand the Box's description : "A pointer type for heap allocation.".

That mean that every element inside the vector, will be allocated somewhere in the heap (and not in a continious space), and that is not really what I'm looking for.
Since all my components of the same type, should be contained inside the same memory location.

That's why I was using a vec<u8> to store all the data of the same type, in the same location. I was able to reinterpret them safefy after because I was also storing the size of each elements.

The issue occur when I'm trying to acces them by reference, because I cannot cast a vec<u8> into a vec<&Position> which is normal, however vec<u8> into vec<Position> was working perfectly fine (since I stored the bytes of every positions inside the vec<u8> and I know the size of the struct Position).

9

Forget this point.

Actually you migth be right, it could work like this I think, thank you @Michael-F-Bryan
playground

10

You don't need to store a Vec<Box<dyn Any>> and have two levels of heap allocation. Just tweak the lookup and insert code so instead of storing a T, you store a Vec<T>. You can add an item using something like self.items.entry(TypeId::of::<Vec<T>>()).or_default().push(item).

11

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.