The additional pointer is to create a trait object, which provides the necessary type erasure to store heterogeneous types in an otherwise homogeneous container.
Your case sounds a bit like this recent thread, and you may be able to use the same approach of storing an entire Vec behind a fat ptr rather than each element in the vec.