Note that grow may be able to extend the existing memory block more often than you'd expect. Once the vector gets large enough for the cost of copying to matter, the underlying realloc can use page manipulation rather than copying.
When you see how cheap realloc can be, the overhead of maintaining non-consecutive buffers seems wasteful. At the very least, it suggests the need to benchmark specific use cases.