What is the maximum length of a vector?

dvshapkin · May 23, 2020, 6:10am

Hi,
let v = vec![0_i32; 500_000_000]; // it's Ok
let v = vec![0_i32; 600_000_000]; // error: panicked at 'capacity overflow', src\liballoc\raw_vec.rs:777:5

600_000_000 < u32::MAX < u64::MAX
What type has capacity?

Cocalus · May 23, 2020, 6:20am

I assume you're on a 32bit platform
size_of::<u32>() * 600_000_000 > isize::max

Most major OSes can only allow half the address space for users processes, the kernel is mapped into the other half, so more than that probably wouldn't work.

Hyeonu · May 23, 2020, 6:19am

In Rust no allocations can have size larger than isize::MAX. It's mostly because the LLVM's getelementptr instruction, which is used for safe ptr offset, takes isize.

dvshapkin · May 23, 2020, 7:15am

isize::MAX >= i32::MAX = 2_147_483_647 > 600_000_000

I also have a 64 bit OS

jhpratt · May 23, 2020, 7:35am

How much RAM do you have, and how much is otherwise used? You are trying to allocate 2.4 GB, after all.

RustyYato · May 23, 2020, 10:11am

It's isize::MAX < 600_000_000 * size_of<i32>()

dvshapkin · May 24, 2020, 2:13am

I have 32Gb RAM on 64-bit OS

dvshapkin · May 24, 2020, 2:19am

it's true,
but panicked at 'capacity overflow', and capacity stores the number of elements, not the size of the memory they occupy

RustyYato · May 24, 2020, 4:10am

True, but the error message is generated here

github.com

rust-lang/rust/blob/4774f9b523c942cb5c0236542b5bcac76f6b6b9a/src/liballoc/raw_vec.rs#L149


/// of allocator for the returned `RawVec`.
#[inline]
pub fn with_capacity_zeroed_in(capacity: usize, alloc: A) -> Self {
    Self::allocate_in(capacity, Zeroed, alloc)
}

fn allocate_in(capacity: usize, init: AllocInit, mut alloc: A) -> Self {
    if mem::size_of::<T>() == 0 {
        Self::new_in(alloc)
    } else {
        let layout = Layout::array::<T>(capacity).unwrap_or_else(|_| capacity_overflow());
        alloc_guard(layout.size()).unwrap_or_else(|_| capacity_overflow());

        let memory = alloc.alloc(layout, init).unwrap_or_else(|_| handle_alloc_error(layout));
        Self {
            ptr: unsafe { Unique::new_unchecked(memory.ptr.cast().as_ptr()) },
            cap: Self::capacity_from_bytes(memory.size),
            alloc,
        }
    }
}

When you try to allocate memory, which only cares about the number of bytes to allocate, not how many elements.

dvshapkin · May 24, 2020, 3:10am

you practically convinced me)
but I still don’t understand what exactly is being overflowed in this case:
the capacity of the vector is not, since this is still the number of elements (and not the number of bytes),
RAM - I have 32 GB of it, and it requires only 2.4 GB
don't be mad at my stubbornness, I just want to understand.

P.S.although you’re probably right, because:
let v = vec![0_u16; 600_000_000]; - Ok

thanks, the question is cleared

mbrubeck · May 24, 2020, 4:53am

If you build your Rust code for a 64-bit target, you’ll be able to use all your RAM. 32-bit programs can only use 2GB.