What is MAX array size?

I need to create as large arrays as possible in a numerical application.

What is the largest array|vec index size in Rust?

1 Like

The maximum length is isize::MAX / size_of::<T>() where T is the element type of your array.

However, for large arrays, it is generally recommended to use a Vec.

8 Likes

What would that be for an array of unsigned 64-bit elements and unsigned 8-bit elements?

I really want to create an array of size 2**64 - 1 elements of any type.
Is that possible?

That would take 18 million terabytes of RAM. You don't have that much memory.

27 Likes

I'm not currently memory limited, I'm array|vec index limited.

I want eventually the capacity to do UInt64::MAX (for a distributed system)
I need now to be able to go over Int32::MAX elements on my laptop.
I just need to know the MAX limit so I know how I have to architect my system resources.

That doesn't sound like an array or vector. Other kinds of collections can have their own restrictions. E.g. there's nothing stopping you from using a HashMap<u64, YourType>.

If you are on a 64-bit system, then there shouldn't be anything stopping you from allocating 2^31 bytes of memory.

On a 64-bit system, isize::MAX is 2^63-1.

However, realistically the real limit is the amount of memory you have, which will be much smaller than the limit I mentioned previously.

3 Likes

On x86-64 even though pointers are 64 bits you can't address that much memory. See the osdev wiki.

2 Likes

I have a 64-bit cpu|system, so I can address and array|vec up to (2**63 - 1) elements.
Correct?

If the element size is zero, then you can have a vector of length 2^63-1, yes.

fn main() {
    let vec = vec![(); 9223372036854775807];
    println!("{}", vec.len());
}

But if you want actual data in the vector, then no. Then creating the vector will fail because it is too large.

4 Likes

Thank you for confirming the answer.
I know what I'm doing, and why.

Yeah, but it would only cost 83.5 Billion dollars :slight_smile:
Amazon

11 Likes

ZSTs allow all the way up to usize::MAX = 2^64-1, for better or worse, since the total size is still zero. The capacity of Vec<ZST> is always usize::MAX as well, regardless of reserve/shrink/etc.

4 Likes

For a distributed system at risk of running out of 64-bit indexes, e.g. disk management handling over 18 exabytes, you can either use u128 or a custom index, but either way you're never materializing the entire thing into an array.

4 Likes

Well that's for sure interesting problem to have. I wonder what the use case is

2 Likes

From the docs for Vec<T>:

Vectors ensure they never allocate more than isize::MAX bytes.

So if the elements take up any space at all (i.e. if they're not ZSTs), the upper limit on the number of elements is essentially half as high as might naïvely be expected.

1 Like

You cannot, in practice, know the maximum possible size in advance. You can know an upper bound (as a few people have pointed out), but the limit will also be determined by the environment in which your program executes - including both the configuration of the hardware, and the configuration of any operating system that may sit between your program and said hardware.

You may find that the practical limit is much, much lower than the bounds given in this thread, even on systems with wide addressing and tons of RAM. The only way to find it, with any real finality, is to allocate an array (or a Vec) and try writing to every element of it; either you will succeed, or the program will be terminated by a fault handler or a panic. And even then, having succeeded once does not guarantee that you will be able to drop and re-allocate another array of the same size, or that you will not be able to allocate a larger array later on.

I would suggest revisiting whatever design criteria you have picked that lead you to trying to find a largest-possible array size. Large arrays are certainly possible, but the largest is probably not as necessary as it seems.

6 Likes

Seems to me that usize/isize are all about the maximum size of the addressable memory the machine one is running ones code on has.

But out OP is wanting make an array-like thing that is not a language array or library Vec, rather it is something that can be indexed like an array but is actually a complicated thing storing it's data over multiple machines. A database. So actual limits of Vec on the machine are besides the point.

I guess if this array-like thing wants to expose an API that allows for indexing with [] it will be limited to usize/isize. Or would it be possible override the indexing operator to use u128?

3 Likes

x[i] is syntactic sugar for std::ops::Index[Mut]::index[_mut](x, i), and it can be implemented using custom index types.

6 Likes

Great. Sky's the limit then.

1 Like