What doing is taking a whole file, splitting it into parts, then writing those individual parts to disk. I eventually want to then take those smaller parts and split them further into even smaller parts. I'm running into an issue when I get to the second loop. So at this point, the whole file has been split in half and I'm trying to take one of the halves and split that in half so it's 1/4.
I'm running into an error in line 21 of the pastebin link where it states I can't convert a usize to u64. The usize value is on line 20 with the new_end_byte variable.
If I try to convert the current u64 variables to usize, then I get other errors. Namely, when I then try to write the file slice I get an error on line 31 in the pastebin because the SeekFrom::Start(start) requires a usize.
The compiler says that I can try to convert the usize into u64 with a try_into().unwrap() and that does work but I was hoping there was a more elegant way to do it that would allow for the conversion without relying on an unwrap, expect, etc.
Ultimately I'm just trying to split a file into smaller parts, loop, and split again. If anyone has a different idea of how to do that, I'd be interested to hear it.
Converting between numeric types can fail. The "elegant" way to do it is to handle it considering the error case, which is what the TryInto trait is for.
In a nutshell, you have to decide whether you handle the error or not. There's no way around it.
So that's what I was talking about trying. I changed all of the variables to usize and when I tried what you were talking about, it created a new error on the first line of that function:
match f.seek(SeekFrom::Start(start)) {
[...]
errors with the SeekFrom::Start requiring a u64, not a usize. So at some point I'd need to convert to u64 anyways since ultimately I'm seeking through the file.
There aren't any supported user platforms on which usize is bigger than u64, so you can simply do start as u64. Keep in mind that going the other way around some_u64 as usize will silently truncate and cause bad results on 32-bit hardware!
Wait...i think that means I misunderstood the two...I thought usize could be bigger than u64 which is why it can cause problems trying to go usize into u64. But you're saying it's the other way around? If I cast a u64 as usize then that can cause problems?
I'd go a little further and say there's little chance of 128+ bit consumer hardware ever happening. A 64-bit address space is already far bigger than could possibly be used by one computer (64-bit is already over 18 exabytes, an exabyte being 1,000,000 terabytes). Even when using large amounts of that for process isolation and whatnot, you still have far, far more virtual memory than could ever be feasibly utilized. I believe some companies are looking into 128-bit distributed computing, but that's not going to affect the average user (or the average programmer).[1]
It’s both ways. Here is an overview of all the cases:
usize is allowed to be 16 bits, so there is no impl From<u32> for usize, and truly portable Rust code must be able to handle that size. However, many Rust libraries cannot practically be used on platforms with such very small amounts of addressable memory, so it is reasonable to reject this case. (Consider adding a compile_error!() to note this, rather than allowing truncation to happen.)
usize is allowed to be 32 bits.
Many processors for “embedded” applications use 32-bit addresses. (However, these platforms usually will not have std.)
WebAssembly is usually 32-bit, though 64-bit is an option. So, if you are writing a library that someone might want to use in WebAssembly (including with filesystem access via WASI), then you should be prepared for this case.
Today most desktop, mobile, and server CPUs are 64-bit, but there is still older 32-bit hardware out there, and some people may be interested in programming for it.
usize is very often 64 bits.
usize is not specifically prohibited from being 128 bits or larger. No current platform has this, but it has not been ruled out yet.
Therefore, my opinion is that if you are writing a Rust library to publish to crates.io, then:
it should be thoroughly prepared for usize being either 32 or 64 bits;
if no_std, it should handle 16 bits if feasible, and
supporting > 64 bits is speculative for a future that may not ever happen, so do it only if it’s straightforward.
If you are not publishing your code for others to use, then of course you only need to worry about the platforms your application will be built for.
That's true for physical memory, but it can be useful to have virtual memory much larger than physical memory, and computers could easily handle more virtual memory.
For instance imagine 10 billion Vecs such that you know each of them may hold up to 10 GB, and you also know the total won't be larger than 1 TB. You can fit in 1 TB physical memory, but it could be useful to just allocate 10 GB of capacity in virtual memory to each Vec so that they never have to reallocate. This wouldn't fit in 64 bit address space.
Similarly virtual memory could be useful to allocate giant stacks in virtual memory to a lot of threads, even though most will never use up those stacks.
That's true, and granted I don't handle commercial systems... but 10 billion independent allocations, each 10GB in size, sounds a little absurd. Even 1TB of physical memory is futuristic for consumer hardware.
I think that if there's ever a platform with 128-bit addresses, Rust will end up splitting usize in two: the type the size of a pointer, which would still be 128-bit, and the type for how big an individual allocation can be, which would probably still be only 64-bit -- or maybe even just 63-bit.
More layers of address translation are useful, yes. But in an individual allocation I don't think anyone needs an exabyte.
I don't think anyone would split usize on 128 bit into 64 bit, because usize is not for numbers, it is for arrays. If you just need numbers you use u32, u64 or u128.
usize is system depended because it will have different size on different bit sistems. Because on 128 bit system array can have 128 bit size index, array needs to return 128 bit size usize.
That's not technically true; in fact, the largest allocation the Rust abstract machine allows on any system is isize::MAX bytes. (I can't remember if that limitation applies to arrays of ZSTs, though...)
.