Handling OutOfMemory in Write for Vec<u8>

1

The io::ErrorKind has a variant for OutOfMemory, which is used in some io methods like read_to_end. The impl Write for Vec<u8> does not handle it though, but easily could by replacing reserve with try_reserve. I'm wondering whether that was a conscious decision, or whether a simple PR would be welcome? Or does such change need a more complex process?

2

Good question! I don't know, but I have some thoughts about it.

The only scenario where it would makes sense is a big vector with a lot of capacity, in most scenarios I think they do not need to be that big, and if you are failing with that could mostly means you don't have enough ram, when linux gets out of RAM/Virtual memory it gets all the system stalled, is a lot safer there crash than try to do something.

Still, do the try_reserve could also be used to panic out of it, but any error could cause the system to collapse.

3

I don't know whether the PR would be accepted, but the OutOfMemory error is typically used when a syscall returns ENOMEM from a failure to allocate in the kernel. Rust rarely attempts to handle other allocation errors.

2 Likes
4

that reminds me other thing, maybe connected, a lot of operations works with the assumption they will work.

Like math operations, variable assignation and that things, I also have not seen error handling in that aspects.

I think is because most of error handling is related to algorithms it self, an error in a low level is not something that they usually take care.

The vector case I think is on the middle point, I would like it to use try_reserve, because the use of Heap is on the implementation of Vec, still a gray area.

If we want that level of details, maybe would be better a new set of instructions with that level of security, and probs most operations will be filled as "try_".

Some times is easier to have other app/layer that checks everything is going right.

1 Like
5

This is a tool to check “math operations, variable assignation and that things”. It is so cool and I cannot help to share with you when seeing

1 Like
6

I love it! I want to test it rn! I'll open some projects and play with it!

That covers a lot of things!

I think the ones also @jhorstmann means is for example, what happens if you make a sum and for a hardware/kernel/external app it fails the operation at a very low level!

7

To clarify my suggestion with pointers to concrete code, BufRead::read_to_end converts a TryReserveError to ErrorKind::OutOfMemory:

    fn read_to_end(&mut self, buf: &mut Vec<u8>) -> io::Result<usize> {
        let inner_buf = self.buffer();
        buf.try_reserve(inner_buf.len())?;
        ...
    }

Vec::write currently does not try, the default allocation error handler would therefore panic.

    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
        self.extend_from_slice(buf);
        Ok(buf.len())
    }

The latter case would actually be much easier to hit, since Write is implemented for Vec with an arbitrary custom allocator (nightly/unstable feature), which might limit the maximum allocation size. So there would be no operating system wide out of memory involved.

1 Like
8

read_to_end() was changed to report out of memory errors in this PR:

There was some discussion of whether it makes sense to do this, mostly around whether the behavior should be documented as guaranteed. (It was documented as not guaranteed.) Consulting the meeting notes, I find that there were two previous PRs for Write:

From the PR comments there, we find a concern for this that there is not for reading, which is that it would be possible to ignore the error (from a previously infallible function) and thus silently discard data. Thus, any new proposal to return OOM errors would need to address this concern in some way.

5 Likes
9

Thanks a lot for finding this previous discussions!

The current situation might be slightly changed, since try_reserve is stable and there is now even an impl From<TryReserveError> for io::Error, but the argument about users explicitly ignoring the result still stands.

Hmm, but I did not see such argument when &[u8]::read_to_end was changed, which was also infallible before.

But given this is easy to solve with a newtype wrapper, I probably won't try to reopen this discussion with another PR.

10

However, you might like to open a discussion on internals.rust-lang.org to gauge interest.

1 Like
11

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.