Failing to read a file due to it not being valid UTF-8

help
1

This is some code I wrote (src/main.rs):

let contents = fs::read_to_string("test.bf").unwrap();

however, when it runs, it panics due to the file not being valid UTF-8 despite the file being UTF-8 encoded. The character it claims is not valid is '.'. I printed out the bytes it read, one thing was odd: the file is only around 50 characters (all ASCII) long but the byte buffer was so big my terminal ended erasing some of the output due to it being so big (well over 10'000 bytes long). I am very confused and I am new to programming (only around 4 months experience) I may be missing something.

2

Maybe there are a ton of invisible unicode characters in your file? Have you tried cat -v test.bf? It should show invisible characters in your console.

3

Yes. The output is:

>++++++++[<+++++++++>-]<.>++++[<+++++++>-]<+.+++++++..+++.>>++++++[<+++++++>-]<++.------------.>++++++[<+++++++++>-]<+.<.+++.------.--------.>>>++++[<++++++++>-]<+.
4

I would suggest doing

dbg!(fs::read("test.bf"));

to look at the individual bytes.

What do you see from that?

5

Show us a hexdump of the file.

6

I got:

[
    62,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    91,
    60,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    62,
    45,
    93,
    60,
    46,
    62,
    43,
    43,
    43,
    43,
    91,
    60,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    62,
    45,
    93,
    60,
    43,
    46,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    46,
    46,
    43,
    43,
    43,
    46,
    62,
    62,
    43,
    43,
    43,
    43,
    43,
    43,
    91,
    60,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    62,
    45,
    93,
    60,
    43,
    43,
    46,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    46,
    62,
    43,
    43,
    43,
    43,
    43,
    43,
    91,
    60,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    62,
    45,
    93,
    60,
    43,
    46,
    60,
    46,
    43,
    43,
    43,
    46,
    45,
    45,
    45,
    45,
    45,
    45,
    46,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    45,
    46,
    62,
    62,
    62,
    43,
    43,
    43,
    43,
    91,
    60,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    43,
    62,
    45,
    93,
    60,
    43,
    46,
    10,
]

EDIT: Just to add to anyone reading, they are all valid ASCII characters. This is why I am confused.

7

Can you upload your complete project, including the relevant data files, to github/gitlab or your favorite service?

8

is there exactly one test.bf file on your file system? or are there multiple directories with different test.bf files (relative paths will point to different files if your program is started in different working directories)?

is the file written by you or is another program generating it?

are you on windows or linux?

9

There is only 1 test.bf file on my entire computer. I use Linux.

10

I read this message after I refactored my code, so, I am unable to give the exact code when I ran the program (and now it works which is not helping my confusion), sorry.

11

I have decided to shut this down as I am unable to help anymore and the problem is not reproducible. If you wish to research the error and why it appeared, the entire code was (the project is bigger than main.rs but I was doing testing on reading files and this was the only code that ran):

use std::fs;

fn main() {
   let contents = fs::read_to_string("test.bf");
   println!("{}", contents);
}

and test.bf was:


>++++++++[<+++++++++>-]<.>++++[<+++++++>-]<+.+++++++..+++.>>++++++[<+++++++>-]<++.------------.>++++++[<+++++++++>-]<+.<.+++.------.--------.>>>++++[<++++++++>-]<+.
12

ok, the issue is outside of rust i think, is your hard drive healthy, what file system are you using?

13

You likely had a different test.bf with invalid characters when it did not work, that's all there is to it.

14

Maybe a previous larger file was overwritten without truncation enabled.

15

I have this problem with one of my projects. I am reading a file supplied by a third party open source project and it has some encoding errors.

I needed to read the file as bytes and convert it myself to utf8

My read loop looks like

loop {
            byte_buf.clear();
            // rather than read a line we need to read the non UTF-8 lines and decode ourselves
            match reader.read_until(b'\n', &mut byte_buf) {
                Ok(0) => break, // EOF
                Ok(_bytes) => {
                    offset += 1;
                }
                Err(msg) => {
                    error!("{}", msg.to_string());
                }
            }
            let buf = Self::bytes_to_utf8(&byte_buf);

and the conversion function

    fn bytes_to_utf8(byte_buf: &[u8]) -> Cow<'_, str> {
        match std::str::from_utf8(byte_buf) {
            Ok(ccc) => {
                Cow::Borrowed(ccc)
            }
            Err(e) => {
                let s = String::from_utf8_lossy(byte_buf);
                warn!("{} - {}", e, s);
                s
            }
        }
    }

Hopefully this can help others who may trip over similar problems.

p.s. I have reported the errors upstream, but it's complicated

16

you might be looking for utf8_chunks, depending on how much of the line with non-utf output you want to keep, or attempt recovery on.

17

if you're working with partially-utf8 strings bstr could also be useful.