A generic reader over multiple types of files: snarled in type-checking

[chetmurthy]

I'm trying to get the following concept implemented:

1. suppose we have a function that takes an io::BufRead and reads it, producing some result type T
   an example would be here: sprs/io.rs at a3982f2fcf6eecf2eacf0239fbb476dcd32ad067 · sparsemat/sprs · GitHub

pub fn read_matrix_market_from_bufread<N, I, R>(
    reader: &mut R,
) -> Result<TriMatI<N, I>, IoError>

2. suppose I want to allow it to work on both gzip and plaintext files. Then I might want to write a combinator like

    pub fn with_file<R, E>(p : &Path,
                           f : &dyn Fn(&mut dyn io::BufRead) -> Result<R, E>,
    ) -> Result<R, E>
    where E : From<io::Error>
    {
        if p.ends_with(".gz") {
            let fp = File::open(p)?;
            let mut buf = io::BufReader::new(fp);
            let mut reader = GzDecoder::new(buf);
            let mut buf = io::BufReader::new(reader);
            let rv = f(&mut buf) ;
            return rv ;
        }
        else {
            let fp = File::open(p)?;
            let mut reader = io::BufReader::new(fp);
            let rv = f(&mut reader) ;
            return rv ;
        }
    }

This would check the file, and based on the extension, either open a BufReader or a GzDecoder (which itself must be wrapped in a BufReader. BTW, I put in those dyn keywords b/c the compiler was complaining. But I don't (yet) understand completely what's going on, sigh.

3. And then I might want to actually use this combinator

        let trimat : TriMatI<Complex64, u64> =
            qrusty::util::fileio::with_file(path,
                                            &|reader : &mut dyn io::BufRead|{
                                                sprs::io::read_matrix_market_from_bufread(reader)
                                                    .map_err(|e| PyException::new_err(format!("matrixmarket_read: {}", e)))
                                            })?;

but this code fails to compile:

error[E0277]: the size for values of type `dyn BufRead` cannot be known at compilation time
   --> pyqrusty/src/lib.rs:204:91
    |
204 | ...                   sprs::io::read_matrix_market_from_bufread(reader)
    |                       ----------------------------------------- ^^^^^^ doesn't have a size known at compile-time
    |                       |
    |                       required by a bound introduced by this call
    |
    = help: the trait `Sized` is not implemented for `dyn BufRead`
note: required by a bound in `read_matrix_market_from_bufread`
   --> /home/chet/Hack/IBMQ/src/sprs/sprs/src/io.rs:139:46
    |
139 | pub fn read_matrix_market_from_bufread<N, I, R>(
    |                                              ^ required by this bound in `read_matrix_market_from_bufread`

Again, I added some borrows and dyns in the hopes of getting the compiler to accept it. But I don't really understand how to make this compile.

It could be that the right answer is just "go read about the typechecking algorithm". OTOH, if anybody has any suggestions for how to get this working, I'd appreciate them.

[quinedot]

You're close. Most places you have type parameters, there's an implicit Sized bound, because when it comes to objects you hold or arguments that functions take by value or return, that's usually what you want. But sometimes you don't want that bound, and this is one of those situations.

(What are non-Sized types?)

They are types which are dynamically sized. For example, Path and str and [u8] are dynamically sized arrays basically; when you have a &str, it's two usize long -- one holds a pointer to the data and the other holds the length of the data.

Type erased types -- dyn Trait types -- are also dynamically sized, because you can turn both a String and a u8 into a dyn Display, for example, but they are different sizes. So they also have to be behind (wide) pointers; in this case, the second usize is a pointer to a vtable (which in turn contains the size and other information).

---

You can turn off the implicit bound by saying R: ?Sized:

 pub fn read_matrix_market_from_bufread<N, I, R>(
     reader: &mut R,
 ) -> Result<TriMatI<N, I>, IoError> 
 where
-    R: io::BufRead,
+    R: io::BufRead + ?Sized,
     I: SpIndex,
     N: NumCast

Playground.

[chetmurthy]

Lovely! Thank you so much! And I learned a little more about the type system to boot! With that extra teaching, I figured out how to do the same thing to the write side. Sweet!

[Cocalus]

If you control the API, you should replace &mut R with R. Since &mut BufRead implements BufRead, you can still pass a &mut into it (among other types). This can help avoid adding ?Sized to a lot of higher level generic functions and keep things simpler.

[chetmurthy]

That yielded another error. The body of read_matrix_market_from_bufread uses the reader in a mutable way.

error[E0596]: cannot borrow `reader` as mutable, as it is not declared as mutable
   --> sprs/src/io.rs:156:5
    |
140 |     reader: R,
    |     ------ help: consider changing this to be mutable: `mut reader`
...
156 |     reader.read_line(&mut line)?;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

[Cocalus]

Do what the compiler suggests and put mut before reader in the parameter list.