Reqwest read large files as byte stream

[markchaitin]

Hello! I have a basic implementation of reqwest using the 'blocking' feature as my needs are small and I already have a multi-threaded approach to the application. A problem arises when a larger file is being requested from the remote server (which does not provide any data chunking). Using the various methods on the response object seem to wait until the full payload has been received before transferring things into bytes. Ideally I'd like to throw bytes in a buffer as they come in so that I can provide some notification that work is being done (or some similar solution).

What I'm seeing is the reader is initialized and loops, but no bytes are ever read from the response. What am I missing? Does my approach make sense?

  fn get_remote_file<P: AsRef<Path> + Debug>(    
      url: String,    
      path: P,    
  ) -> Result<(), reqwest::Error> {    
      let mut response = get(&url)?;                                                                                                   
      let content_length = response.content_length().unwrap();                                                                         
                                                                                                                                   
      let mut bytes: Vec<u8> = Vec::with_capacity(content_length as usize);                                                            
      let mut total = 0;                                                                                                                                                                                                                            
      'byte_reader: loop {                                                                                                             
          match response.read(&mut bytes) {                                                                                                                                                                      
              Ok(b) => {                                                                                                                                                                                         
                  debug!("bytes read: {}", b);                                                                                                                                                                   
                  total += b;                                                                                                                                                                                                                                                                                                                                                                      
                  if total > 0 && b == 0 {                                                                                                                                                                       
                      break 'byte_reader;                                                                                                                                                                        
                  }                                                                                                                                                                                              
              }                                                                                                                                                        
              Err(e) => {                                                                                                                                              
                  error!("Failed to stream data: {}", e);                                                                                                              
              }                                                                                                                                                        
          }                                                                                                                                                            
      }                                                                                                                                                                
      Ok(())                                                                                                                                                           
  }

[drewkett]

You need to initialize bytes with zeroes not just give it a capacity. read takes a slice so passing it a zero length Vec passes read a zero length buffer and no data is read

[alice]

I would avoid the total > 0 condition on your break.

[markchaitin]

After playing with it I realized I was conflating a couple things. Final solution was to open a writer, pass in an initialized byte array to the read call, and write all bytes out to the writer after every read instance (in range 0..bytes_read). Terminate loop when total_read == content_length.

[Hyeonu]

Or you can use io::copy() function which does it in more optimized way.