[Solved] Problem with reqwest 0.10 async client

help
1

Hello i'm using rust to do a scraper of images i have seen that the crate reqwest now can be done in async way but i have a problem with the client

i'm using a client because i have to download various images

Before i was using reqwest in the sync way and with a get to every html and img.

Now is 2020 and i want it to do it async and reusing the client.

The problem perhaps is simple for you but nor for me

when i have to download the image and put it in a file i'm doing:

async fn bajar_imagen(url_imagen: &str, directorio_capitulo: &Path, cliente: &reqwest::Client) {
   // dbg!(&url_imagen);
    let mut url = String::from("https:");
    let url2 = url_imagen.to_string();
    url.push_str(&url2);
    let url = Url::parse(&url).expect("Fallo en la tranformación url de la imagen");

    match cliente.get(url.as_ref()).send().await{
        Ok(res) => {
        match res.text().await {
            Ok( doc1) => {
            let vec: Vec<&str> = url_imagen.split('/').collect();
            if vec.len() >= 9 {
                let vec2: Vec<&str> = vec[8].split('?').collect();
                // dbg!(vec2[0]);
                if !vec2.is_empty() {
                    let new_file = directorio_capitulo.join(vec2[0]);
                    let mut file = File::create(new_file).expect("Fallo al crear fichero vacio");
                    io::copy(&mut doc1.as_bytes(), &mut file).expect("Fallo al copiar al fichero");
                } else {
                    println!("{}", Red.bold().paint("Han cambiado urls de imagenes"));
                }
            } else {
                println!("{}", Red.bold().paint("Han cambiado urls de imagenes"));
            }
        }
        Err(e) => {
            println!(
                "{}, {}",
                Yellow
                    .bold()
                    .paint("Error al bajar imagen mirar status, borrar directorio"),
                e
            );
            borrar_directorio_y_panic(&directorio_capitulo);
        }
        }
        }
        Err(e) => {
            println!(
                "{}, {}",
                Yellow
                    .bold()
                    .paint("Error al bajar imagen mirar status, borrar directorio"),
                e
            );
            borrar_directorio_y_panic(&directorio_capitulo);
        }
    }
}

i'm not sure why the file is not a good jpg. i know that the url is correct, but i don't know how to transform
this in to a file.

io::copy(&mut doc1.as_bytes(), &mut file).expect("Fallo al copiar al fichero");

Any help?

2

You need to surround your code with three backtics to mark them as code like this:

```
// your code here
```

Anyway, you should not use std::io::copy or the File from the standard library in an async context. You need to use tokio::fs::File instead, and one way to use it is as below:

use tokio::prelude::*;

let mut response = reqwest::get("https://hyper.rs").await?;
let mut file = tokio::fs::File::open("filename").await?;
while let Some(chunk) = response.chunk().await? {
    file.write_all(&chunk).await?;
}

Note that this code avoids reading the full file into memory by writing chunks to the file as they arrive.

3

Thanks this will improve my code, (now i will edit the main topic to better codification)

Doing this the file copy in async way, but my problem is the same:

with the url:
//l.mangatown.com/store/manga/30589/020.0/compressed/b019.jpg?token=963ddeefd4c38b5b4a64ed9ca7aed6e1fe7106a9&ttl=1577966400

i put an https: before and then in obtain the url:

when i create a file b019.jpg

but when i pass what i'm obtaining with the client (i think that this is the problem)
i obtain a file of 395Kb but not a valid .jpg
how i can use the client in the reqwest to obtain a file . perhaps i'm doing something wrong with the .send .as_bytes or with the .text when i'm trying to obtain the image.

Thanks in advance

4

Well when you use text, you get, well, a string. And strings must be valid utf-8, which images typically are not.

5

Ok, but when i change the code to

match cliente.get(url.as_ref()).send().await {
        Ok(res) => {
            match res.as_bytes().await {
                Ok(doc1) => {
                    let vec: Vec<&str> = url_imagen.split('/').collect();
                    if vec.len() >= 9 {
                        let vec2: Vec<&str> = vec[8].split('?').collect();
                        // dbg!(vec2[0]);
                        if !vec2.is_empty() {
                            let new_file = directorio_capitulo.join(vec2[0]);
                            let mut file =
                                File::create(new_file).expect("Fallo al crear fichero vacio");
                            io::copy(&mut doc1.as_bytes(), &mut file)
                                .expect("Fallo al copiar al fichero");

the compiler says :slight_smile:

error[E0599]: no method named as_bytes found for type reqwest::async_impl::response::Response in the current scope
--> src/main.rs:297:23
|
297 | match res.as_bytes().await {
| ^^^^^^^^ method not found in reqwest::async_impl::response::Response

error: aborting due to previous error

I think that i don't know how to reuse the client to get the image, perhaps it's more simple to use your example and use and reqwest::get to get the image (but i think this is suboptimal)

6

Use Response::bytes instead of a as_bytes

Or if you want chunks at a time, use chunk or bytes_stream

7

I'm seeing that i'm more near of the good solution

but i need and example of using the client and not (reqwest::get) to get a file

because now the compiler don't like to write in a file syncronism, if the file it's not complete

and i'm understanding better the problem with async and programing.

Thanks to all in advance

8

Hello everybody. now i'm having a little problem i have changed my code to download in async way the image and trying to put it, in an asyncfile

 match cliente.get(url.as_ref()).send().await {
       // Ok(res) => {
       //     match res.bytes().await {
                Ok(mut doc1) => {
                    let vec: Vec<&str> = url_imagen.split('/').collect();
                    if vec.len() >= 9 {
                        let vec2: Vec<&str> = vec[8].split('?').collect();
                        // dbg!(vec2[0]);
                        if !vec2.is_empty() {
                            let new_file = directorio_capitulo.join(vec2[0]);
                            dbg!(&new_file);
                            let mut file = tokio::fs::File::open(new_file).await.expect("Fallo al crear el fichero async");
                            while let Some(chunk) = doc1.chunk().await.expect("Fallo al obtener datos de la imagen") {
                                file.write_all(&chunk).await.expect("Fallo al escribir en el fichero");
                        }

I have changed the await? for await.expect because my fucntion don't return an error

but now when i try to execute the program i obtain this:

thread 'main' panicked at 'Fallo al crear el fichero async: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5

I think that rust is trying to write in the file before it was created....

but i need thah the file was from tokio::fs::file to be an async file and write in async ways....

Any good solution and simple solution?

9

Ah sorry, you need File::create instead of File::open.

10

Now i have it working but now i'm using this:

                    match tokio::fs::File::create(new_file).await {
                        Ok(mut file) => {
                            while let Some(chunk) = doc1
                                .chunk()
                                .await
                                .expect("Fallo al obtener datos de la imagen")
                            {
                                match file.write_all(&chunk).await {
                                Ok (_) => {}
                                Err(e) => {
                                println!(
                                "{}, {}",
                                Yellow
                                    .bold()
                                    .paint("Fallo al escribir en el fichero, borrar directorio"),
                                e
                            );
                            borrar_directorio_y_panic(&directorio_capitulo);
                                }
                                }
                    
                            }
                        }
                        Err(e) => {
                            println!(
                                "{}, {}",
                                Yellow
                                    .bold()
                                    .paint("Error al crear fichero, borrar directorio"),
                                e
                            );
                            borrar_directorio_y_panic(&directorio_capitulo);
                        }
                    }

I have changed all the await? for await and match to if an error happens delete the directory, and show the error in the terminal

but i don't know hot to change the last .expect because i'm not very sure about the more rustecian way of doing it

while let Some(chunk) = doc1.chunk().await?

Any ideas how i can get data in async way and if an error happens delete the directory?

11

OK, after getting my sleeping hours i have done it

But another question about the compiler brings to life.

This is my code now:

async fn bajar_imagen(url_imagen: &str, directorio_capitulo: &Path, cliente: &reqwest::Client) {
    // dbg!(&url_imagen);
    let mut url = String::from("https:");
    let url2 = url_imagen.to_string();
    url.push_str(&url2);
    let url = Url::parse(&url).expect("Fallo en la tranformación url de la imagen");

    match cliente.get(url.as_ref()).send().await {
        Ok(mut doc1) => {
            let vec: Vec<&str> = url_imagen.split('/').collect();
            if vec.len() >= 9 {
                let vec2: Vec<&str> = vec[8].split('?').collect();
                // dbg!(vec2[0]);
                if !vec2.is_empty() {
                    let new_file = directorio_capitulo.join(vec2[0]);
                    //  dbg!(&new_file);
                    match tokio::fs::File::create(new_file).await {
                        Ok(mut file) => {
                            while let Some(chunk) = match doc1.chunk().await {
                                Ok(x) => x,
                                Err(e) => {
                                    println!(
                                "{}, {}",
                                Yellow
                                    .bold()
                                    .paint("Fallo al obtener datos de la imagen, borrar directorio"),
                                e
                            );
                                    borrar_directorio_y_panic(&directorio_capitulo);
                                    panic!();
                                }
                            } {
                                match file.write_all(&chunk).await {
                                    Ok(_) => (),
                                    Err(e) => {
                                        println!(
                                "{}, {}",
                                Yellow
                                    .bold()
                                    .paint("Fallo al escribir en el fichero, borrar directorio"),
                                e
                            );
                                        borrar_directorio_y_panic(&directorio_capitulo);
                                    }
                                }
                            }
                        }
                        Err(e) => {
                            println!(
                                "{}, {}",
                                Yellow
                                    .bold()
                                    .paint("Error al crear fichero, borrar directorio"),
                                e
                            );
                            borrar_directorio_y_panic(&directorio_capitulo);
                        }
                    }
                } else {
                    println!("{}", Red.bold().paint("Han cambiado urls de imagenes"));
                    borrar_directorio_y_panic(&directorio_capitulo);
                }
            } else {
                println!("{}", Red.bold().paint("Han cambiado urls de imagenes"));
                borrar_directorio_y_panic(&directorio_capitulo);
            }
        }
        Err(e) => {
            println!(
                "{}, {}",
                Yellow
                    .bold()
                    .paint("Error al bajar imagen mirar status, borrar directorio"),
                e
            );
            borrar_directorio_y_panic(&directorio_capitulo);
        }
    }
}

fn borrar_directorio_y_panic(directorio_capitulo: &Path) {
    match fs::remove_dir_all(&directorio_capitulo) {
        Ok(_) => {
            println!("{}", Green.bold().paint("Se ha borrado directorio"));
        }
        Err(e) => {
            println!(
                "{}, {}",
                Red.bold().paint("No he podido borrar directorio"),
                e
            );
        }
    };
    panic!();
}

I'm not very sure because i have yo put a panic!() in a match

yes i know that is for transform a result in to a option to get the other match with let Some()
works

But i have a function that in the finish it will panic always , the theory is that is going to remove the directory and stops all the work

i don't know if there is a better solution, thanks in advance

12

You can mark the function with a return type of ! if it always diverges.

fn borrar_directorio_y_panic(directorio_capitulo: &Path) -> !
13

Many Thanks!

Now i will change the topic to solved

14

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.