Beginner - Help With Reqwest Crate Console Printing

[greenie]

this is quite literally my first day learning Rust, and I am trying to get the hang of requests using the Reqwest crate.

In this quick program, I am just trying to scrape the view count on a Youtube video. The program runs with no issues, but it is not actually printing the view count onto the console.

I am using Reqwest and Scraper to make this happen - I am trying to use Selector to parse the page and I just inspect element'd to find the class name for where the view count is stored.

All help is appreciated, and constructive criticism is welcomed as I continue to learn this. thanks!

extern crate reqwest;
extern crate scraper;

use scraper::{Html, Selector};


fn main() {
    println!("Getting view count...");
    scrape_view_data("https://www.youtube.com/watch?v=clcmJMfOj3c");
}

async fn scrape_view_data(url:&str) {
    let req = reqwest::get(url).await.unwrap();
    assert!(req.status().is_success());
    let doc_body = Html::parse_document(&req.text().await.unwrap());

    let view = Selector::parse(".view-count style-scope yt-view-count-renderer").unwrap();

    for view in doc_body.select(&view) {
        let views = view.text().collect::<Vec<_>>();
        println!("{}", views[0]);
    }
}

The result:

warning: crate `Tests` should have a snake case name
  |
  = note: `#[warn(non_snake_case)]` on by default
  = help: convert the identifier to snake case: `tests`

warning: unused implementer of `Future` that must be used
 --> src\main.rs:9:5
  |
9 |     scrape_view_data("https://www.youtube.com/watch?v=clcmJMfOj3c");
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_must_use)]` on by default
  = note: futures do nothing unless you `.await` or poll them

warning: 2 warnings emitted

    Finished dev [unoptimized + debuginfo] target(s) in 1.07s
     Running `target\debug\Tests.exe`
Getting view count...

Thanks!

[cole-miller]

Welcome! The second warning from the compiler tells you what's going on:

  = note: futures do nothing unless you `.await` or poll them 

To elaborate: your scrape_view_data is an async fn, which means that when you call it the runtime will not actually do anything until it reaches a .await point. Try changing the last line in main to scrape_view_data(/* ... */).await;

The "async book", Asynchronous Programming in Rust, has more information (and there are many people on this forum who know more about this topic than I do).

[greenie]

Thank you for replying so fast -

When I try this, I get this error:

only allowed inside `async` functions and blocks

And I know Async functions are not allowed as the main function.

Any other ideas or tweaks?

[cole-miller]

Ah, my mistake (I told you I wasn't an expert at this async stuff!). Looks like you want block_on from the futures crate:

https://rust-lang.github.io/async-book/01_getting_started/04_async_await_primer.html

[greenie]

Do you know where I would incorporate this into my code?

[mbrubeck]

If you don't need any concurrency, you might find the non-async reqwest::blocking API a lot simpler to use.

If you do want to try out async code, you can get started by adding a dependency on the futures crate and then do this:

use futures::executor::block_on;

fn main() {
    println!("Getting view count...");
    block_on(scrape_view_data("https://www.youtube.com/watch?v=clcmJMfOj3c"));
}

[greenie]

Thank you - when I try this out, I get a new error -

thread 'main' panicked at 'not currently running on the Tokio runtime.'

Any ideas?

[mbrubeck]

Ah, sorry, I forgot that async reqwest depends on the Tokio runtime. You can use the tokio::main macro to write an async main function.

See the Tokio docs for basic instructions, and the reqwest examples for some simple programs that use it with reqwest.

Or, if you just want to write a simple program without concurrency, try request::blocking.

[alice]

With async code using Tokio:

use scraper::{Html, Selector};

#[tokio::main]
async fn main() {
    println!("Getting view count...");
    scrape_view_data("https://www.youtube.com/watch?v=clcmJMfOj3c").await;
}

async fn scrape_view_data(url:&str) {
    let req = reqwest::get(url).await.unwrap();
    assert!(req.status().is_success());
    let doc_body = Html::parse_document(&req.text().await.unwrap());

    let view = Selector::parse(".view-count style-scope yt-view-count-renderer").unwrap();

    for view in doc_body.select(&view) {
        let views = view.text().collect::<Vec<_>>();
        println!("{}", views[0]);
    }
}

With reqwest::blocking.

use scraper::{Html, Selector};

fn main() {
    println!("Getting view count...");
    scrape_view_data("https://www.youtube.com/watch?v=clcmJMfOj3c");
}

fn scrape_view_data(url:&str) {
    let req = reqwest::blocking::get(url).unwrap();
    assert!(req.status().is_success());
    let doc_body = Html::parse_document(&req.text().unwrap());

    let view = Selector::parse(".view-count style-scope yt-view-count-renderer").unwrap();

    for view in doc_body.select(&view) {
        let views = view.text().collect::<Vec<_>>();
        println!("{}", views[0]);
    }
}

It seems to me that you should be using the second method in this case.