Hi there, Im trying to write scraping async library for rust, learning on the go and just now got blocked by an error that I don't understand. Here is my public api:
let mut crabweb = CrabWeb::new();
crabweb.on_html("td.title > a.storylink", |request: &mut Request, a: &Element| async move {
println!("found a link");
if let Some(href) = a.attr("href") {
println!("Found link {} on {}", href, request.url);
request.navigate(href).await?;
}
Ok(())
}).await;
crabweb.navigate("https://news.ycombinator.com/").await?;
let worker = crabweb.new_worker();
let handle = tokio::spawn(async move { worker.start().await; });
crabweb.run().await?;
handle.await?;
Here is my crawler struct definition:
pub struct CrabWeb<Fut, F>
where Fut: Future<Output=Result<()>>,
F: Fn(&mut Request, &Element) -> Fut {
on_html_callbacks: HashMap<&'static str, F>,
navigate_ch: Channels<String>, // async_std channels
// ... some other crap
}
I get following error that makes almost no sense to me:
error: borrowed data cannot be stored outside of its closure
--> src/main.rs:40:95
|
38 | let mut crabweb = CrabWeb::new();
| ----------- borrowed data cannot be stored into here...
39 |
40 | crabweb.on_html("td.title > a.storylink", |request: &mut Request, a: &Element| async move {
| _______________________________________________------------------------------------____________^
| | |
| | ...because it cannot outlive this closure
41 | | println!("found a link");
42 | | if let Some(href) = a.attr("href") {
43 | | let href = href.clone();
... |
48 | | Ok(())
49 | | }).await;
| |_____^ cannot be stored outside of its closure
error: aborting due to previous error
I do not capture anything from outside of this closure, request does share channels with CrabWeb struct, I'm a bit puzzled on how to solve this issue.
Thanks in advance!