Future cannot be sent between threads safely (axum + scraper)

help
1

Hi everyone,

I recently began migrating a backend I'm working on from Actix to Axum. So far I have quite liked it. Unfortunately, I've come across a rather mindboggling problem. I have the following handler function:

use axum::{http::status::StatusCode, extract::State, debug_handler};
use base64::Engine;
use reqwest::Client;
use scraper::{Selector, Html};
use serde_json::{Value, json};
use crate::utils::{XibendHandlerResult, cache::{XibendCache, CommonDataCache}, response::XibendResponse, BASE64_JPEG_PREFIX, BASE64_ENGINE};

// The constant CSS selector string
const BANNER_SELECTOR: &str = "div.large-auto";

pub async fn banner(State(client): State<&'static Client>) -> XibendHandlerResult<Value> {
    let raw_data = String::from_utf8(XibendCache::get_common_data(CommonDataCache::OYKBanner).await?)?;

    let html = Html::parse_document(&raw_data);
    let banner_selector = Selector::parse(BANNER_SELECTOR)?;

    let image_url = html.select(&banner_selector).next().map(|url| {
        url.value().attr("style").map(|style| {
            style.replace("background-image: url(", "")
                .replace(");", "")
        })
    }).flatten();

    let image_encoded = match image_url {
        Some(url) => {
            let raw = client.get(&url).send().await?.bytes().await?;
            Some(String::from(BASE64_JPEG_PREFIX) + &BASE64_ENGINE.encode(&raw))
        },
        None => None,
    };

    let image = json!({"image": image_encoded});
    Ok(XibendResponse(StatusCode::OK, image))
}

This function does not throw any errors on its own, but when calling it from an Axum .route() I get this almost meaningless error message:

the trait bound `fn(axum::extract::State<&'static Client>) -> impl Future<Output = Result<XibendResponse<serde_json::Value>, XibendError>> {banner::banner}: Handler<_, _, _>` is not satisfied
the following other types implement trait `Handler<T, S, B>`:
  <MethodRouter<S, B> as Handler<(), S, B>>
  <axum::handler::Layered<L, H, T, S, B, B2> as Handler<T, S, B2>>

Luckily, I was instructed to use the debug_handler macro that gives a more clear error message. Using it I get a few different errors, all being similar to this:

future cannot be sent between threads safely
within `tendril::tendril::NonAtomic`, the trait `Sync` is not implemented for `Cell<usize>`

This error also points to the client.get(&url).send().await?.bytes().await? call. As far as I understand it, using scraper and async together is the problem, since scraper is not Send + Sync. That is why I want to ask: how could I fix this function? All kinds of solutions are welcome.

2

You can do something like

let image_url = tokio::task::spawn_blocking(move || {
    let html = Html::parse_document(&raw_data);
    let banner_selector = Selector::parse(BANNER_SELECTOR).unwrap();
    html.select(&banner_selector)
        .next()
        .map(|url| {
            url.value().attr("style").map(|style| {
                style
                    .replace("background-image: url(", "")
                    .replace(");", "")
            })
        })
        .flatten();
})
.await?;

The types that aren't threadsafe get created and used on a single background thread

1 Like