tokio::JoinSet and tokio::spawn vs hickory_resolver's futures that aren't `'static`

help
1

The code at the end of this post is cut down from a program that's supposed to use hickory_resolver to do a bunch of DNS lookups concurrently. It won't compile, with the error

error[E0597]: `resolver` does not live long enough
  --> src/bin/async-resolve-multi.rs:17:24
   |
10 |         let resolver = Resolver::builder_with_config(
   |             -------- binding `resolver` declared here
...
17 |             let task = resolver.lookup_ip(host);
   |                        ^^^^^^^^ borrowed value does not live long enough
18 |             futures.spawn(task);
   |             ------------------- argument requires that `resolver` is borrowed for `'static`
...
21 |     });
   |     - `resolver` dropped here while still borrowed

The issue appears to be that the future returned by resolver.lookup_ip contains an internal reference to the Resolver object, which makes it not 'static (since the Resolver object obviously does not have static lifetime).

I am at a loss for how to fix this. Advice for related problems seems to be to convert the reference that isn't 'static to an Arc<Box<>> or something along those lines, but that's not an option here because I don't control the code that's producing the troublesome reference. Does anyone have a suggestion?

use hickory_resolver::Resolver;
use hickory_resolver::config::ResolverConfig;
use hickory_resolver::name_server::TokioConnectionProvider;
use tokio::runtime::Runtime;
use tokio::task::JoinSet;

fn main() {
    let rt = Runtime::new().unwrap();
    let results = rt.block_on(async {
        let resolver = Resolver::builder_with_config(
            ResolverConfig::default(),
            TokioConnectionProvider::default()
        ).build();

        let mut futures = JoinSet::new();
        for host in ["www.example.com.", "www.google.com."] {
            let task = resolver.lookup_ip(host);
            futures.spawn(task);
        }
        futures.join_all().await
    });
    for res in results {
        match res {
            Ok(r) => {
                println!("{}:", r.query().name.to_utf8());
                for addr in r.iter() {
                    println!("\t{}", addr);
                }
            }
            Err(e) => {
                println!("error: {e}");
            }
        }
    }
}

Note: This question is closely related to Newb: Tokio::JoinSet and Hickory-Resolver::Resolver Tokio runtimes interfering. However, since that question was posted, the AsyncResolver interface has been removed from hickory_resolver. Also, I'm trying to use one Resolver object to make many DNS queries, in order to get the benefit of Resolver's internal caching.

2

A solution with Arc would look like:

use hickory_resolver::config::ResolverConfig;
use hickory_resolver::name_server::TokioConnectionProvider;
use hickory_resolver::Resolver;
use std::sync::Arc;
use tokio::runtime::Runtime;
use tokio::task::JoinSet;

fn main() {
    let rt = Runtime::new().unwrap();
    let results = rt.block_on(async {
        let resolver = Arc::new(
            Resolver::builder_with_config(
                ResolverConfig::default(),
                TokioConnectionProvider::default(),
            )
            .build(),
        );

        let mut futures = JoinSet::new();
        for host in ["www.example.com.", "www.google.com."] {
            let resolver = resolver.clone();

            let task = async move { resolver.lookup_ip(host).await };

            futures.spawn(task);
        }
        futures.join_all().await
    });
    for res in results {
        match res {
            Ok(r) => {
                println!("{}:", r.query().name.to_utf8());
                for addr in r.iter() {
                    println!("\t{}", addr);
                }
            }
            Err(e) => {
                println!("error: {e}");
            }
        }
    }
}
3

Huh. I had convinced myself that that wouldn't work, because no matter what I did with the Arc, the future created by calling lookup_ip would necessarily contain a bare reference to the resolver. I guess maybe that is the point of wrapping the call in async move { ... }? To encapsulate the bare reference, so that the compiler can see it doesn't survive the Arc handle that the closure owns?

4

Yes, we move the owned (i.e. 'static) Arc to the task, the async move { ... } future. Thus we can guarantee that the Resolver won't get dropped while the task is running or waiting to be scheduled.

1 Like
5

Another solution: The need for 'static comes in because spawning a task with JoinSet (which uses tokio::spawn() under the hood) requires the tasks to be sendable between threads, but there are other async combinators that can run multiple futures concurrently without a multithreading dependency that you can use instead. Some options, all from the futures crate, include:

  • Push the resolver.lookup_ip(host) futures into a FuturesUnordered instance and then iterate over the results as they're produced with while let Some(r) = futures.next().await { ... } (next() being provided by StreamExt).

    • Note that this comes with a potential footgun: While inside the body of the while loop, the lookup_ip() futures are not being polled, so (depending on how exactly the futures work) spending too much time inside the body may result in timeouts at the next futures.next().await call.

  • Alternatively, if you strictly need all futures to complete before iterating over any of the results, you can use join_all().

1 Like