Returning a future to the return of a function that takes a &self

hey there :3,

i am trying to send a few mails and would like to do so asynchronously using futures::future::join_all to await them all at once.

however i am running into the issue that lettres send takes in a &self returing a Pin<Box<dyn Future...> . I understood Futures similarily to closures, so from my understanding this means i am returning a closure that moves the &self into itself, and returning the entire closure means returning borrowed data that is owned by the function returning the future.
most of my attempts to solve this ran into this issue one way or another.

below is my minimum reproduction, you can also find it in this repo here where the later commits also show some solutions i tried without much success. the solutions i tried where:

  • currying as to return out an owned mailer
  • using lifetimes (but tbh i am too shaky with those and kinda failed i tried it out similar to this answer and this answer (not linked properly cuz limit on new users qq): users rust lang thread: /t/returning-futures-then-from-a-function/47431/2
  • futures::future::FutureExt sharable (doesn't work because the lettre::smtp::Error doesn't implement Clone but also doesn't really make sense as far as i can tell.)
  • futures::future::FutureExt map doesn't work either as that is just another closure and just moves the ownership/borrowing issue)

Minimum Reproduction:

Cargo.toml

[package]
name = "min_rep_returning_future_ref_self"
version = "0.1.0"
edition = "2021"

[dependencies]
futures = "0.3.30"
lettre = { version = "0.11.4", features = ["tokio1-rustls-tls", "dkim", "smtp-transport", "pool", "builder", "hostname"], default-features = false }
tokio = { version = "1.36.0", features = ["full"] }

src/main.rs

use std::future::Future;
use std::pin::Pin;
use futures::future::join_all;

use lettre::transport::smtp::response::Response;
use lettre::AsyncSmtpTransport;
use lettre::AsyncTransport;
use lettre::Tokio1Executor;
use lettre::{message::header::ContentType, transport::smtp::authentication::Credentials, Message};

#[tokio::main]
async fn main() {
    let mail_addr = std::env::var("MAIL_ADDR").unwrap();
    let mail_username = std::env::var("MAIL_USERNAME").unwrap();
    let mail_pw = std::env::var("MAIL_PW").unwrap();
    let mail_relay = std::env::var("MAIL_RELAY_ADDR").unwrap();

    let mail_contents = fetch_guys();
    let handles = mail_contents.into_iter().map(|(content, subject, to)| {
        send_mail(
            content,
            subject,
            to,
            &mail_addr,
            (&mail_username, &mail_pw),
            &mail_relay,
        )
    });

    let results = join_all(handles).await;
}

fn fetch_guys() -> Vec<(&'static str, &'static str, &'static str)> {
    vec![
        ("hey there just testing", "test mail", "mail@example.com"),
        ("yet another test", "test mail 2", "mail@example.com"),
    ]
}

fn send_mail(
    content: &str,
    subject: &str,
    to: &str,
    from: &str,
    credentials: (&str, &str),
    relay_addr: &str,
) -> Pin<Box<dyn Future<Output = Result<Response, lettre::transport::smtp::Error>> + Send>> {
    let credentials = Credentials::new(credentials.0.to_string(), credentials.1.to_string());

    let email = Message::builder()
        .from(from.parse().unwrap())
        .to(to.parse().unwrap())
        .subject(subject)
        .header(ContentType::TEXT_PLAIN)
        .body(content.to_string())
        .unwrap();

    let mailer: AsyncSmtpTransport<Tokio1Executor> =
        AsyncSmtpTransport::<Tokio1Executor>::relay(relay_addr)
            .unwrap()
            .credentials(credentials)
            .build();

    mailer.send(email)
}

Rather than return the Pinned Box Future, I would just call await at the end of mailer.send(email)

Then, to fix your lifetime issue, I have two thoughts:

  1. Change your method to take in Arc<String>, wrap all the env variables in an Arc, and clone() them inside the loop
  2. Add a .leak() at the end of the unwrap() for env variable, so that they change from a String to '&'static str, and change the send_mail method to take in &'static str for those parameters.

I wrote a solution that does both here using the code from your repo:

#[tokio::main]
async fn main() {
    let mail_addr: &'static str = std::env::var("MAIL_ADDR").unwrap().leak();
    let mail_username = Arc::new(std::env::var("MAIL_USERNAME").unwrap());
    let mail_pw = Arc::new(std::env::var("MAIL_PW").unwrap());
    let mail_relay = Arc::new(std::env::var("MAIL_RELAY_ADDR").unwrap());

    let mail_contents = fetch_guys();
    let handles = mail_contents.into_iter().map(|(content, subject, to)| {
        send_mail(
            content,
            subject,
            to,
            mail_addr,
            (mail_username.clone(), mail_pw.clone()),
            mail_relay.clone(),
        )
    });

    let results = join_all(handles).await;
}

fn fetch_guys() -> Vec<(&'static str, &'static str, &'static str)> {
    vec![
        ("hey there just testing", "test mail", "mail@example.com"),
        ("yet another test", "test mail 2", "mail@example.com"),
    ]
}

// trying out https://users.rust-lang.org/t/function-that-takes-a-closure-with-mutable-reference-that-returns-a-future/54324/2
async fn send_mail(
    content: &str,
    subject: &str,
    to: &str,
    from: &'static str,
    credentials: (Arc<String>, Arc<String>),
    relay_addr: Arc<String>,
) -> Result<Response, lettre::transport::smtp::Error> {
    let credentials = Credentials::new(credentials.0.to_string(), credentials.1.to_string());

    let email = Message::builder()
        .from(from.parse().unwrap())
        .to(to.parse().unwrap())
        .subject(subject)
        .header(ContentType::TEXT_PLAIN)
        .body(content.to_string())
        .unwrap();

    let mailer: AsyncSmtpTransport<Tokio1Executor> =
        AsyncSmtpTransport::<Tokio1Executor>::relay(relay_addr.as_str())
            .unwrap()
            .credentials(credentials)
            .build();

    let future = mailer.send(email);

    future.await
}

I don't know if it works--I didn't want to try to set up credentials for myself--but it compiles and will fail correctly.

In terms of approaches, my understanding is that 1 handles resources better but at a cost of overhead, while 2 is faster but will lead to memory leaks.

thanks for replying! from my understanding the Arc in your solution is more of a side product related to having send_mail it self be an async function. However it also works fine with the prior types and references without an Arc. what satisfies the borrow checker from my understanding is the direct awaiting (because now mailers &self isn't returned with the Future, and its an Result that then can be returned passing ownership, and what the join_all awaits is not the send call but the send_mail which blocks on the mailer.send(email) by awaiting it):

And I think i am starting to understand now. I thought up until now that because if I'd await there like you are doing i would effectively be using a blocking verison (plus some overhead of the tokio runtime), because I'd be awaiting the individual send mail calls.
However I am starting to suspect that it just works :tm: because the Futures would have their execution triggered by the join_all, thus all running along side each other.

One thing that is still a mystery to me here would then be if tokio would then actually properly handle the resources between the the execution of each of the Futures. My async mental model is very much thread based - which i know is very much wrong, but yeah :grimacing:: Like sending theses mails and their smtp requests should afaik be done asynchronously next to each other because you want to avoid taking up IO with smtp while waiting (same as with http requests). So then is tokio able to infer or just via being a runtime able to then work this out with the futures being nested?

Another inportant observation to make is this:

This is indeed the return type of your function, but when you mark it async, you get syntax sugar, and you don't have to talk about the Pin<Box<dyn Future<>>>, only

and you must use .await in this case to obtain the right type to return.

Note that your original function didn't use .await at all. This means you could've just removed async from your function, and kept the return type as it was. It would still return the Future from the send method, and it would still have the same behavior as the async version.

And a final note: .await it's not blocking. It does stop the current function from progressing, but the executor is free to schedule other Futures onto the same thread, so the mails are sent in parallel. If you find this model confusing, you might want to look into JavaScript's Promises and async/await, which uses a similar model, but is a bit easier to understand (no multithreading, no lifetimes and ownership, etc.).

thank you as well for replying :slight_smile:
whoops yeah it's good that you point out the function signature, I was aware of that but made a mistake in writing the minimum reproduction version (and that what i posted here). my actual code had send_mail as a synchronous function without the async that just returns the future.

regardning the part about being able to remove async and not having the await: this runs into the original problem with the returning borrowed data. if send_mail doesn't return the result but rather the future, it behaves as if returning a closure into which a &mailer gets moved, because of sends signature.

and yes i will definitly read up on async and try to build up my intuition regarding it.