Single trait to implement read/write/list files from local and remote sources

[tirithen]

I'm trying to create a web service that can stream files from various sources. I want to declare a Source trait that each source must implement with methods for listing, reading and eventually writing files but I have a hard time finding the right pattern.

In the code below I get problems with Source not being "object safe" due to the generic parameter R.

What would be a good pattern to use to have multiple source types some local, some remote/network ones implement the same Source trait to read/write/list files?

use std::collections::HashMap;

use anyhow::{anyhow, Result};
use async_std::path::PathBuf;
use async_trait::async_trait;
use tokio::{io::{BufReader, AsyncRead}, fs::File};

#[async_trait]
pub trait Source {
    // async fn list(&self, path: PathBuf, path_prefix: PathBuf) -> Result<Vec<FileMeta>>;
    async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>>;
    // async fn writer<W: AsyncWrite>(&self, path: PathBuf) -> Result<BufWriter<W>>;
}

#[derive(Clone)]
pub struct Local {
    root: PathBuf
}

impl Local {
    pub async fn new(root: PathBuf) -> Result<Self> {
        Ok(Self { root: root.canonicalize().await? })
    }

    fn root(&self) -> PathBuf {
        self.root.clone()
    }

    async fn resolve(&self, path: PathBuf) -> Result<PathBuf> {
        let path = path.strip_prefix("/").unwrap_or(&path);

        let mut result = self.root();
        result.push(path);

        result.canonicalize().await?;

        if !result.starts_with(self.root()) {
            return Err(anyhow!("Requested path is outside source root"));
        }

        Ok(result)
    }
}

#[async_trait]
impl Source for Local {
    async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>> {
        let file = File::open(self.resolve(path).await?).await?;
        let reader = BufReader::new(file);
        Ok(reader)
    }
    
    /*
    async fn writer<W: AsyncWrite>(&self, path: PathBuf) -> Result<BufWriter<W>> {
        todo!()
    }
    */
}

/*
The idea is to allow other file sources, HTTP, SSH, S3 ect. as long as they implement 
the Source trait 

#[derive(Clone)]
pub struct RemoteHTTP {
    server_url: String
}

#[async_trait]
impl Source for RemoteHTTP {
    async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>> {
        todo!()
    }
    
    async fn writer<W: AsyncWrite>(&self, path: PathBuf) -> Result<BufWriter<W>> {
        todo!()
    }
}
*/

pub struct Config {
    sources: HashMap<String, Box<dyn Source>>,
}

impl Config {
    pub async fn load() -> Result<Self> {
        let local = Local::new("/tmp/".into()).await?;
        // let remote = RemoteHTTP::new("https://example.org".into());

        let mut sources: HashMap<String, Box<dyn Source>> = HashMap::new();
        sources.insert("local".into(), Box::new(local));
        // sources.insert("remote".into(), Box::new(remote));

        Ok(Self { sources })
    }

    pub fn sources(&self) -> HashMap<String, Box<dyn Source>> {
        self.sources.clone()
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let config = Config::load().await;

    // Store various sources into a config map
    let local = Local::new("/tmp".into()).await?;
    config.sources.insert("local".into(), Box::new(local));



    // Create a read stream from one of hhe sources
    if let Some(source) = config.sources.get("local".into()) {
        let reader = source.reader("a-file".into()).await?;
        // stream data with an actix HTTP service using: HttpResponse::Ok().streaming(reader)
    }

    Ok(())
}

[semicoleon]

You've actually got a more fundamental error here, that the other errors were hiding. If you comment out basically everything except the trait and the impl you get this error

error[E0308]: mismatched types
   --> src/main.rs:56:12
    |
53  |     async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>> {
    |                     - this type parameter
...
56  |         Ok(reader)
    |         -- ^^^^^^ expected type parameter `R`, found struct `tokio::fs::File`
    |         |
    |         arguments to this enum variant are incorrect
    |
    = note: expected struct `tokio::io::BufReader<R>`
               found struct `tokio::io::BufReader<tokio::fs::File>`

The construct you want there is impl AsyncRead not a generic type, though impl Trait in trait methods isn't stable. You could use an associated type, but that will cause problems too since you'll have to specify a concrete value for the associated type to produce a trait object.

Instead you can just use More Trait Objects

#[async_trait]
pub trait Source {
    async fn reader(&self, path: PathBuf) -> Result<Box<dyn AsyncRead>>;
}

#[async_trait]
impl Source for Local {
    async fn reader(&self, path: PathBuf) -> Result<Box<dyn AsyncRead>> {
        let file = File::open(self.resolve(path).await?).await?;
        let reader = BufReader::new(file);
        Ok(Box::new(reader) as Box<dyn AsyncRead>)
    }
}

There are some methods on AsyncReadExt you won't be able to call on a trait object though.

I will also note you appear to be mixing tokio IO types with async_std IO types (specifically PathBuf). async_std appears to just lazily spin up its runtime, but having two separate async runtimes going is probably not ideal.^[1]

---

1. you can see both runtimes are active by stopping the program in a debugger right after canonicalize gets called and listing the active threads. You'll see tokio runtime threads and async_std runtime threads ↩︎

[tirithen]

Thank you so much for the guidance, even if I have done some Rust coding by now, I still have a lot left to learn about what is possible with traits and dyn.

And sure there is not much sense using two runtimes, I'll clean that up in time.

This seem to be exactly the solution that I needed.