I'm currently working on a project for reducing boilerplate for creating Rust CLI applications. I've been trying to work on things on and off for a while now, and it inevitably leads to failure as soon as I try making it variable on what output format to use for console logging. I'll also be optionally adding OpenTelemetry support if environment variables for it are set.
Here are the goals:
- (working) Vary the console output destination based on a CLI param
- (working) Vary the console output format based on a CLI param
- (working) Filter the console output based on
RUST_LOGenv viaEnvFilter - (working) The above should exist as its own layer so that other layers will not be filtered by console filtering.
- (not working) Add another parallel output to a file.
Note that I'm not involving OpenTelemetry yet but I'm trying to get to a state where I have two separate layers: one for the console with its own filtering rules, and another to an arbitrary place (I'm just using json to a file for now) which is not filtered.
My working code can be seen here in a Gist but I'll post it here as well:
//! This quick CLI demo shows how to set up tracing for console logging with a custom format defined
//! at runtime. We *must* do type erasure here by putting each "format" (layer) inside of a Box, but
//! it does indeed work.
// :dep clap = { version = "4", features = ["derive"] }
// :dep sysexits = "0.9"
// :dep tokio = { version = "1", features = ["full"] }
// :dep tracing = "0.1"
// :dep tracing-appender = { version = "0.2", features = ["parking_lot"] }
// :dep tracing-core = "0.1"
// :dep tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
use clap::{Parser, ValueEnum};
use std::fmt::Display;
use sysexits::Result;
use tracing_appender::non_blocking::{NonBlocking, WorkerGuard};
use tracing_core::Subscriber;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry::LookupSpan;
use tracing_subscriber::{EnvFilter, Layer};
#[derive(Debug, Clone, Parser)]
pub struct App {
#[clap(short = 'F', long = "format", default_value_t)]
format: LogFormat,
#[clap(short = 'D', long = "dest", name = "DESTINATION", default_value_t)]
dest: LogDest,
}
#[derive(Debug, Clone, ValueEnum)]
pub enum LogFormat {
Full,
Pretty,
Compact,
Json,
Off,
}
impl Default for LogFormat {
fn default() -> Self {
Self::Full
}
}
impl Display for LogFormat {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"{}",
match self {
Self::Full => "full",
Self::Pretty => "pretty",
Self::Compact => "compact",
Self::Json => "json",
Self::Off => "off",
}
)
}
}
#[derive(Debug, Clone, ValueEnum)]
pub enum LogDest {
Stderr,
Stdout,
}
impl Default for LogDest {
fn default() -> Self {
Self::Stderr
}
}
impl Display for LogDest {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"{}",
match self {
Self::Stderr => "stderr",
Self::Stdout => "stdout",
}
)
}
}
// important shit
impl App {
pub fn console_layer<S>(
&self,
writer: NonBlocking,
) -> Option<Box<dyn Layer<S> + Send + Sync + 'static>>
where
S: Subscriber + for<'span> LookupSpan<'span>,
{
match self.format {
// FIXME missing is extra field configuration per type of layer, with type erasure we
// can't configure them
LogFormat::Full => Some(
tracing_subscriber::fmt::layer::<S>()
.with_writer(writer)
.boxed(),
),
LogFormat::Pretty => Some(
tracing_subscriber::fmt::layer::<S>()
.pretty()
.with_writer(writer)
.boxed(),
),
LogFormat::Compact => Some(
tracing_subscriber::fmt::layer::<S>()
.compact()
.with_writer(writer)
.boxed(),
),
LogFormat::Json => Some(
tracing_subscriber::fmt::layer::<S>()
.json()
.with_writer(writer)
.boxed(),
),
LogFormat::Off => None,
}
}
pub fn writer(&self) -> (NonBlocking, WorkerGuard) {
match self.dest {
LogDest::Stderr => tracing_appender::non_blocking(std::io::stderr()),
LogDest::Stdout => tracing_appender::non_blocking(std::io::stdout()),
}
}
}
#[tokio::main]
async fn main() -> Result<()> {
let args = App::parse();
eprintln!("{args:?}");
let filter = EnvFilter::from_default_env();
let (writer, guard) = args.writer();
let formatter = args.console_layer(writer);
// layer filters first and then outputs (if an output is defined)
let layer = tracing_subscriber::registry().with(filter).with(formatter);
tracing::subscriber::set_global_default(layer).expect("unable to register telemetry");
// if we hit a panic, send it to tracing as well
std::panic::set_hook(Box::new(move |panic_info| match panic_info.location() {
Some(location) => {
tracing::error!(
"panic at {}:{}: {}",
location.file(),
location.line(),
panic_info
)
}
None => {
tracing::error!("panic: {}", panic_info)
}
}));
// test the logging
tracing::trace!(param = "a", "it's a trace");
tracing::debug!(param = "b", "it's a debug");
tracing::info!(param = "c", "it's an info");
tracing::warn!(param = "d", "it's a warning");
tracing::error!(param = "e", "it's an error");
// manual clean up
drop(guard);
Ok(())
}
Now, within main, I'm attempting to add a JSON file appender and my code has been updated to the following:
#[tokio::main]
async fn main() -> Result<()> {
let args = App::parse();
eprintln!("{args:?}");
let filter = EnvFilter::from_default_env();
let (writer, guard) = args.writer();
let formatter = args.console_layer(writer);
// ERROR here in calling the boxed method
let layer = tracing_subscriber::registry().with(filter).with(formatter).boxed();
let file_layer = Some(tracing_subscriber::fmt::layer()
.json()
.with_writer(File::create("log.json").unwrap())
.boxed());
tracing::subscriber::set_global_default(
tracing_subscriber::registry().with(layer).with(file_layer),
)
.expect("unable to register telemetry");
}
The massive type failure is:
error[E0599]: the method `boxed` exists for struct `Layered<Option<Box<dyn Layer<Layered<..., ...>> + Send + Sync>>, ...>`, but its trait bounds were not satisfied
--> examples/logconfig.rs:135:77
|
135 | let layer = tracing_subscriber::registry().with(filter).with(formatter).boxed();
| ^^^^^ method cannot be called due to unsatisfied trait bounds
|
::: /home/naftuli/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-subscriber-0.3.19/src/layer/layered.rs:22:1
|
22 | pub struct Layered<L, I, S = I> {
| ------------------------------- doesn't satisfy `_: Layer<Layered<EnvFilter, Registry>>`
|
= note: the full type name has been written to '/home/naftuli/devel/src/github.com/naftulikay/slowburn/target/debug/examples/logconfig-b1e66d936692f11a.long-type-9788004849417096537.txt'
= note: consider using `--verbose` to print the full type name to the console
= note: the following trait bounds were not satisfied:
`Layered<EnvFilter, Registry>: __tracing_subscriber_Layer<Layered<EnvFilter, Registry>>`
which is required by `Layered<Option<Box<dyn __tracing_subscriber_Layer<Layered<EnvFilter, Registry>> + Send + Sync>>, Layered<EnvFilter, Registry>>: __tracing_subscriber_Layer<Layered<EnvFilter, Registry>>`
For more information about this error, try `rustc --explain E0599`.
This seems it might be related to the EnvFilter but I'm not sure. As shown above, if I only have one logical layer (filter to output), I don't get the compiler error. As soon as I attempt to have another parallel layer, that's when everything descends into madness.
I've shed a lot of blood getting to this point, I've made many attempts over probably more than a year to get here to actually be able to vary the output format successfully.
I've tried putting the EnvFilter in an Arc and that didn't seem to do the trick, I don't think with will accept an Arc. I assume that I might need to do additional boxing for type erasure but I'm not sure where to start.
If I'm able to get this working, I'll definitely open source it since getting a proper tracing setup for a basic CLI application has been a huge challenge for me and I want to save others the time and effort if possible (including myself from ever having to repeat this again).
