Good evening,
I'm working on a Axum backend service, which will be used as a "template" for all services in a microservice architecture. The last missing part to make the template complete is a tracing solution which sends the traces to a Zipkin endpoint (in our case this endpoint is provided by Grafana Tempo) with the "B3 Multi" headers (b3 propagation).
To instrument the application I used the tracing crate (tracing on crates.io).
With this every route handler as well as every other subsidiary function looks something like this
/// Route handler for:
/// GET /api/users
#[tracing::instrument(ret, level = "trace", skip(pool))]
#[utoipa::path(...)]
async fn get_users(State(pool): State<DbPool>) -> ApiResult<UsersDto> {
services::user::get_users(&pool).await
}
As far as I understand the concepts of tracing, what my application needs to do when receiving a request is check, if there are already B3 headers set on the incoming request.
If this is the case, the span created for the route handler by #[tracing::instrument(...)] would need to adopt the X-B3-TracingId header as trace ID and X-B3-SpanId as parent span ID.
If no headers are present on the incoming request, the span created by #[tracing::instrument(...)] could generate a new trace ID and span ID but they should probably be automatically set as headers on the outbound response.
As I said this is what I think should happen. Please correct me if I got something wrong.
With my current setup I unfortunately cannot see any of the expected behavior described above. On incoming requests with existing B3 multi headers, the trace ID and span ID are ignored and the newly created span gets a new (unrelated) trace ID. This leads to traces which are not shown as subsequent requests of one another.
The only "good" thing I observed is, that the outbound response contains the X-B3-Sampled header, which I definitely didn't set manually. This gives me a little hope that my current tracing setup at least somewhat works. But apart from this header, there is no other B3 related header set.
To archive the behavior described above I set up the tracing like follows:
use tracing_subscriber::{prelude::*, EnvFilter, Registry};
opentelemetry::global::set_text_map_propagator(
opentelemetry_zipkin::Propagator::with_encoding(
opentelemetry_zipkin::B3Encoding::MultipleHeader,
)
);
// Filter
let filter = EnvFilter::builder()
.parse("back_rust_template")
.into_report()
.change_context(InitError::Tracing)?;
// OpenTelemetry stuff
let tracer = opentelemetry_zipkin::new_pipeline()
.with_service_name("back_rust_template")
// .with_collector_endpoint("http://otel-collector:9411")
.install_batch(opentelemetry::runtime::Tokio)
.into_report()
.change_context(InitError::Tracing)?;
let telemetry =
tracing_opentelemetry::layer::<Registry>().with_tracer(tracer);
// Console stuff
let console = tracing_subscriber::fmt::layer().pretty();
// Create subscriber
let sub = Registry::default()
.with(telemetry)
.with(console)
.with(filter);
// Register stubscriber
tracing::subscriber::set_global_default(sub)
.into_report()
.change_context(InitError::Tracing)?;
// Axum router
let router = Router::new()
.nest("/api",
Router::new().merge(user::router(&db_conn_pool)),
...,
)
.layer(response_with_trace_layer())
.layer(opentelemetry_tracing_layer());
Because this is quite a set of dependencies I think it's also relevant to see their versions and features.
These are the dependencies related to tracing:
# Tracing instrumentation
tracing = "0.1.37"
# Tracing subscriber (for capturing instrumentation globally)
tracing-subscriber = { version = "0.3.16", features = ["env-filter"] }
# Others
tracing-opentelemetry = "0.18.0"
opentelemetry = { version = "0.18.0", features = ["rt-tokio"] }
opentelemetry-otlp = { version = "0.11.0", features = ["reqwest-client"] }
opentelemetry-zipkin = { version = "0.16.0", features = ["reqwest-client"], default-features = false }
axum-tracing-opentelemetry = { version = "0.10.0", features = [] }
To be honest I unfortunately don't understand all of the dependencies and why they are needed. For example I don't really know why all the OpenTelemetry dependencies are still needed. But I assume this is due to the fact that there is no tracing-zipkin crate as far as I know.
First of all thank you for reading all of this!
If some of you know how to get the header propagation to work I would be very happy to hear from you. But I'm also very grateful for help of any other kind, for example switching to another protocol (supported by Grafana Tempo) or a completely different set of libraries.
Thank you all very much 
Edit: Yesterday I tried to use the Jaeger protocol with opentelemetry-jaeger instead of Zipkin, but that doesn't propagate any headers either.