Why and how Tracing spans

HJVT · March 2, 2026, 3:51pm

I am looking at an observability setup, in which logs go through tracing, then get exported in hotel format, and end up in Grafana.

Where I look at them, and realize that all the fields I've painstakingly recorded into my spans have disappeared without a (ha) trace!

It seems like I need to manually attach them to events. We'll, we've got a Layer, I'll just add a on_event and record extra fields from a span into it. Except that the record method on Event takes... A Visitor? Which sounds like the opposite of what I want, despite the docs otherwise using verbiage like "record fields into an Event".
The API of tracing seems to be purporse-made to be confusing to my brain specifically.

Anyways, am I even going in the right direction here?

nerditation · March 3, 2026, 1:58am

a visitor is just a callback object.

the term "visitor" is very metaphoric, just like many OOP terminologies, and I believe it was the GoF design patterns book that coined it, or at least made it a "standard" term.

I feel the same from time to time.

it takes some time to get familiar with the tracing API, but there are good reasons for the design, mostly for efficiency and performance.

in tracing, most heavy duty work is done "lazily" or "on-demand" on the subscriber side, so that the user side ^[1] can have very small performance overhead for features they don't use or are disabled.

this trade-off made the subscriber side API a little bit complicated. notably, it is heavily callback based, not query based, by which I mean, suppose your subscriber (or a layer of it) want to access the fields of an event, you can't simply "iterate" through some form of name-value pairs, you have to "record" them using a callback (or, "visitor").

the reason is, spans and events are meant to be lightweight, they DON'T store the fields (e.g. for later retrieval) when the user code records any! it is the subscriber who decide how to deal with the field data: to store them in memory, to serialize them to persistent storage, to send them to a remote server, etc. whatever you do, you need a visitor for it.

not just the "app" code, also library code too! ↩︎

HJVT · March 3, 2026, 2:23am

So if I'm reading this right, what I should be doing is to just write the fields, from both the event and the span, out into my OTEL delivery method.

nerditation · March 3, 2026, 3:50am

I don't know about your actual use case, can't comment on that, but you may refer to the tracing-opentelemetry crate's Layer implementation, which uses a SpanEventVisitor to extract the field data from the tracing events into an opentelemetry::Event struct.

in order to do so, the visitor needs to hold an mut reference to an opentelemetry::Event:

github.com/tokio-rs/tracing-opentelemetry

src/layer.rs

91c4faa00


      
          let mut otel_event = otel::Event::new(
              String::new(),
              crate::time::now(),
              vec![
                  KeyValue::new(
                      Key::new("level"),
                      Value::String(meta.level().as_str().into()),
                  ),
                  target,
              ],
              0,
          );
          
          let mut builder_updates = None;
          event.record(&mut SpanEventVisitor {
              event_builder: &mut otel_event,
              span_builder_updates: &mut builder_updates,
              sem_conv_config: self.sem_conv_config,
          });

where the callbacks simply re-packages the field name and value into opentelemetry::KeyValue, and pushes them into events.attributes, after checking for certain special keys, e.g.:

github.com/tokio-rs/tracing-opentelemetry

src/layer.rs

91c4faa00


      
          fn record_i64(&mut self, field: &field::Field, value: i64) {
              match field.name() {
                  "message" => self.event_builder.name = value.to_string().into(),
                  // Skip fields that are actually log metadata that have already been handled
                  #[cfg(feature = "tracing-log")]
                  name if name.starts_with("log.") => (),
                  name => {
                      self.event_builder
                          .attributes
                          .push(KeyValue::new(name, value));
                  }
              }
          }

Topic		Replies	Views
More tracing questions help	5	1376	July 26, 2023
Adding custom fields to tracing events help	3	3447	March 17, 2022
How to extract message from tracing `Event`, in custom impl of `Layer` trait help	7	1133	October 3, 2024
Changing a field's value in a `tracing` span help	3	3008	February 10, 2024
Global fields to always log on any event in tracing_subscriber help	4	2668	January 24, 2023

Why and how Tracing spans

Related topics