Cernan: a telemetry and log aggregation server


#1

Hi all!

I’ve been steadily dribbling out libraries like quantiles or hopper in support of telemetry and log aggregation work I’ve been doing as a part of Postmates Infra. Today we’ve released the main show, cernan!

Cernan is meant to take a broad view of both telemetry–point at time measures of some time series–and log lines–semi-structured data from files–allowing you to collect, aggregate and transform each, as well as associate metadata throughout. Cernan was inspired by high-scale aggregators like brubeck or statsite as well as log-manipulation pipelines like heka or hindsight. Cernan combines all of these into one unified model and is meant to be easy to extend.

In the present release we support ingestion from these sources:

  • statsd
  • graphite
  • line-oriented log files
  • a native protobuf-based protocol

Ingested metrics can multiplexed through lua programmable filters and, from there, out to these sinks:

  • Amazon Firehose
  • InfluxDB
  • Native
  • Prometheus
  • Wavefront

Adding support for new sources and sinks is pretty straightforward. Our load generation tool evans is also released today and I’ve been very, very satisfied how amendable Rust is to constructing high-confidence, high-performance server systems.


#2

Awesome domain for Rust. Thanks for letting us know about it.