I’ve been steadily dribbling out libraries like quantiles or hopper in support of telemetry and log aggregation work I’ve been doing as a part of Postmates Infra. Today we’ve released the main show, cernan!
Cernan is meant to take a broad view of both telemetry–point at time measures of some time series–and log lines–semi-structured data from files–allowing you to collect, aggregate and transform each, as well as associate metadata throughout. Cernan was inspired by high-scale aggregators like brubeck or statsite as well as log-manipulation pipelines like heka or hindsight. Cernan combines all of these into one unified model and is meant to be easy to extend.
In the present release we support ingestion from these sources:
- line-oriented log files
- a native protobuf-based protocol
Ingested metrics can multiplexed through lua programmable filters and, from there, out to these sinks:
- Amazon Firehose
Adding support for new sources and sinks is pretty straightforward. Our load generation tool evans is also released today and I’ve been very, very satisfied how amendable Rust is to constructing high-confidence, high-performance server systems.