Announcing rq - A command-line tool for performing record queries ✨


#1

I created a tool that I call rq (record query). It’s a tool that’s used for performing queries on streams of records in various formats.

Repository: https://github.com/dflemstr/rq

The goal is to make ad-hoc exploration of data sets easy without having to use more heavy-weight tools like SQL/MapReduce/custom programs. rq fills a similar niche as tools like awk or sed, but works with structured (record) data instead of text. It is similar to jq, but supports more record formats and has a far more advanced query engine.

Examples

(Taken from the GitHub repo)

Simple JSON formatting:

highlighting

A more advanced query:

# When searching for Rihanna on Spotify, what are the 3 most
# frequently occuring artists we get back?
$ curl 'https://api.spotify.com/v1/search?q=rihanna&type=track&limit=50' > data.json
$ rq --format=compact \
  'select "$.tracks.items[*]" |
   flatMap artists |
   countBy name |
   flatMap (o)=>{_.toPairs(o)} |
   orderBy 1 desc |
   take 3' < data.json
["Rihanna",49]
["Drake",7]
["Calvin Harris",2]

Record formats

rq supports more record formats than just JSON. At the time of writing, these formats are supported:

  • Apache Avro (input only)
  • CBOR
  • HJSON
  • JSON
  • MessagePack
  • Google Protocol Buffers (input only)
  • YAML

Supporting libraries

rq uses serde for input/output, and the Duktape Javascript interpreter to run query processes, and coroutines/an user-space scheduler to run many Javascript functions asynchronously. (I could make it multi-threaded too but currently it’s not)

While making rq, I thus made a few libraries that I will try to polish a bit more and publish on crates.io:

  • duk and duktape-sys for Duktape C bindings. These bindings are quite advanced, integrating logging, Common.js module loading and arbitrary object interaction from Rust.
  • serde-protobuf for deserializing Protobuf with serde dynamically (i.e. without having to run protoc to generate Rust code). You can also create a custom Protobuf schema via a Rust builder API at runtime if that’s what you wnat.
  • serde-avro for deserializing Avro container files with serde. Should support uncompressed, deflate and snappy compressed container files, but I haven’t tested it extensively.

Help wanted!

rq works well right now and I’m using it a lot in my day-to-day work. However it has a huge surface area and it’s my first major Rust project so any kind of contribution is welcome!

  • I created a bunch of issues in the issue tracker tagged with language L- and expected required experience level E- that anybody can pick up.
  • Please give feedback on the code! I haven’t written a lot of Rust code so a code review of any component is very welcome.
  • Use rq and add the features you need! I’ve only added support for the stuff I have needed personally and I’m not very imaginative so feel free to submit what’s missing! Even feature requests are welcome!

Crate of the Week