Comparison of way too many Rust ASN.1 DER libraries

[george.eisenfeld]

Hi everybody. This is a bit of a long first post. I hope you like it and I hope I don't come off as too ranty, but I'm having a bit of a hard time which lead me to this investigation.

About me

I'm a newcomer to the Rust community. So far I have largely written toy programs, but an opportunity came up at work to potentially write something in Rust. This is my first foray into building a real application in Rust, and also the first time I've really looked into third-party libraries (everything I've done thus far has been with the standard library). Apologies in advance if this is a bit of a rant, but I'm just venting my frustrations.

For context, I have mostly been working in Golang on a day-to-day basis. While I think that Rust is a very promising language, where it really seems to be lacking is in the standard library department, at least if you're used to the incredibly feature-rich one that Golang provides. I'm trying to translate an existing Golang application because we think Rust is a better fit overall, but the lack of a comprehensive standard library is not helping me here.

I know this is a holy war. I've read all about it. I don't think that my opinions will change Rust, and I just want to say my peace here before I get on with it, but my first challenge is to parse an ASN.1 DER data structure, and I cannot relate to you what a night and day experience this is coming from Golang.

In Golang this is all built into the standard library in the encoding/asn1 package. In general I was really surprised to discover that Rust has no encoding support in the standard library whatsoever and this kind of blew my mind. No hex! No Base64! Apparently people use third-party libraries for this?!

Okay, sorry for that digression, back to Rust.

Which Rust ASN.1 DER library to use?

I went to the Rust Crates site to try to find a third-party ASN.1 DER library. I was somewhat surprised, and a little bit horrified at what I found.

There are just so many, and no real guidance, especially as a newcomer, as to what to use. Here are a few of them I found:

• asn1
• asn1-cereal
• asn1_der
• asn1_der_derive
• asn1derpy
• basn1
• bcder
• dasn1-der
• der
• der-parser
• der_derive
• derp
• eagre-asn1
• picky-asn1-der
• rasn
• red_asn1
• red_asn1_derive
• simple_asn1
• serde_asn1_der
• yasna

20 different libraries! Which one should I use? Trying to google for this it seemed like people just gave random answers with no justification, and everyone had a different favorite library. Which ASN.1 library defines me as a person? Well, I'd like a "simple" library, but I'm also "picky", so which do I
choose?

I couldn't find any good guidance for this online, so here I am trying to write
up my findings for you.

A quick sidebar: this is way too many libraries, especially for someone coming from a language where this is just built into the standard library and works! I get that Rust wants to have a small standard library, but if there's going to be a third-party library ecosystem to support this kind of thing, it'd be
tremendously helpful if the people working on these twenty different competing ASN.1 DER libraries could maybe get together and combine their efforts and collectively work on a single great ASN.1 library, please!

Also at this point anyone thinking of making yet another ASN.1 library for Rust should probably not do that and contribute to one of the existing ones, please! This is already confusing enough.

[george.eisenfeld]

Note to the forum mods: I'm getting a really annoying "Sorry, new users can only put 2 links in a post." message even though I haven't posted any links?! I'm going to try splitting this up into multiple posts and see if that works.

Narrowing down the list

Since there are so many candidates, I decided to start whittling down the list by getting rid of all of the ones with less than 10,000 downloads. I picked this number arbitrarily, but many of them only had hundreds/thousands of downloads. Sorry if there's a really good "diamond in the rough" library I ignored because of this, but this list was too huge and I needed to cut it down to a smaller number of libraries to review in depth.

That left me with this list (which surprisingly doesn't include asn1):

• asn1_der
• bcder
• der
• der-parser
• derp
• picky-asn1-der
• simple_asn1
• yasna

8 libraries to review... it's better than 20 at least!

Comparison of the "top 8" Rust ASN.1 DER libraries

Note that I'm not going to pick a "winner" here. I'd really like other people's help and opinions about that. However I thought I'd make a sort of stats/"feature matrix" which gives you an overview at a glance, and then write up some specific thoughts on each one.

Finally, I'll whittle it down to my top 3 choices, but you can change my mind!

Feature	asn1_der	bcder	der	der-parser	derp	picky-asn1-der	simple_asn1	yasna
Downloads	591k	36k	202k	489k	25k	26k	1M	466k
Year started	2018	2019	2020	2017	2017	2019	2017	2016
Last release	1 mo	4 mo	4 dy	26 dy	1 yr	8 mo	1 dy	11 mo
# of releases	20	12	18	48	13	5	10	10
GitHub	8	10	83❗️	52	0	18❗	7	27
Rev deps	10	5	4	18	3	4	14	22
Panic-free		‍
Zero-copy		‍
no_std
Doc Grade	A	B	A+	A+	B	B	B	B

• : means I found this information
• : tried to find it and couldn't. Probably a no but I may be mistaken.
• GitHub for these are on a repo containing many libraries so they might be unfair

Looking over several of these libraries I noticed some common themes which I incorporated into the table above. For example, three things that stood out to me:

• Panic-free: seems like a really good thing to have
• Zero-copy: I wasn't going out of my way to find this but I noticed several of the libraries offered it so I thought it'd be useful to track
• no_std: I wasn't really sure what this was but it seems like a way to shut off the standard library for embeded projects. This may eventually be useful for me but isn't right away, but I thought I'd track it in the table.

Sorry for the somewhat arbitrary "grades" on the documentation and I didn't want to be too mean because I know writing good docs is hard. But some docs were better than others and I tried to capture that as best I could, particularly libraries that had lots of code examples and did a good job
explaining how they worked.

Looking over the documentation, and I really hope I don't sound like a broken record when I say this, a lot of them had very similar APIs and in fact a very similar API to Golang's encoding/asn1.
That is to say, it looks like several of them rely on traits similar to Golang's BinaryMarshaler/BinaryUnmarshaler such as Source/Sink, Decode/Encode, FromASN1/ToASN1, DEREncodable/BERDecodable, etc. I'm sorry to say it but this really feels like the reason there's so many libraries is just a big bikeshedding debate about names and these libraries developers should just merge them together into one really good library.

[george.eisenfeld]

My top 3 finalists

Based on the information I collected above, I whittled the list down to my top 3 choices which I'll review a bit more in depth and post some notes.

Without further ado, here they are (in alphabetical order):

• asn1_der
• der
• der-parser

Runners up include simple_asn1 and yasna. I feel a bit bad leaving out the most popular library (simple_asn1) by downloads and most popular by reverse dependencies (yasna), but as it turns out I didn't really end up liking them and I'm a bit confused why they're as popular as they are.

I ended up picking these three because they're mature options and I was very excited by these features:

• Panic-free
• Zero-copy
• Great documentation

asn1_der: guaranteed panic-free!

There are several reasons why I liked this library. It's the #2 in terms of downloads and has great documentation. I also like the "official"-sounding name asn1_der even if that's not a great reason.

You might've noticed I gave it a for "panic-free". I discovered it uses a very cool looking no-panic library to actually guarantee that it's panic-free. The other libraries I highlighted merely claim to be panic free. This one actually guarantees it, AFAICT.

der: excellent documentation, interesting overflow handling approach

Of all of the libraries I investigated, der is by far the best documented and one of the most complete, even though it's pretty much the newest library that I investigated. There are extensively detailed code examples, and I found the Decode/Encode traits very easy to understand and like how they provide
a common abstraction that everything else is built on. Also it has an "official"-sounding name.

It was very nice to see that der is written by the Rust Crypto project as my use case is also cryptography. One of the things I looked for in each of the libraries was how they handle overflows, as this is a classical source of bugs in DER implementations. All of the "finalists" appeared to handle overflows, but this one appears to use a Length type which always checks overflows which
I thought was really interesting!

der-parser: mature and well documented

I hate to say it but the main thing I don't like about this library is its name. At first I thought it was a parser-only library, but I found out it actually supports serialization too! Kind of confusing.

All of that out of the way, this seems like a widely used and mature library with excellent documentation. It seems to have the longest, most well-established track record among the "finalists" I picked with the highest number of reverse dependencies.

[george.eisenfeld]

Conclusion

I am still not sure whether to pick asn1_der, der, or der-parser. I would be grateful to hear what people who have firsthand experience with these libraries have to say. And please let me know if there's anything I missed, including new up-and-coming libraries, or if I gave libraries like simple_asn1 or yasna an unfair evaluation.

Also I'd be really interested to hear if there is a better place for these sorts of evaluations that maybe I missed. If there's some project that tracks, that'd be great!

[H2CO3]

@george.eisenfeld

no encoding support in the standard library whatsoever […] I get that Rust wants to have a small standard library

Rust's std is intentionally minimalistic, and it's not a l'art pour l'art, purely theoretical or æsthetic, "we want to be minimal because that's what the God of Computing demands" decision.

There's a very practical and painful reason behind it: once you commit to putting an API into std, there's no going back, it has to be supported forever, due to Rust's backward compatibility guarantees. And we don't want to make the mistake of supporting broken, dangerous, slow, difficult to use, easy to misuse, or otherwise badly-designed and suboptimal APIs.

Many of the core "3rd-party" crates you are thinking about, e.g. rand, and even the serialization framework, serde, are actually written by members of the Rust language/library core teams. They are official, supported, maintained, and very easy to depend on, thanks to Cargo. Developing them separately from std makes it possible to experiment, collect real-life, long-time experience from programmers who use them, and improve the APIs accordingly, without having to continuously think about breaking changes, deprecations, and fixing bugs without requiring a new edition of the language.

---

As for how to choose a crate, a couple of practical considerations would be, in decreasing order of importance:

• Its code quality, that you judge by yourself, by reading its source:
  • Is it nicely-formatted, split into modules and functions?
  • Is it documented extensively in doc comments? Does it provide examples? Does it document fallibility and panics?
  • Does it use Clippy, does it have default and non-default warnings turned on/off?
  • Does it use type-level programming for statically ensuring invariants and protecting against misuse?
  • Does it interoperate with downstream code nicely, e.g. by providing common implementations of standard or widely-used traits, such as Clone?
• Its author, and whether s/he is an official Rust representative
• Its maintenance status and last update date
• Whether it transitively uses unsafe and the amount of unsafe code
• The number of transitive dependencies and the crate size
• The number of downloads it has received, the number of releases, and the date of the first release (how long it has been around).

Based on these criteria, I would recommend the der-parser crate if you are trying to parse the data by hand, or asn1_der and serde_asn1_der if you are looking for Serde support.

[george.eisenfeld]

Thanks! I'm not really familiar with Serde, but it looks potentially quite useful, for even more than just ASN.1.

[H2CO3]

@george.eisenfeld

potentially quite useful, for even more than just ASN.1.

Indeed, Serde is a completely format- and type-agnostic serialization framework. The idea is that it provides a layer of connection between serialization formats (that implement the Serializer and Deserializer traits) and your own custom data types (that implement or derive the Serialize and Deserialize traits). It maps any Rust type onto a set of primitives (such as integers, strings, tuples, key-value maps, etc.), which are presumably supported by most formats.

In this manner, serializer formats need not worry about the specifics of the types they can support, because they can just support the primitives and have Serde automatically generate the necessary code for all custom types. Likewise, your own data types don't need to duplicate code for serializing into JSON, ASN.1, TOML, plist, bincode, etc., because they only need to specify how they convert themselves into a tree of primitive types.

Serde implements the "parse, don't validate" pattern: it fallibly converts from the serialized data to user-defined types directly, instead of the usual approach of going through an intermediate representation and validating it only to dynamically and usnafely extract data from it. Furthermore, Serde uses statically-dispatched traits for all of this, so the performance of the generated and optimized code will usually come close to the performance of hand-written parsers and printers.

[scottmcm]

@george.eisenfeld

which surprisingly doesn't include asn1

Note that there's no contribution gates for crates.io, so this isn't all that surprising. It's relatively common for the first mover to take the most "obvious" crate name, but then for that to end up languishing. Or even just for a new way to come along and people end up moving -- this has happened multiple time with error handling libraries, for example.

[XAMPPRocky]

As one of the authors of those libraries I think I can provide some perspective on why there are so many libraries in this space

Firstly, I think it’s important to understand that ASN.1 is an incredibly complex set of languages, codecs, and data models, both to describe and implement, and it’s an incredible effort to fully support all it has to offer, and while it is a standardised format, it’s complexity allows a wide variety of different encoders and decoders. So it’s quite easy for someone’s library to not be suitable for a given desire, some may want more specificity to the point of only caring about a single standard, others more generic APIs.

Also important to understand that before you could put ASN.1 in std, we’d need equivalents for all of ASN.1’s types like bit strings in std first. Getting all of those in std would take significant design work and consensus building to get those merged and stabilised. So while ideally it might be great if there was a single reliable implementation, the chances of it happening in std are slim to none, and probably still couldn’t cover every use case.

---

@george.eisenfeld

Thanks! I'm not really familiar with Serde, but it looks potentially quite useful, for even more than just ASN.1.

Not to be the bearer of bad news, but it’s actually not possible to correctly implement an ASN.1 parsing library with Serde. I know this because I spent weeks trying to build one, and reached fundamental limits (as well as needing some ugly workarounds) in Serde’s design such as correctly handling Option types in structs, which requires more information than serde can give you. So any library using serde for ASN.1 will be incorrect/incomplete.

---

As an aside, I would recommend checking out my crate rasn, despite the sub 10k download count, I’m pretty sure it’s only crate in that list that also supports BER and CER, and correctly handles lots of edge cases that most of the crates don’t handle correctly when I reviewed their implementations.

It’s also much more ecosystem friendly, as its designed similarly to serde‘s traits, which allows you to have different encoders accept the same ASN.1 type, so if you needed your own encoder you could still use types defined with rasn‘s traits. This is also what lets you define and have the same type encodable to BER, CER, and DER from a single trait implementation.

[H2CO3]

@XAMPPRocky

reached fundamental limits (as well as needing some ugly workarounds) in Serde’s design such as correctly handling Option types in structs, which requires more information than serde can give you.

I'm curious what the limitation regarding Option is. I reckon running into serde bugs before, but none of them was related to incorrect handling of Option.

[XAMPPRocky]

The limitations in serde aren’t specific to Option it just comes up when implementing Option.

The limitations in serde are as follows:

• You’re restricted to serde’s data model. This is fine for formats like JSON where you can easily map data types from JSON to Rust, but if your data format takes advantage of complex types as primitives (e.g. bit strings, object identifiers). It becomes hard to impossible to encode.

• It’s incredibly difficult to pass type metadata, if your data format uses extra information such as what “tag” it has, you can’t simply pass that information from your Deserialize type to your Deserializer decoder. You can work around this some with newtype wrapper hacks, by creating specialised types that send that identifying information in the &’static str using deserialize_newtype_struct to specialised Deserializers that only work for that type.

This can get you most of the way there to implementing ASN.1 with serde, however it makes the types incompatible with other formats, as you have to add newtype wrappers which expect to be used in an ASN.1 context, and thus don’t generate what you expect when you try to serialize to JSON. Even ignoring that caveat, using specialised Deserializers don’t solve everything because…

• All Serializer and Deserializer methods take self by value. This means there’s no real way to get information about the operation after you’ve performed it, other than the data, as you can’t use them at all after you’ve called a de/ser method. So if you need metadata about the operation, (e.g. the total length encoded, tag+data), you’re simply out of luck.

Where this comes up with Option<T> because ASN.1 has no concept of nullable types, specifying optionality is part of the field, and it indicates presence, not nullability. Since most ASN.1 codecs are binary, the only way to distinguish the tag in a field from anything else is by knowing all possible tags beforehand. Another piece of needed metadata is the tag of the type, and this tag can either be the tag of T in the Option or it could be overridden at the field level with a different tag.

Getting this information isn’t currently possible with serde because the only way to pass extra info is to pass newtype deserializers which you then can’t query for information after you’ve called it.

[H2CO3]

I understand the other limitations, however, I think there is a common idiom for solving the following problem:

@XAMPPRocky

All Serializer and Deserializer methods take self by value. This means there’s no real way to get information about the operation after you’ve performed it, other than the data, as you can’t use them at all after you’ve called a de / ser method.

Serializer and Deserializer are often implemented for &mut My[De]Serializer instead of the type itself. This allows callers of methods on these two traits to still inspect the instance after the [de]serializer methods returned.

[XAMPPRocky]

@H2CO3

Serializer and Deserializer are often implemented for &mut My[De]Serializer instead of the type itself. This allows callers of methods on these two traits to still inspect the instance after the [de]serializer methods returned.

I am aware of that pattern, and I might not have been clear about where this was needed that makes this pattern useless. Which is the coder that's passed into Deserialize::serialize/Serialize::serialize , to the Deserialize type, it's just D not &mut D, so even if D: &mut MyDeserializer, in the Deserialize implementation, the Rust code still has to be valid with move semantics.

The workaround this is to ignore the Deserializer you're given and only use concrete types for coders in the implementation, however that significantly reduces the utility of these implementations, since they can only ever code to one format.

[george.eisenfeld]

@XAMPPRocky

As an aside, I would recommend checking out my crate rasn , despite the sub 10k download count, I’m pretty sure it’s only crate in that list that also supports BER and CER, and correctly handles lots of edge cases that most of the crates don’t handle correctly when I reviewed their implementations.

Hi @XAMPPRocky,

Wow! I was really seriously worried that when I did a naive "10k or bust" download limit I'd overlook something, and this is exactly the sort of "diamond in the rough" library I was thinking of! I should probably update my chart above to include this because it looks really promising despite the small number of downloads.

Thank you very much for pointing it out and I will look into it soon and probably update the chart. This is the whole reason I created this post, to discover if there were cool new libraries. Thanks a lot for your reply.

[riking]

@george.eisenfeld

In Golang this is all built into the standard library in the encoding/asn1 package. In general I was really surprised to discover that Rust has no encoding support in the standard library whatsoever and this kind of blew my mind. No hex! No Base64! Apparently people use third-party libraries for this?!

The only reason this is in the Go standard library is to make net/http work.

This then suggests an obvious selection method that I would've started with: look at what the popular HTTPS client / server libraries use for certificate parsing.

@george.eisenfeld

Note to the forum mods: I'm getting a really annoying "Sorry, new users can only put 2 links in a post." message even though I haven't posted any links?! I'm going to try splitting this up into multiple posts and see if that works.

With my Discourse Staff hat on: we're aware, this is a bug caused by heading anchor auto links. We're working on it.

[jsha]

Hi @riking!

@riking

This then suggests an obvious selection method that I would've started with: look at what the popular HTTPS client / server libraries use for certificate parsing.

Unfortunately this won't be too informative. The most popular TLS backend for HTTP clients is native-tls, which uses the platform libraries SChannel and SecureTransport on Windows and macOS, respectively, and OpenSSL on Linux. The second most popular, rustls () uses webpki for parsing certificates. Webpki is great, but does not expose arbitrary DER-parsing functionality.

@george.eisenfeld I feel your pain! I've spent about 6 years writing Go full-time, and 1 year writing Rust part-time. I really miss the robust stdlib, and find it hard to evaluate which crates to use for given functionality. I also spend a lot of time stressing over how many dependencies I'm pulling in (many of which turn out to be from the same person or team). A couple of resources I've found useful:

https://lib.rs is a crate-finding tool that is more opinionated and has a different ranking algorithm from crates.io.

cargo-supply-chain can tell you more about your dependencies, taking into account common ownership and/or publish rights.

@H2CO3

Many of the core "3rd-party" crates you are thinking about, e.g. rand , and even the serialization framework, serde , are actually written by members of the Rust language/library core teams. They are official, supported, maintained, and very easy to depend on, thanks to Cargo.

This was a big stumbling block for me as a beginner. It's not immediately obvious that this is the case, and even after someone told me about this, it was hard to put to use until I learned who some of the members of the core team are.

The cool thing about the Rust community is that it's very welcoming. I went through a similar process to what you did, choosing an HTTP library. I didn't find anything that met all my needs, but I picked something that met most of them, and started sending patches. A few months later I was invited to become a maintainer. More recently I stared sending patches to rustdoc, and as of last week I'm on the rustdoc team!

Thanks for sharing your research process, and best of luck with the rest of your application! I hope you'll come back and share it when it's ready.

[H2CO3]

@jsha

It's not immediately obvious that this is the case

I think it's always a good idea to be curious and start reading the source of the library you are just about to depend on, so as to be able to judge the quality of the code. This will in turn necessarily involve finding its VCS repository. And if that repository happens to be under the rust-nursery or even the rust-lang organization, then well, that should be indicative of something at least.