JSON parser preserving formatting/whitespace in round trip (streaming, no_std)

1

Hi all,
I wanted to pick up a question from a thread that's since been closed: No_std streaming JSON parser.

Disclosure: I'm the author of the bufjson crate.

@Vorpal, I noticed your requirement:

I have need for a streaming JSON parser that preserves formatting so that a document can round trip through my program exactly. I.e. I would need spaces and newlines to roundtrip as well as how numbers are formatted (for example 2 vs 2.0).

These requirements are solved by the bufjson streaming JSON parser! In bufjson:

  • All token text is available as the exact original text from the underlying JSON text being parsed.
  • This includes numbers and whitespace (there's a dedicated pseudo-token for whitespace).

You didn't mention strings in your post, but bufjson also allows perfect round-tripping of strings. This is important because most JSON parsers immediately expand escape sequences, so that, for example, the input string "\u0066oo" will be quietly expanded to "foo", preventing you from roundtripping it back as "\u0066oo". The bufjson crate solves this by explicitly separating tokenization/parsing from escape expansion, so that escape expansion is a choice and you always have access to the underlying literal string token if you need it.

The string escape expansion feature is very powerful and there are lots more examples of tricky roundtripping edge cases. Here's just one more. The forward slash or solidus (U+002F) / can be optionally escaped in JSON, so there are at least four ways of writing the same thing: /, \/, \u002f, and \u002F. A typical JSON parser will normalize them all to \ and then whatever output layer you are using may then attempt to re-normalize it: some JSON serializers will write it as \ and others as \/. This creates a lot of possible ways the input can be permuted when you want it to be echoed exactly. Again, bufjson is your solve!

If this sounds useful and you want to find the right place in the crate, take a look at:

  • The lexical::Content trait.
  • The lexical::Content::unescaped method.
  • The free functions lexical::unescape and lexical::unescaped_cmd.
2

@Vorpal, found another one of your old posts that's also related: Format preserving Json parser?. It's probably a bit late for you at this point (!) but I can imagine that others may share the same use case.

3

Looks potentially interesting (assuming it isn't AI slop, I haven't had time to kook past the readme yet, but it has refreshingly few emojis; I hate that this is how looking at a new repo has to start off these days).

I never ended up doing the thing that would have needed such a parser, and at this point I doubt I will (given the amount if free time I have). But I shouldn't say never.

4

Thanks for the reply.

It's all hand-written code that I worked on over many months.

FWIW, not only is there no LLM slop, but my work life is so inundated with workslop of various kinds (and a clueless manager class pushing for more and more and more of it) that a personal project like this is my only outlet to enjoy the process of creation by writing code. Also FWIW my CONTRIBUTING.md specifically bans contributing AI slop and if I'm lucky enough to get any contributions I'll be enforcing it!

Of course, these days with the slop flying fast and loose in every which direction, it's hard to take some stranger internet person's word for it, but I believe the handcrafted nature of the library, which has an API that took me many iterations to stabilize, should make it a pleasure to use and I sure hope you find some time for your project just so you can check it out a bit more!