I'm working on writing a program that processes the JSON for the reMarkable tablet. There is a field in the JSON (iconCode) that outputs a Unicode character in escaped form. I want to use serde / serde_json to deserialize this file, add custom templates, and serialize it back out. Here is a trimmed example of the template.json file:
When I've attempted to round-trip this, I have only been able to get it to output as "iconCode": "" or in Rust's escape form of \u{e9fe}. When I use those versions, the remarkable does not display the template correctly in the selection UI for the first option and fails to correctly display any template with the second.
I have a full example with tests in this branch if that is helpful. I feel like this should be possible, but none of the combinations I've tried so far have ended up successful. I've tried String, Vec<u8>, newtype wrapper around String with a custom Serialize instance, and other approaches.
My understanding so far is that this RFC removed the ability to represent \u#### style escapes in Rust strings and that's causing the straightforward approach to not be successful.
That means that remarkable's json handling is simply broken, as "\ue9fe" and "" are equal as the spec is concerned.
Perhaps its parser is attempting some other encoding? Crazy idea – push '\u{feff}' char at the front of the file? Another crazy idea – something in LOCALE settings?
What is this straightforward approach you've tried? Is this escape_unicode? Yeah, this escapes in Rust's escaping style which doesn't match json's. Perhaps just try writing escaping manually (as a post-processing step after json is serialized). Playground.
let mut out = String::with_capacity(s.len());
for c in s.chars() {
if c as u32 <= 127 {
out.push(c);
} else {
write!(out, r"\u{:4x}", c as u32);
}
}
(this still relies on json serializer to handle all the other escapes, and just escapes non-ascii utf charactes)
Edit: this loop won't cover full unicode range! See @quinedot's response below
I thought it might be tricky/tedious to target JSON strings so as to be robustly correct, but as it turns out JSON whitespace (and thus all non-string data) is supposed to be ASCII.
Here's an updated playground to handle the surrogate pair cases. You could alternatively make a newtype that implements Display.