I know that Neon can be used to implement Rust libraries in Node, but I wonder if someone has attempted to go the other way and embed Node into Rust?
My use case is the following: I am using a Node.js library, which produces pretty large amounts of data (think hundreds of Mb of JSON). Currently, I communicate between Rust and Node through
stdout, and I need to serialize these huge trees in Node before reparsing then in Rust. This is both memory-consuming and slow.
So your best bet may be to use a binary serialization format that is faster to serialize in Node. Perhaps write a native Node module that exports JS data straight to Rust’s data structures, and then pipe that through stdin?
That makes sense. Perhaps protobuf or something along these lines. The problem is that my Rust data structure is very complex (it’s 20kloc of generated code), so I’m a bit afraid of choking under hundreds of thousands of lines of protobuf
What do you mean “straight to Rust’s data structures”?
If your data was simpler (e.g. just a lot of numbers), then serializing to a typed array (
ArrayBuffer) in pure JS could work too.
If you use
#[repr(C)] structs, without pointers, and ensure they are defined exactly the same on both sides, you’ll be able to just load them into memory in Rust and cast to your struct, rather than parse every field.
The data structure is at https://binast.github.io/binjs-ref/binjs_es6/ast/index.html (the
Script struct, more precisely). As you can see, it’s anything but simple
Also, I just realized that I probably can’t use protobuf. At least, serde tends to choke on the stack depth of the data. Maybe another implementation.