Embedding Node.js in Rust?


#1

I know that Neon can be used to implement Rust libraries in Node, but I wonder if someone has attempted to go the other way and embed Node into Rust?

My use case is the following: I am using a Node.js library, which produces pretty large amounts of data (think hundreds of Mb of JSON). Currently, I communicate between Rust and Node through stdin/stdout, and I need to serialize these huge trees in Node before reparsing then in Rust. This is both memory-consuming and slow.


#2

I’m guessing your bottleneck is not in Rust<>Node communication (stdin is pretty fast), but inside Node.js where it needs to convert JavaScript objects into something that can be exported outside V8. If you’re hoping to avoid the conversion step, you’ll end up using V8’s internal representation of objects, which is going to be relatively slow compared to Rust’s structs.

So your best bet may be to use a binary serialization format that is faster to serialize in Node. Perhaps write a native Node module that exports JS data straight to Rust’s data structures, and then pipe that through stdin?


#3

That makes sense. Perhaps protobuf or something along these lines. The problem is that my Rust data structure is very complex (it’s 20kloc of generated code), so I’m a bit afraid of choking under hundreds of thousands of lines of protobuf :slight_smile:

What do you mean “straight to Rust’s data structures”?


#4

If your data was simpler (e.g. just a lot of numbers), then serializing to a typed array (ArrayBuffer) in pure JS could work too.

If you use #[repr(C)] structs, without pointers, and ensure they are defined exactly the same on both sides, you’ll be able to just load them into memory in Rust and cast to your struct, rather than parse every field.


#5

The data structure is at https://binast.github.io/binjs-ref/binjs_es6/ast/index.html (the Script struct, more precisely). As you can see, it’s anything but simple :confused:

Also, I just realized that I probably can’t use protobuf. At least, serde tends to choke on the stack depth of the data. Maybe another implementation.