Advice on structure of a complex Rust/Golang/wasm/Svelte app

help
#1

I want to build a static app - a p2p github replacement - using a mix of Rust and Golang compiled to Wasm with a Svelte frontend (because I think Svelte + Rust/wasm has good tooling and I like Rust), which will rely on functionality provided by a Golang library (based on git-bug to implement issues in local and remote repos), and a git implementation (using a suitable Rust library).

Both the git and git-bug parts will need access to a filesystem interface to access github repositories which will be stored on a p2p storage system (https://safenetwork.tech).

So I'm wondering how best to structure this. git-bug uses the Go os package which has a POSIX style fs interface, and my p2p storage will have a similar fs API made available via JS in the browser.

So I think I'll need to create a Rust fs emulation library which:

  • can be called from git-bug in Golang by substituting the Go os package
  • can be called from git-rs (or something providing git functionality) in Rust
  • routes fs calls to the JS p2p storage API

This way my Svelte frontend can just talk to Rust, and the Rust can route any fs calls to the p2p storage API.

I've tested out a couple of trivial wasm apps in Rust and Golang, but not made anything which calls between Golang and Rust yet, so I'm wondering if my approach makes sense.

Any feedback or alternative ideas appreciated.

#2

That is quite a complex project indeed!

While I don't know everything about the languages involved, I think the general approach makes sense. However, I would also break things into slightly smaller pieces.

Here is how I would approach it:

  1. Use tinygo to compile the Go library to a WASM as a target. Since that uses LLVM (unlike the official Go toolchain), it should hopefully be clearer about symbols and be processed similar to Rust. (That is just a guess on my part.)
  2. See if you can get a Go test program to write to the filesystem using WASI's filesystem API. This will also allow you to see what filesystem calls the library needs.
  3. Try to get the git2-rs Rust git library to build for WASM. In particular, the *-sys libray in C might be tricky to cross compile (though at least I added a hack flag to bindgen you can use as a last resort).
  4. Make sure that libgit2 will work well with the WASI filesystem API as well.
  5. Get a simple Rust main to call into the Golang library you wrote before. At this point, you should be able to integrate all the pieces except the P2P part.
  6. Write the Rust library for the surface of the filesystem calls you need.
  7. Profit!!

If any step gives you trouble -- as it very well may -- then you'll at least hopefully have a clearer idea as to how to work around it.

It's not much, but I hope it helps. Good luck!

1 Like
#3

@jhwgh1968 thanks so much for thinking about this and giving me all those suggestions. I suspect this is more complex than I imagined, but it always seems to be the way - I usually end up trying to build things in ways that push into unknown territory!

In the interim I realised that to start I could try just doing this in Go, because there's a go-git library, and although I hadn't realised before, it is already being used by git-bug. For this your first two suggestions are of course still applicable.

There's still a case for Rust, because I've learned that wasm and Go are not yet made for each other, but we'll see. If I get this working, it just might be a case for rewriting git-bug in Rust, but that's way down the line.

I've also discovered that there's a thing, a filesystem layer called go-billy, by the author of go-git which he uses to test it with an in memory filesystem, and which would be an ideal way for me to interface with my p2p Safe filesystem in the browser. More on the go forum, where I posted about using go-git here

So no Rust for now :frowning_face: but it still looks like it will be best in the long run. Step by step!

I'll leave this unsolved for now because any further hints or warnings about my using Rust with git-bug could be useful. Thanks again for your input.