I plan to write an LLVM frontend for Clojure, partially because it’s awesome, partially for my own education, and partially because I see a demand: Many Clojurists writing scripts must currently employ somewhat lackluster tooling and language support built around the unworkable, unstable server-side infrastructure of node.js, which doesn’t really even implement threading. Conversely, Clojure on its native Java platform takes about a second just to start up on my 3.5GHz quad-core, let alone intern everything inside of an actual Clojure module. The alternative I see is to write a Clojure implementation that can ahead-of-time compile images into assembly. Currently, I call this project Flak (no meaning; sounds cool).
However, much of the pragmatic appeal around the Clojure project is its seamless, near-reflexive operation with imperative language runtimes, e.g. Java from the original Clojure sources, node.js and browser API from ClojureScript — references to variables in global scope and interning modules from their respective packaging systems can be more reflexive than in any other lisp-family language, period, and it isn’t as though this hasn’t been thoroughly explored in other areas of lispdom: Just look at the amazing work on cl-autowrap.
I believe the ability to drop down into a lower level language so effortlessly is an important aspect of the marriage of parallel paradigm and productivity that makes Clojure pragmatic, and remains responsible for a big part of what makes it compelling to use.
The upshot is that seamless interoperation with an imperative ecosystem, the host platform, is part of the Clojure tradition, but presents a bit of a challenge in building something similar on Rust. As I’ve brought this project to lisp users, especially those familiar with Clojure, there has been great excitement around the prospect of that level of interoperation with an ecosystem like Rust’s, which is rapidly gaining in popularity and stands only to grow in the near future (AFaIK). Rust semantics have the added bonus of beautifully complementing Clojure’s approach to concurrency by solving problems in ways that Clojure and Clojurists cannot and/or will not (might as well be writing Java). The ultimate goal would be completely effortless interoperation with Rust embracing Rust idiom through ergonomic declarations of dependencies, and leveraging Cargo in to be able to write forms such as
(extern crate some-lib)
mapping to Rust source like
extern crate some_lib; in the top-level module of a normal crate, and, following that, intern Rust definitions much as Rust code does, e.g.
(use some-lib::SomeStruct) (.meth (SomeStruct::new (& ["Here's" "a" "few" "args"])))
…and call into the appropriate method, as with respect to Java libraries under the JVM. A separate runtime would likely be implemented with a rudimentary generational garbage-collection scheme, incorporating a record in binding meta-data reminiscent of wrapping everything in
Arc<T> before passing vars to subsequent generations, in which I resort to mark-and-sweep: This results in some different fundamental semantics than the ownership system. I hope to loosen up the memory constraints around Rust types partially in the interest of clojuresque flexibility of paradigm, and partially because I simply don’t understand it very well, but don’t find that memory safety to be a big goal for a language whose only implementations are currently garbage-collected. Clojure makes similar concessions with respect to mutability in favor of concurrency, it still runs blazing-fast on top of the JVM. If you feel this should be done differently, I would love to hear how, but I value the ergonomics of this approach.
Here we finally come to the bit that applies to the forum. Although I would love to interoperate with rust, I would also like to avoid resorting to simply using Rust source code as a compile target, for fear of driving up compilation times and having to build around the compiler as a separate component — the consensus among more experienced lispers seems to have been been that an integral feature of a lisp is to be able to JIT-evaluate top-level forms in a loop, dumping their bindings into the image and results into stdout on-demand: It is no coincidence that virtually every self-respecting lisp-family language implementation boasts a read-eval-print loop. The ability to introspect on code as data and program in terms of transformations of this data is arguably what defines a lisp.
Thus, many have suggested that rather than limit myself to statically compiling rustc -> LLVM IR -> asm, building a runtime on top of Rust, I build a separate, minimalist runtime happening to be compatible with Rust’s. There are a few complexities I find a little confounding in this approach, which, especially starting out, may make it more feasible simply to target Rust and somehow dynamically incorporate its artifacts into an image, or perhaps even its mid-level intermediate representation. There are questions, many probably implied somewhere in the midst of the context dump, for either path from this fork in the road, and their answers will determine how I proceed.
Foremost on my mind now: Is it possible to resolve qualified symbols without introspecting on sources in order to make use of precompiled Rust libraries, e.g.
.rlib? (One rather difficult kink to work out is how to robustly hash namespaces consistently with
Of course, there are some considerations that have to be taken into account in the course of addressing this question, but given that even constructors of Structs are exposed essentially as free-standing static functions, with some pipework around fleshing out the runtime, they don’t seem too damning to resolve. This would likely be the simplest solution to exposing quick, dirty interop still satisfactory for everyday use and leveraging what’s already been written in a blazing-fast systems programming language in early iterations without requiring Rust libraries to expose a C interface.
All of these complexities for what would appear to be a rather brittle and lackluster solution to interoperating with Rust led me to my second question: Would it be feasible to somehow enlist
librustc to generate a lisp image? I envision reducing the EDN sexpr frontend to something like Rust high-level intermediate representation and passing it into the rest of the compiler machinery to retrieve bindings that could then be easily serialized and trivially recalled, as in shared object files. To take this approach would probably be best in keeping with the Clojure philosophy of “embracing the host platform.” However, despite my enjoyment of using Rust, I haven’t a good enough idea of its internals to lay out a good implementation plan on the spot.
Maybe I’ve been going about all of this all wrong. My only prior programming has been at the application level, so I would really like some guidance. Looking forward to reading any and all of what y’all have to say. If this is an inappropriate forum for this topic (and it may well be), please be so kind as to point me somewhere I can take my inquiry and my wall.