I am developing a project that is a little bit like a compiler, in that it defines a new programming language, can compile that language into bytecode and is partially written in that language. This means that, in order to build my project, I need to already have an executable of that project available to compile to bytecode the source files written in the new language.
Currently, I just ship the parts that are written in the new programming language as precompiled blobs (in addition to their source code) in the repo, and I have a convoluted build system which, assuming we have both the precompiled blobs and their source, allows one to modify the sources and then recompile everything. However, it is quite painful since it requires adding a feature only useful for this process, some conditional code based on that feature, and all in all I have to compile my project thrice, interleaved with human intervention.
I tried automating it with build tools like make, but that ended up in cyclic dependency errors (of course). In the end, I had to write a bash script to tie together, but it's still a bit hacky.
I was wondering what build systems other compilers or compiler-like projects use? I can't be the first one to face this kind of issues.
Thanks for the reference, it's a nice read indeed. In fact, my project currently uses a schema similar to the WebAssembly one described in the article.
However, my question was more about the implementation details rather than the general idea behind bootstrapping. For instance, my project requires version n to compile version n+1, as Rust does, but it does so with an intermediate n+0.5 version. The problem is, version n, n+0.5 and n+1 all share the same repo, but of course not the same commit. Assuming I have, for instance, version n and n+0.5 available, I try to compile version n+1. Compilation fails, I realize there was a bug in the code that led to version n+0.5. Should I delete every change I made between n+0.5 and n+1 to patch the bug, then redo all the work? Should I keep two copies of my whole project? None of these seems like a decent solution...
(I haven't tried to do any of this, so just throwing ideas around...)
You could setup a makefile that builds compiler-binary-abcdef in some kind of archive directory (outside the repository) by checking out revision abcdef into a temporary folder and running the build there. If the makefile in the repository fetches its bootstrapping compiler by recursively calling make in the archive, it should recursively walk back the bootstrapping chain until it finds either a version that's already built or an early version that uses an external compiler like clang or rustc.