How to refactor a code generator

help
1

I am currently responsible for maintaining a code generator. It does this by parsing type definition files and then generating code in the target language. Specifically, its current working steps are as follows:

  1. Parsing type definition file by pest.
  2. convert pest node to struct.
  3. impl generator trait for these struct.
  4. generate codes by minijinja.

In minijinja, I need to generate code based on types, but since there is no type specialization function, type processing can only rely on exhaustive.

Now it is very difficult to fix bugs and add new features. Each modification needs to be tested. And automated testing is very difficult to implement, basically it can only be done manually.

Currently, convert node to struct and impl generate trait are split into two crates. However, both of them are difficult to maintain.

I now hope to refactor it (I have already done it once before) to make it more structured. I really don't want my code to be like this now. :frowning:

2

I have worked on code generation in the past, using Python to generate C code (Python->C). I also created a mess. :sweat_smile: I am also slowly testing Rust->Rust code generation. I am puzzled by your statement:

automated testing is very difficult to implement, basically it can only be done manually.

It was certainly not easy, but in my case all test ran automatically (although test coverage was not close to 100%). Would you care to elaborate what you mean?

3

Currently my project structure is like this:

- c_lib
-- rust_lib(submodule)
--- code generator

There is no dependency between clib and rustlib. The only purpose of using clib as a submodule is to lock the version of the code generator.

The code generator generates three target codes: C/C++/Rust, each target with two variants.

Rustlib itself is compiled into a dynamic library using cc and exports cpp header files. Of course, Rustlib itself can also be used directly.

After I modify the code generator, I have to test whether all three target codes work properly. This means I need to compile all three libraries, generate code, and finally compile the generated code. Just thinking about it is troublesome.

Of course, these are all solvable, but what really pains me is that the code generator is too fragile. Any small change may cause a part to stop working properly.

4

Procedural macros are basically similar to code generators, and perhaps some well-maintained procedural macros can provide some help.

Hi, @dtolnay You are undoubtedly an expert in the field of rust proc_macro, do you have any suggestions about this?

5

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.