Hello All,
I have just responded to 2019 Nim Community Survey and got the sense that being more active in the community is appreciated, even wanted, so decided to ask for opinions about an issue I have.
I am working on an experimental web framework where I utilize Nim's metaprogramming features which I really love. Most things are working very well and I am quite happy with the way things are progressing, except this one issue.
Here is what I tried and failed:
The error I got by trying to manipulate the AST is typechecked nodes may not be modified and when I searched in the code, I found about allowSemcheckedAstModification legacy option.
Since it is marked legacy and the idea of modifying AST that already passed semantic checks didn't feel very bright, I went with an alternative solution where I call the compiler twice:
- I compile app.nim here to generate the AST I wanted, then I write repr from that AST into a nim file in the same folder.
- Then I compile that generated file.
This is done automatically and not really too bad, but I wonder if anyone would suggest something better. I don't know if there is a way to run semcheck again manually from inside a macro. That way, every time a new record type is defined (.i.e record macro is called) modified AST may be checked again so in that case I may use the legacy option without worry.
This is my first message on the forum, but I have browsed through and read a lot of discussions. I have started to use Nim more and more in the last few months after more than a year of trying it the first time. Just wanted to say happy to be here. :)
Hey and thank you!
I am aware of the weirdness of the thing I am trying to achieve, but happy to see that there is at least some other person doing something similar. :)
Your understanding about record calls is perfect; that is exactly how it works. The issue is actually that app.nim is imported by all record definers as those generated constants in app.nim are used in the generated procedures in record defining modules. For that, app.nim gets called first.
What I do currently, i.e. a two-pass compilation works perfectly to achieve the end result I want. A better (future) solution would be importing the compiler itself and having a custom compilation process for this. A midway solution can be just using the legacy option and not worrying about type checking the modified AST since all the added statements are just constant definitions where I am already sure about the type correctness.
I was just wondering if there is already a way to do this, hence this post.
Also, I now just wonder if it would work if I make the type for app macro typed and record for untyped. I haven't tested this yet, since I already changed the whole process to generate a nim file and this possibility only occurred to me in the last half hour, but it seems like a good idea. Compiler probably runs macros in two phases first untyped ones and then typed ones. :D
Thank you again for the kind welcome and informative answer!
I'm not sure if it's applicable to your issue but when I need to generate AST in multiple phase, I just store intermediate values in a {.compileTime.} registry and generate the final AST at the end.
For instance Synthesis state machines can be generated from proc from multiple modules and the result can be generated in place(s) where it's needed. https://github.com/mratsim/Synthesis/blob/v0.1.0/synthesis/factory.nim#L77
Hey,
Thank you so much for the answer and the link. That is almost exactly how I was doing it, but for the sake of simplicity, I didn't give the full definitions in the diagram I provided.
However, my issue is not the generation of an AST combining multiple macro calls. It is that this generated AST is injected where these multiple modules all import and use.
Expanding the example in the diagram a little bit more:
- There is an app.nim that is imported by multiple modules, let's say product.nim, supplier.nim, and customer.nim.
- The generated AST contains constants defined based on these importing modules, for example productTypeIndex = 1, supplierTypeIndex = 2, and customerTypeIndex = 3.
- These modules will use the generated constants, for example customer.toHtml() proc (which is also generated) will use app.customerTypeIndex.
Of course if it was just these integers, it might have been better to have those in each generated module, but there are some large strings that are generated by combination of all the defined types.
Update: Idea about typed/untyped failed. Having typed/untyped arguments doesn't change the order of call for macros, even in the same module. Compiler calls all macros in the same order they are called in the source code with no phases if I understand correctly.
I will try to legacy option and/or go back to generating nim file for now. :)
Thank you all for the answers.
For the record, after trying a few other things, I went back to two-phased compilation, i.e. first writing the generated source code in a file and then compiling properly. I have, however, added a conditional symbol (-d option to compiler) to first phase and using this, eliminated running most other macros from this phase. This way, I don't have compilation output doubled which was the only annoying thing about running the compiler twice.
While I didn't originally like the idea of adding the generated source file into the codebase, it turned out to be quite useful to have the source there for development. I even committed it into the repo.
I love Nim. With every trouble resolved, the overall experience gets only better.