Hello, I have a small feature proposal that I didn't feel comfortable making an RFC for.
Synopsis
Introduce a "wrap" statement which is allowed only on top-level code. wrap X replaces the AST of the entire program on all following lines with the result of passing it through the macro X.
Example
# a.nim
macro implementRecur(n: untyped): untyped =
## Allows functions to refer to themself by calling ``recur()``
## To suppress this behavior, annotate the function with ``{.literalRecur.}``
...
# b.nim
import a
wrapwith a.implementRecur
proc factorial(n: int): int =
if n == 0:
return 1
return n * recur(n - 1)
proc funky(): bool {.literalRecur.} =
let recur: bool = false
return recur
Rationale
Currently, the most typical way to implement a macro similar to implementRecur would be to implement a recursive macro which operates on function definitions. Then, all function definitions that want to reap the benefits of this behavior must be annotated with {.recursive.}.
To me, this feels redundant. I don't want to have to say "hey, I'm using the recur keyword over here"" every time I use the recur keyword; it should be obvious by the fact that I'm using the ``recur`` keyword.
Post-script: But, okay, what if I actually meant to refer to a variable named recur? That's what {.literalRecur.} is for. You have to annotate one case or the other; it would be nice to annotate the (likely) rarer one (in this case, literalRecur). If referring to recur literally is more common in a use-case, don't use wrapwith and instead annotate each function.
Consider the => macro from sugar. It requires no special annotation or extra code in order to use; the macro itself is enough. Why? Because Nim happens to provide sufficient tools for it to work (namely, auto). Should these tools not have existed, => would need an annotation much like implementRecur or {.recursive.}.
I posit that the reason the example recur macro does need annotation is simply because Nim happens to not supply enough tools for it to do its job. If, for instance, macros had the power to traverse up the AST as well as down, implementRecursive and {.recursive.} wouldn't be needed; a macro recur(ast: untyped): untyped) would be sufficient.
Noting this, I claim that these two macros are of "equal conceptual power". That is, they work on a similar enough level that I claim that neither should require annotation.
Now, the example recur macro does need annotation, the best we can do is to make it as unobtrusive as possible. This is what wrapwith is about.
This proposal does not add any novel behavior to the language. wrapwith X \n restOfCode may be currently written instead as X: \n\t restOfCode. Since no novel behaviour is added, the proposal can (in some sense) be considered fairly safe.
Removing or adding a DSL will no longer require a change in indentation, which is much nicer for VCS.
PostScript
I've had some trouble reifying my thoughts here, largely because a lot of the reason that I think this should be a feature is simply because it "feels right" to me. Arguments 1 & 2 are somewhat philosophical and perhaps poorly explained. If so, I apologize, and they may be ignored.
Quelklef and I discussed this a bit on IRC. I have mixed feelings about it. But I do think it is worth making an RFC for.
The most well known example of a similar feature that I have seen is Racket, which lets you have module level reader macros. (@Quelklef, I was finally able to find the name of the feature: Racket languages)
@see: https://docs.racket-lang.org/guide/hash-languages.html
This is how Racket does things like their Scribble DSL.
My two concerns are: 1.) That it gives too much freedom, making it too easy for people to make their own "flavors" of Nim, so to speak, which can make code communication, sharing, and re-use harder. The counter argument is that we already have this problem, Macros can be abused even today, so we have to teach restraint and responsibility when using Macros. idk the right answer here.
2.) At some point a feature like this crosses the boundary from Macro into "compiler plugin". It might be better / more broadly useful to instead focus on the modularization plan for the Nim V2 as described here: https://github.com/nim-lang/Nim/milestone/5
That would get us this feature, plus a lot of other benefits. At the cost of being a much more difficult and longer term project though.
can't you just do this?
# a.nim
macro implementRecur*(n: untyped): untyped =
## Allows functions to refer to themself by calling ``recur()``
## To suppress this behavior, annotate the function with ``{.literalRecur.}``
...
# b.nim
import a
{.push implementRecur.}
proc factorial(n: int): int =
if n == 0:
return 1
return n * recur(n - 1)
{.pop.}
proc funky(): bool {.literalRecur.} =
let recur: bool = false
return recur
# test.nim
import strutils
echo "Hi $1" % ["Planet"]
# wrap.nim
import macros
macro wrap(f: string): untyped =
result = parseStmt f.strVal.staticRead
echo result.treeRepr
# feel free to magle that thing in any way you like here
wrap("test.nim")
they make semantics clear in one place in the source code
For this reason per-module macros invocations should maybe be restricted to the top of the module (as #? strongSpaces is now), with only imports and static code allowed before them: the user anyway has to look at the module's top for imports, to know how to read/understand the module's code. But anyway for now imports are allowed throughout the modules, to read a part of it the user has to look through all preceding top-level code. The proposed per-module macros are supposed to be invoked only at the top-level too, not much difference with imports.
Who gets to dine first?
The macros are invoked in the order in which their invocations follow, the user is in full control, the macro's author doesn't and shouldn't decide this. It's the same with today's per-block and per-routine macros, again no difference.
For this reason per-module macros invocations should maybe be restricted to the top of the module
OK, that would be much better, but still pretty non-local.
the user anyway has to look at the module's top for imports, to know how to read/understand the module's code
That's a good point, imports are another non-local thing we already have, just like converters. The latter should IMHO be kicked out of the language spec and for the former I have no better solution. I just think that adding another "thing at the top of the source you have to keep in mind" makes things less obvious.
... not much difference with imports.
The macros are invoked in the order in which their invocations follow, ...
OK, that sounds workable. Would following wrapwith statements be part of the AST that a wrapper macro gets to mangle? I hope not, because IMHO that could produce very non-obvious results.
To wrap it up: I see the advantages of wrapwith, but I'm not sure they warrant giving up good old "if it's not indented, it does what you think it does".
Your points are generally correct, just don't fit to Nim as it is - they are not satisfied already.
Imports just add symbols from other modules to a module
Not in Nim.
You yourself already mentioned converters - code starts to work differently just by importing a module.
Add to this methods - just by importing a module you change dispatching, already existing calls take different paths, it's not mirrored in your code. But why methods - just regular procs do the same, you can add a better (more specific) overload match by an import and change logic of an alredy working code. I.e. you don't add any calls for it - just an import.
Concepts: their matching is changed by just adding symbols.
mixin: new symbols are taken into account, just their adding changes how the code works. Changes may be not less at all, than with per-module symbols, the latter just adds a friendly syntax.
Scarcely these are all the features that change program's logic just by addition of symbols. These all of coarse lessen code's evidence, for that increasing its expressiveness, and so far the latter is Nim's virtue.
So my "just adding symbols" does a lot more than I made it look like, thanks for pointing that out. I didn't think of the drastic changes in code behavior importing these existing language features cause, maybe because I'm just used to them. That being said, I still think that changing the existing code itself with meta-programming is a different thing than changing its behavior by importing entities which are generally defined on the same semantic level.
These all of coarse lessen code's evidence, for that increasing its expressiveness, and so far the latter is Nim's virtue.
Not the only virtue, though. The slogan is "Efficient and expressive programming." and the efficiency part has a lot to do with readability through "Keep it simple, keep it obvious". IMHO Nim is pretty good at this so far. Oh, and "Efficient" comes first :o)
I'll try. :) Sorry, it then becomes long.
Situation: one has macro, which takes a statement list, and a procedure, which body (contents) needs to be processed with that macro:
macro m ...
# it can be used so:
m:
echo "smth"
doSmthElse()
proc p =
... # We want to apply `m` to these statements
Instead of this
macro m ... # expects a statement list as the argument
proc p =
wrapwith m
...
one can principally do
proc m2 ... # rewritten to take a procedure as the argument
proc p {.m2.} =
...
. But if m is not created for this specific use (to process p), but imported from somewhere, or is used already for blocks of code (m: ...), then it cannot be just changed to take a proc instead. Even if it's newly created, making it to work with both
proc p {.m.} = ...
# and
m:
... # stmtList
is a complication.
One can instead write yet another macro, which takes p, gets its body and passes it to m, which is again an extra burden for user.
So, then, if wrapwith is allowed within inner levels, including inside procs, then one can just use an existing macro, which takes a statement list (i.e. is supposed to be used as m: stmt1; stmt2), or write just one macro for both cases, expecting just a statement list in it:
import theModuleWithThatMacro
m:
echo "a statement"
echo "yet another"
proc p = # {.m.} would not work, `m` doesn't expect a proc
wrapwith m
echo "m gets this statement"
echo "together with this, in a statement list"
So +1 for allowing in any statement list.
Ah! This is a good point. And I can't help but imagine that there are many macros in place now which operate conceptually on a statement list, but have been defined to work on a proc definition so that the common case is convenient.
This would encourage writing more general macros in the future.
I'm afraid of the language becoming too complicated. When not simply this
with m:
include part_to_be_indented
That is explicit and transparent. All we really need is "indented" include.
This would work, but I think it is a poor solution. I'm gonna use a similar example with different names to illustrate why:
# myDSL.nim
macro DSL(body: untyped): untyped = ...
# wrap.nim
import myDSL
DSL:
include body.nim
First, anyone reading or editing body.nim will be totally unaware that it is being affected by the DSL macro until they read wrap.nim.
Also, every usage of body.nim will need to be annotated with a DSL: wrap. If, say, DSL gets deprecated and its use needs to be removed from the project, each usage of body.nim needs to get changed.
Because DSL is affecting body.nim as a white box rather than a black box, its usage should be apparent from within body.nim.
First, anyone reading or editing body.nim will be totally unaware that it is being affected by the DSL macro until they read wrap.nim.
I'm not convinced that's true. Isn't that just how re-usable code works? You always have to look at the context of its use. In fact, the whole idea of wrapwith suffers the problem with transparency to a much greater degree. It's modifying the rest of the file, and it's not obvious to the reader because the rest of the file is not indented. I would find such code hard to understand unless I were already intimately aware of details of the language.
If you're worried that someone reading the other file would wonder how it might be used, rename it to something clear, e.g. body_to_be_included.nim.
Besides, your solution still allows precisely the situation you consider a problem:
wrapwith m
include body
Finally, I don't think wrapwith has been well-considered. E.g. what happens with multiple wrapwith statements in the same file? Is it obvious to the reader which direction they bind, upward or downward in the file? With indentation, it's obvious.
I think your basic idea has merit, but with indentation you can do it today, right? Aside from possible aesthetic objections, are there any fundamental problems with indented-include?
I'm not convinced that's true. Isn't that just how re-usable code works? You always have to look at the context of its use.
Let's say body.nim uses the recur macro outlined in my root post of this thread. It then relies on this macro/DSL and expects it; each invocation requires a call to implementRecur wrapping include body.nim for body.nim to work properly. It's not reusable code in the way you're thinking.
In fact, the whole idea of wrapwith suffers the problem with transparency to a much greater degree. It's modifying the rest of the file, and it's not obvious to the reader because the rest of the file is not indented.
This is a very valid criticism. My best response, as outlined, is that wrapwith would be required to be placed at the top of a file (or StmtList) so it's easily locatable. Programmers would know then to just check for any wrapwith statements if they see any funky business.
If you're worried that someone reading the other file would wonder how it might be used, rename it to something clear, e.g. body_to_be_included.nim.
This doesn't address the "real" issue here: body.nim relies on a certain macro and so should be "bundled with" it.
Besides, your solution still allows precisely the situation you consider a problem:
My solution solves the problem. Could it be ignored by programmers like you're showing? Well, ...yes. But any language feature can be abused.
# body.nim
wrapwith m
<code>
# a.nim
import body # works out-of-the-box
Finally, I don't think wrapwith has been well-considered. E.g. what happens with multiple wrapwith statements in the same file? Is it obvious to the reader which direction they bind, upward or downward in the file? With indentation, it's obvious.
A previous post of mine outlines how this would work.
I think your basic idea has merit, but with indentation you can do it today, right? Aside from possible aesthetic objections, are there any fundamental problems with indented-include?
Yes, with indentation, this is possible. The point is to make it much cleaner to use and stack these kinds of macros. This way, you can transparently define and use your own language features. Indenting an entire file is slightly more than just an aesthetic issue.