Hello,
I'm working on some serialization / deserialization using msgpack and I was wondering if there was a way to use the macros system to iterate over field.
To illustrate here's what i'm ttrying to do :
type
Model = object
field1: string
field2: string
field3: int
FooObj = object
field1: Tensor[float]
field2: Table[string, float]
field3: int
proc newFooObj(model: Model): FooObj=
var field1: Tensor[float]
unpack(model.field1, field1)
var field2: Table[string, float]
unpack(model.field2, field2)
var field3: int
unpack(model.field3, field3)
result = FooOBj(field1: field1, field2: field2, field3: field3)
I have a dozen of type (for each serailized / deserialzed type) and each type has lots of fields so writing by hand is doable by very tedious.
I'm not familiar with the macro system but I was wondering if there was a way to iterate over the fields of an object with or without a bracket type (I've used dumpTree and noticed a difference) and apply so I could simply pass 2 type with identical field name and generate a procedure:
type
Model = object
field1: string
field2: string
field3: int
FooObj = object
field1: Tensor[float]
field2: Table[string, float]
field3: int
magicalProcGeneratingMacro[Model, FooObj](procName)
I'm not experienced enough with Nim's macro system to see the solution (if there is any) on my own or if I should just give up and do it by hand.
There are iterators fields and fieldPairs:
I thought fields could only iterate over element of the same type. If not, how do you use fields to iterate over fields of 2 different type ?
type
FooObj = object
field1: Tensor[float]
field2: Table[string, float]
field3: int
Model = object
field1: string
field2: string
field3: int
var model: Model
var foo: FooObj
for name, t, m in fieldPairs(model, foo):
echo name, t, $m
I get an error type mismatch: got <FooObj> but expected 'Model = object
I have never used that iterators myself.
But I would strongly assume that we can only iterate over one object at a time.
As we can only iterate over one string or over one sequence at a time.
But of course we can collect the results of each iteration is some way, and finally output all collected data in the desired format.
Field fieldPairs are great!
I use it to print arbitrary structures and deserialize json in a more looser way:
https://github.com/treeform/print/blob/master/src/print.nim#L79
https://github.com/treeform/jsutils/blob/master/src/jsutils/jsons.nim#L28
If all of your procs are going to look like newFooBar above there, it's possible to generate with a macro.
import macros, tables
type
Tensor[T] = object
discard
Model = object
field1: string
field2: string
field3: int
FooObj = object
field1: Tensor[float]
field2: Table[string, float]
field3: int
proc unpack[T; U](arg: T, to: var U) = discard
proc newVar(name, dtype: NimNode): NimNode =
result = nnkVarSection.newTree(
nnkIdentDefs.newTree(
ident(name.toStrLit.strVal), # replace by new ident
dtype,
newEmptyNode()
)
)
echo result.repr
macro genNewObjProc(obj, model: typed): untyped =
let objFields = obj.getType[1].getTypeImpl[2] # get recList of type
let modelFields = obj.getType[1].getTypeImpl[2] # get recList of type
doAssert objFields.len == modelFields.len
var body = newStmtList()
let modelIdent = ident"model"
# variable to hold object constructor `FooBar(field1: field1,...)`
var objConstr = nnkObjConstr.newTree(obj)
for i in 0 ..< objFields.len:
let modelName = ident(modelFields[i][0].toStrLit.strVal) # replace by new ident
doAssert eqIdent(objFields[i][0], modelName)
let objType = objFields[i][1]
body.add newVar(modelName, objType)
body.add quote do:
unpack(`modelIdent`.`modelName`, `modelName`)
# add to object constructor
objConstr.add nnkExprColonExpr.newTree(modelName, modelName)
# add resulting `FooObj` call
let resIdent = ident"result"
body.add quote do:
`resIdent` = `objConstr`
let procParams = [obj, # return type
nnkIdentDefs.newTree(modelIdent,
model,
newEmptyNode())]
result = newProc(name = ident("new" & obj.toStrLit.strVal),
params = procParams,
body = body)
echo result.repr
genNewObjProc(FooObj, Model)
I'm not sure how helpful it is to get such a macro if one isn't familiar with macros. But since it's fun to write I might as well give you a solution. :)
If all of your procs are going to look like newFooBar above there, it's possible to generate with a macro.
Yes, it's exactly what I'm looking for. A way to generate several procedure that follow a pattern.
I've been trying my hand at writing macros that generate proc but there is a lot to unpack regarding the Nim macro system, it's not as intuitive as the rest of the language :).
I'll look into your solution since I may need to adapt a few things (I've simplified the real uses cases to summarize it into a single problems).
Thanks everyone for the help !
I'll look into your solution since I may need to adapt a few things (I've simplified the real uses cases to summarize it into a single problems). The goal is also to learn Nim's macro as well. I've now spent probably as much time on macros than it would have took to write the solution it by hand, but it's not as fun.
If you have questions about my code there or general macro questions, just ask. I'll try to help!
I didn't about the quote do tricks, that's useful !
Why do you have to use backtick "`" inside quote do ? Is it to interpret macro inside a block of code ?
First of all see of course the docs here:
https://nim-lang.github.io/Nim/macros.html#quote%2Ctyped%2Cstring
and the macro tutorial, specifically:
https://nim-lang.github.io/Nim/tut3.html#introduction-generating-code
So the basic idea is that quote do allows you to write exactly the code you want to generate. However, in most cases that's not really very helpful, because if you can explicitly write your code, you could also just write a template / proc. That's where the back ticks come in to perform actual quoting of NimNodes defined in the current scope. They will be inserted in those places.
quote do is thus just a nice way to avoid having to build the AST manually (as I for instance do in the newVar proc), but keep the ability to insert NimNodes you calculate / determine somehow based on what the macro is supposed to accomplish.
Another thing to keep in mind when using quote do is about the stuff that's not quoted with back ticks. As a rule of thumb (someone please correct me):
The second means that if you want to refer to some variable that will be known in the scope in which the macro is used, you have to create the identifier manually and quote it. Due to the first point you fortunately don't have to do the same for procedures you want to use.
I think i understand most of it. Now, I just need to practice macros more so I can come up with the solution my self next time :).
Thanks again