Hi, I'm creating protobuf for Nim (both protocompiler and runtime library)
I'm trying to do as little as possible in protoc of the code generation, and do most of it by using Nims metaprogramming power.
Unfortunately, I want to reuse parts of the AST but I don't know how. See below work in progress what protoc currently produces:
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: Msg.proto
import protobuf
type
ItemRow* = object
id*: int32
quantity*: int32
cost*: int32
proc WriteTo*(message: ItemRow, output: var CodedOutputStream) =
PbWriteTo(message, output):
id: PbInt32 @1
quantity: PbInt32 @2
cost: PbInt32 @4
proc MergeFrom*(message: var ItemRow, input: CodedInputStream) =
PbMergeFrom(message, input):
id: PbInt32 @1
quantity: PbInt32 @2
cost: PbInt32 @4
As you can see both procs call a macro, PbWriteTo() or PbMergeFrom(), which generate the actual implementation to write or read binary streams. The codeblock passed to these macro's is the same, so I would like to do something like:
PbMessage ItemRow:
id: PbInt32 @1
quantity: PbInt32 @2
cost: PbInt32 @4
proc WriteTo*(message: ItemRow, output: var CodedOutputStream) =
PbWriteTo(message, output)
proc MergeFrom*(message: var ItemRow, input: CodedInputStream) =
PbMergeFrom(message, input)
I know how to generate the actual type in the PbMessage() macro. (the Nim in Action book a great resource!) But the actual type does not store the field numbers. How can I temporarily save the AST (or codeblock) that is given to PbMessage() macro and reuse it in the PbWriteTo() and PbMergeFrom() macros?
I thought about generating the procs inside the PbMessage() macro, but then the generated code will be very cryptic to end users of protobuf-nim, and they won't be able to see the function prototypes of the WriteTo() and MergeFrom() procs.
@xomachine You're a boss
I put
static:
var PbFieldInfo: NimNode
in the file, filled the var in one macro with it's passed code block, and
echo treeRepr(PbFieldInfo)
in the other macro indeed prints out the AST of the codeblock that was given to the first macro. Amazing that we can wield such power at compile time haha :)
Thanks for this insight, I completely forgot about the static keyword.
@andrea Yeah will do. It will take some time though, I'm mostly only working on it in my weekends. Progress can be seen on my GitHub https://github.com/ivankoster/protobuf-nim
I'm sorry I cannot help with the code, but I just wanted to say that I'm very happy to hear about this effort, thank you very much! :D
Some semi-related food for thought:
I'd like to learn what other people think about that idea.
How would you guys model the one_of feature described here in nim?
I was thinking along these lines: Given the proto file:
message ConformanceRequest {
oneof payload {
bytes protobuf_payload = 1;
string json_payload = 2;
}
WireFormat requested_output_format = 3;
}
Compile it to the following nim file:
PbMessage ConformanceRequest*:
PbOneOf payload:
protobuf_payload: PbBytes @1
json_payload: PbString @2
requested_output_format: WireFormat @3
And then let the macros then expand to something like:
type
ConformanceRequest_payload_enum = enum
not_set = 0,
protobuf_payload,
json_payload
type
ConformanceRequest_payload_inner {.union.} = object
protobuf_payload: seq[uint8]
json_payload: string
type
ConformanceRequest_payload = object
kind: ConformanceRequest_payload_enum
inner: ConformanceRequest_payload_inner
type
ConformanceRequest = object
payload: ConformanceRequest_payload
requested_output_format: WireFormat
# some `.` overload on ConformanceRequest_payload to set the kind according to which payload you set
# not sure if `.` overload can see if it is used as getter or setter
And then use it like:
var cr : ConformanceRequest
assert(cr.payload.kind == not_set)
cr.payload.json_payload = "Hello"
assert(cr.payload.kind == json_payload)
I'm not really sure of the usage of the Kind enum is idiomatic nim style and also the manual says the union pragma doesn't support garbage collected values, so I guess the seq and string are off limits?
The C++ implementation generates a bunch of has_xxx() and set_xxx() functions, but I think that is pretty ugly, nim should do better right?