This is not a question, more like a blog post. I don't have a blog so I do it here. Feel free to comment or ask questions.
In my past I have learned the programming language Go, and I really started to like the interface type that Go offers, and I wondered, weather it is possible to port this go languge feature to Nim with the aid of macros. For those of you, who know interfaces only from Java, in go interfaces work a bit different and here is the official documentation for it: https://gobyexample.com/interfaces .
First of all, I would like to show you a bit of the advangae of this Go interface type, by comparing it the the options c++ can give you. The c++ version would be almost identical to a Java solution.
virtual methods
struct MyInterface {
virtual int foo() = 0;
virtual int bar() = 0;
virtual ~MyInterface(){}
};
struct MyImplementationA: public MyInterface {
int foo() override { return 17; }
int bar() override { return 4; }
};
struct MyImplementationB: public MyInterface {
int m_foo;
int m_bar;
int foo() override { return m_foo; }
int bar() override { return m_bar; }
};
// polymorphic usage
int foobar(MyInterface* mi) {
return mi->foo() + mi->bar();
}
#include <iostream>
using std::cout;
using std::endl;
int main() {
MyImplementationA a;
MyImplementationB b;
b.m_foo = 32;
b.m_bar = 8;
cout << foobar(&a) << endl;
cout << foobar(&b) << endl;
}
In this version the virtual method dispatch in foobar is done with the vtable. With the declaration of the first virtual method in MyInterface c++ adds a hidden field to the struct, the so called vtable pointer. that points to a struct with method pointers to the method implementations. Both MyImplementationA and MyImplementationB change the value of this pointer to the vtable of teir type. The method invocation foo in foobar opens the vtable, and reads from there which method to invoke. Dependent on the type it is either MyImplementationA::foo or MyImplementationB::foo. Because of this explicit inheritance the interface implementation needs to be newer than the interface itself. Also small objects that need to be compact in memory should better not have virtual methods because of this added hidden field. C# and Java work almost identical, just that in java everything is virtual.
templates
struct MyImplementationA {
int foo() { return 17; }
int bar() { return 4; }
};
struct MyImplementationB {
int m_foo;
int m_bar;
int foo() { return m_foo; }
int bar() { return m_bar; }
};
// polymorphic usage
template <typename T>
int foobar(T* mi) {
return mi->foo() + mi->bar();
}
#include <iostream>
using std::cout;
using std::endl;
int main() {
MyImplementationA a;
MyImplementationB b;
b.m_foo = 32;
b.m_bar = 8;
cout << foobar(&a) << endl;
cout << foobar(&b) << endl;
}
This version beats all problems of the previous version, but not without introducing new problems. There is no hidden field in neither MyImplementationA nor MyImplementationB. foobar is compiled for each type it is called separately. Meaning that the compiler can also inline the the content from foo and bar. Templates in c++ have a huge problem of being recompiled not only for each type, but also for each compilation unit. So depending on the complexity of foobar and the amount of compilation units where foobar is used, this version can really explode compilation time in bigger projects. Nim generics are very much like c++ templates, so I guess they can have this flaw as well. I just don't have big enough projects yet to notice any major concerns with Nim's generics, but that doesn't prove that they do not exist.
Go Interface
package main
import "fmt"
type MyImplementationA struct {}
func (this *MyImplementationA) foo() int {
return 17;
}
func (this *MyImplementationA) bar() int {
return 4;
}
type MyImplementationB struct {
m_foo,m_bar int
}
type MyInterface interface {
foo() int
bar() int
}
func (this *MyImplementationB) foo() int {
return this.m_foo;
}
func (this *MyImplementationB) bar() int {
return this.m_bar;
}
// polymorphic usage
func foobar(mi MyInterface) int {
return mi.foo() + mi.bar();
}
func main() {
a := MyImplementationA{}
b := MyImplementationB{32, 8}
fmt.Println(foobar(&a))
fmt.Println(foobar(&b))
}
The Go version does neither put a hidden field in the struct datatype or mark anything as virtual. In fact virtual is not a keyword in go nor does it have an equivalent. Methods just become virtual, as soon as the type gets casted into an interface type.
This works, because the vtable is not part of the struct. An instance of MyInterface is actually a tuple of two pointers. The first one is the vtable pointer, and the second one is the pointer to the struct that implements the interface. With this design decision the Go compiler can just create a vtable for any type that has the methods required to implement it. And when the method foobar grows over time, and get's used by a lot of different types, it still get's compiled only once. If you remember one of Go's main features is it's compilation speed.
Now I want the same concept of the Go interfaces in Nim. The idea is to do the same thing in two steps:
- build an example case with everything, including vtable and iterface written manually.
- write macros that create the exact same vtable and interface that were manually created in step 1
To not make this post too long, I have joined thise two steps into one step. A macro call is in this code followed by a multiline comment that exactly represents the generated source code
import interfacemacros
type
MyImplementationA = object
data : array[0,byte]
MyImplementationB = object
foo : int
bar : int
createInterface(MyInterface):
proc foo(this : MyInterface) : int
proc bar(this : MyInterface) : int
#[
type
MyInterfaceVtable = object
foo: proc (this: pointer): int
bar: proc (this: pointer): int
MyInterface = object
objet: pointer
vtable: ptr MyInterfaceVtable
proc foo(this: MyInterface): int =
this.vtable.foo(this.objet)
proc bar(this: MyInterface): int =
this.vtable.bar(this.objet)
]#
proc foo(this : ptr MyImplementationA) : int = 17
proc bar(this : ptr MyImplementationA) : int = 4
implementInterface(MyInterface, MyImplementationB)
#[
let MyImplementationBVtable = MyInterfaceVtable(
foo: proc (this: pointer): int = cast[ptr MyImplementationB](this).foo,
bar: proc (this: pointer): int = cast[ptr MyImplementationB](this).bar
)
proc MyInterface_InterfaceCast(this: ptr MyImplementationB): MyInterface =
MyInterface(objet: this, vtable: MyImplementationBVtable.unsafeAddr)
]#
implementInterface(MyInterface, MyImplementationA)
#[
let MyImplementationAVtable = MyInterfaceVtable(
foo: proc (this: pointer): int = cast[ptr MyImplementationA](this).foo,
bar: proc (this: pointer): int = cast[ptr MyImplementationA](this).bar
)
proc MyInterface_InterfaceCast(this: ptr MyImplementationA): MyInterface =
MyInterface(objet: this, vtable: MyImplementationAVtable.unsafeAddr)
]#
proc foobar(mi: MyInterface) : int =
return mi.foo + mi.bar
var
a : MyImplementationA
b : MyImplementationB = MyImplementationB(foo : 32, bar : 8)
echo foobar(MyInterface_InterfaceCast(a.addr))
echo foobar(MyInterface_InterfaceCast(b.addr))
The differences two the Go version are:
- The interface declaration is in a macro call.
- The vtables for the different types are requested explicitly with implementInterface(MyInterface, MyImplementation)
- The conversion between pointer type and interface type is explicit with MyInterface_InterfaceCast
- error messages when the interface can not be implemented are not really nice
One thing to note is, I could change MyInterface_InterfaceCast from a proc to a converter, and maybe I should to make this even more go like. I just don't like implicit conversions, I think they are scary, because they hide procedure calls from the code reader, and could therefore decrease code maintainability.
And for completion, here are the macros that do the magic:
import macros
macro createInterface*(name : untyped, methods : untyped) : untyped =
name.expectKind nnkIdent
let
vtableRecordList = nnkRecList.newTree
vtableIdent = newIdentNode($name.ident & "Vtable")
vtableTypeDef = nnkTypeSection.newTree(
nnkTypeDef.newTree(
vtableIdent,
newEmptyNode(),
nnkObjectTy.newTree(
newEmptyNode(),
newEmptyNode(),
vtableRecordList
)
)
)
var newMethods = newSeq[NimNode]()
for meth in methods:
meth.expectKind(nnkProcDef)
let
methodIdent = meth[0]
params = meth[3]
thisParam = params[1]
thisIdent = thisParam[0]
thisType = thisParam[1]
if thisType != name:
error thisType.repr & " != " & name.repr
let vtableEntryParams = params.copy
vtableEntryParams[1][1] = newIdentNode("pointer")
vtableRecordList.add(
nnkIdentDefs.newTree(
methodIdent,
nnkProcTy.newTree(
vtableEntryParams,
newEmptyNode(),
),
newEmptyNode()
)
)
let call = nnkCall.newTree(
nnkDotExpr.newTree( nnkDotExpr.newTree(thisIdent, newIdentNode("vtable")), methodIdent ),
nnkDotExpr.newTree( thisIdent, newIdentNode("objet") ),
)
for i in 2 ..< len(params):
let param = params[i]
param.expectKind(nnkIdentDefs)
for j in 0 .. len(param) - 3:
call.add param[j]
meth[6] = nnkStmtList.newTree(call)
newMethods.add(meth)
result = newStmtList()
result.add(vtableTypeDef)
result.add quote do:
type `name` = object
objet : pointer
vtable: ptr `vtableIdent`
for meth in newMethods:
result.add meth
#echo result.repr
macro implementInterface*(interfaceName, implementationCandidateName: typed) : untyped =
let
vtableSymbol = interfaceName.symbol.getImpl[2][2][1][1][0]
vtableRecordList = vtableSymbol.symbol.getImpl[2][2]
let vtableValueSymbol = newIdentNode($implementationCandidateName.symbol & "Vtable")
let
objectConstructor = nnkObjConstr.newTree(vtableSymbol)
vtableValueDecalaration =
nnkLetSection.newTree(
nnkIdentDefs.newTree(
vtableValueSymbol,
newEmptyNode(),
objectConstructor
)
)
for identDefs in vtableRecordList:
let
methodName = identDefs[0]
params = identDefs[1][0]
lambdaBody = quote do:
cast[ptr `implementationCandidateName`](this).`methodName`()
call = lambdaBody[0]
for i in 2 ..< len(params):
let param = params[i]
param.expectKind(nnkIdentDefs)
for j in 0 .. len(param) - 3:
call.add param[j]
# leave out () when not needed
if call.len == 1:
lambdaBody[0] = call[0]
methodName.expectKind nnkIdent
objectConstructor.add nnkExprColonExpr.newTree(
methodName,
nnkLambda.newTree(
newEmptyNode(),newEmptyNode(),newEmptyNode(),
params.copy,
newEmptyNode(),newEmptyNode(),
lambdaBody
)
)
result = newStmtList()
result.add vtableValueDecalaration
let castIdent = newIdentNode($interfaceName.symbol & "_InterfaceCast")
result.add quote do:
proc `castIdent`(this: ptr `implementationCandidateName`) : `interfaceName` = `interfaceName`(
objet : this,
vtable : `vtableValueSymbol`.unsafeAddr
)
# echo result.repr
conclusion: Nim turns out to be a very powerful programming language.
This version here not only gives you a version you can clone right away from github, but also introduces some improvements that will not be inserted into this version anymore.
Nice work, but please refrain from facts/myths that don't have anything to do with your otherwise very nice article.
This is so bad that several bigger c++ projects completely abandoned templates.
Which ones exactly? LLVM, wxWidgets, Urho3D, Ogre, Unreal Engine 4 and even GCC afaik all use templates. We're not in the 80ies anymore where templates where this special bug ridden feature with stupid syntax that nobody understood.
If you remember one of Go's main features is it's compilation speed.
Go's compile times are nothing special at all. Nim sometimes beats Go's compile times, Delphi always beats it, D can beat it and yet D too has extensive templating everywhere...
Which ones exactly? LLVM, wxWidgets, Urho3D, Ogre, Unreal Engine 4 and even GCC afaik all use templates. We're not in the 80ies anymore where templates where this special bug ridden feature with stupid syntax that nobody understood.
Ok, I will search for examples, and update the post accordingly. I definitively heared it in talks about c++. And listing big projects that use templates does not beat the argument. Additionally I've heard from Unreal Engine 4 developers, that changing the source code has very long builds, even long incremental builds maybe because of templates, but that not a prove and not my own experience.
Go's compile times are nothing special at all. Nim sometimes beats Go's compile times, Delphi always beats it, D can beat it and yet D too has extensive templating everywhere...
Ok I did not do the benchmark, but correct would be that Go advertises with fast compilation speed. But you are right, in my experience it was not too fast. In my code is was always slower than advertised :/. But I can't exclude that I did something wrong that cause this increased compilation speed, because I never programmed in pure Go, I always had wrappers.
@bpr I never said that monomorphization is a harmful language feature. I just say that someone needs to be aware of it, and that in some contexts this monoporphization is in fact a bad thing.
But in c++ it actually is harmful, because the way c++ deals with templates is, to create all required instances per compilation unit, and then at link time the linker throws all duplicates away. And that I consider as harmful, because the same function might get compiled 100 times and then gets thrown away 99 times. But the problem here is more on the way how c++ compiles its sourcecode, and not on how templates generally work. c++ has the same problem with datatypes that are declared in headers.
surprisingly, this Go like interface is very similar with what I have implemented when creating high level wrapper for Chromium Embedded Framework in nimCEF.
in my experience, Nim macro not only capable simulate Go like interface, but also interface + simulated inheritance, something not available in Go. Indeed Nim's macro can make Nim a very powerful language.
How about:
type
MyImplementationA = object
MyImplementationB = object
m_foo,m_bar: int
proc foo(this: MyImplementationA): int =
return 17
proc bar(this: MyImplementationA): int =
return 4
proc foo(this: MyImplementationB): int =
return this.m_foo
proc bar(this: MyImplementationB): int =
return this.m_bar
proc foobar[T](mi: T): int =
return mi.foo() + mi.bar()
var a = MyImplementationA()
var b = MyImplementationB(m_foo: 32, m_bar: 8)
echo foobar(a)
echo foobar(b)
Interfaces as generated by your macro can in fact be useful, but I do not think that they have necessarily to match Go semantics 1 to 1 - in particular, while this kind of interfaces do solve an actual problem, it seems to me that type switch does not offer much.
If you are in a situation where you can enumerate the cases upfront (which is needed to use the type switch), you can easily find an alternative way to express this
What's a pity interface is a keyword so it can't be used for declaring interfaces.
Interfaces can become particularly handy if used as a result type, it seems to me. Either way, it would be great to provide interface inheritance:
createInterface HealthGenerator:
proc genHealth(g: HealthGenerator): Health
createInterface EnergyGenerator:
proc genEnergy(g: EnergyGenerator): Energy
createInterface BeautyGenerator of HealthGenerator, EnergyGenerator:
proc genBeauty(g: BeautyGenerator): Beauty
type
Pear = ...
proc genHealth(g: Pear): Health = ...
proc genEnergy(g: Pear): Energy = ...
proc genBeauty(g: Pear): Beauty = ...
let pear = Pear(...)
let beautifier: BeautyGenerator = pear # vtable's pointer points at p
echo beautifier.genEnergy()
echo beautifier.genHealth()
echo beautifier.genBeauty()
let energiser: EnergyGenerator = beautifier # new vtable's pointer still points at p
echo energiser.genEnergy()
#echo energiser.genHealth() # but has other procedure pointers
#let other: BeautyGenerator = energiser # so the information is lost
A simple compile-time procedure returning bool value is all the converter needs.
A runtime time label for interfaces sounds like an interesting idea but let's see what happens when we provide them:
let
pear = Pear(...)
beautifier: BeautyGenerator = pear
energiser: EnergyGenerator = beautifier
other: BeautyGenerator = energiser.getType()
if beautifier.realType of Pear:
pearLover.eat(cast[ptr Pear](beautifier.obj)[])
proc burn(g: EnergyGenerator): Energy =
result = g.genEnergy()
if g.getType of SmokeGenerator:
toSmokeGenerator(g).genSmoke()
Which is why it sounds more reasonable to either not provide any type label at all or provide it only for explicit type casts, NOT conversions to other interfaces, which isn't obvious and may be confusing for some users if type-switch or similar transparent real-type-getters will be available.
My use cases would be virtually identical to the vtable usage of the standard library for things like streams. I just find that syntax extremely annoying and would love to have it abstracted away without having additional runtime costs.
In my perfect world this would be an addition to concepts, so that they can either make the function implictely generic as they do now or do dynamic dispatch via implicitely generated vtable.
With that said I also would prefer the pointers to be dereferenced before the proc is called. That way keeping it alive out of scope would be save and you still could use a var param to get it mutably. That way interfaces would also mirror the static duck typing awesomeness and clean syntax you generally have in nim while also having the advantages of dynamic dispatch where necessary.
Finally, I do agree that type reflection on interfaces doesn't really seem necessary. The tight coupling seems to remove the whole purpose the interface had in the first place. An if you aren't interested in that there are, like, three other ways to do the same thing with less overhead and ugly hacks. Any use case that couldn't be solved with variant types, methods or generics more cleanly?
@Udiknedormin
What you call inheritance I would, if I do implement it, rather do as embedding.
createInterface BeautyGenerator:
HealthGenerator
EnergyGenerator
proc genBeauty(g: BeautyGenerator): Beauty
You can se Go type embedding for example.This would create implicit conversions to HealthGenerator and EnergyGenerator. I think it's possible with not too much effort, which doesn't mean that I will implement it now.
A minor minor variant that works with var parameters instead of ptr:
import macros
macro implementInterface(interfaceName: typed) : untyped =
let
interfaceNameStr = $interfaceName.symbol
vtableSymbol = interfaceName.symbol.getImpl[2][2][1][1][0]
vtableRecordList = vtableSymbol.symbol.getImpl[2][2]
let
objectConstructor = nnkObjConstr.newTree(vtableSymbol)
for identDefs in vtableRecordList:
let
methodName = identDefs[0]
params = identDefs[1][0]
lambdaBody = quote do:
cast[var T](this).`methodName`()
call = lambdaBody[0]
for i in 2 ..< len(params):
let param = params[i]
param.expectKind(nnkIdentDefs)
for j in 0 .. len(param) - 3:
call.add param[j]
# leave out () when not needed
if call.len == 1:
lambdaBody[0] = call[0]
methodName.expectKind nnkIdent
objectConstructor.add nnkExprColonExpr.newTree(
methodName,
nnkLambda.newTree(
newEmptyNode(),newEmptyNode(),newEmptyNode(),
params.copy,
newEmptyNode(),newEmptyNode(),
lambdaBody
)
)
let
getVtableReturnStatement =
nnkReturnStmt.newTree(newCall("addr", newIdentNode("theVtable")))
globalVtableIdent = newIdentNode("theVtable")
getVtableProcIdent = newIdentNode("get" & interfaceNameStr & "Vtable")
getVtableProcDeclaration = quote do:
proc `getVtableProcIdent`[T](): ptr BananasVtable =
var `globalVtableIdent` {.global.} = `objectConstructor`
`getVtableReturnStatement`
result = newStmtList()
result.add getVtableProcDeclaration
let castIdent = newIdentNode("to" & $interfaceName.symbol)
result.add quote do:
converter `castIdent`[T](this: ptr T) : `interfaceName` = `interfaceName`(
objet : this,
vtable : `getVtableProcIdent`[T]()
)
result.add quote do:
converter `castIdent`[T](this: var T) : `interfaceName` = `interfaceName`(
objet : this.addr,
vtable : `getVtableProcIdent`[T]()
)
result.add quote do:
converter `castIdent`(this: `interfaceName`): `interfaceName` = this
when defined(interfacedebug):
echo result.repr
macro createInterface*(name : untyped, methods : untyped) : untyped =
name.expectKind nnkIdent
let
nameStr = $name.ident
vtableRecordList = nnkRecList.newTree
vtableIdent = newIdentNode(nameStr & "Vtable")
vtableTypeDef = nnkTypeSection.newTree(
nnkTypeDef.newTree(
vtableIdent,
newEmptyNode(),
nnkObjectTy.newTree(
newEmptyNode(),
newEmptyNode(),
vtableRecordList
)
)
)
var newMethods = newSeq[NimNode]()
for meth in methods:
meth.expectKind(nnkProcDef)
let
methodIdent = meth[0]
params = meth[3]
thisParam = params[1]
thisIdent = thisParam[0]
thisType = thisParam[1]
if thisType != name:
error thisType.repr & " != " & name.repr
let vtableEntryParams = params.copy
vtableEntryParams[1][1] = newIdentNode("pointer")
vtableRecordList.add(
nnkIdentDefs.newTree(
methodIdent,
nnkProcTy.newTree(
vtableEntryParams,
newEmptyNode(),
),
newEmptyNode()
)
)
let call = nnkCall.newTree(
nnkDotExpr.newTree( nnkDotExpr.newTree(thisIdent, newIdentNode("vtable")), methodIdent ),
nnkDotExpr.newTree( thisIdent, newIdentNode("objet") ),
)
for i in 2 ..< len(params):
let param = params[i]
param.expectKind(nnkIdentDefs)
for j in 0 .. len(param) - 3:
call.add param[j]
meth[6] = nnkStmtList.newTree(call)
newMethods.add(meth)
result = newStmtList()
result.add(vtableTypeDef)
result.add quote do:
type `name` = object
objet : pointer
vtable: ptr `vtableIdent`
for meth in newMethods:
result.add meth
result.add newCall(bindSym"implementInterface", name)
when defined(interfacedebug):
echo result.repr
Usage:
import interfacemacros
type
MyBananas = object
a,b : int
OtherBananas = object
foo,bar: int
proc foo(banan : var MyBananas) : int =
banan.a + banan.b
proc bar(banan : var MyBananas) : int =
banan.a * banan.b
proc baz(banan : var Mybananas, a,b : int, c : float) : float =
echo "baz MyBananas"
float((a+banan.a) * (b+banan.b)) / c
proc baz(banan : var OtherBananas, a,b : int, c : float) : float =
float((a+banan.foo) * (b+banan.bar)) / c
createInterface(Bananas):
proc foo(this : Bananas) : int
proc bar(this : Bananas) : int
proc baz(this : Bananas, a,b : int, c : float) : float
proc foobar(mbi: Bananas): int =
mbi.foo + mbi.bar
proc bazinga(args : varargs[Bananas, toBananas]): int =
echo "got ", args.len, " args"
for arg in args:
result += arg.foo * arg.bar
echo "baz in bazinga ", arg.baz(7,8,3.14)
echo "baz in bazinga 1 ", arg.baz(7,8,21)
proc main() =
var
banan = MyBananas(a: 17, b: 4)
otherbanan = OtherBananas(foo: 17, bar: 4)
echo "hallo welt!"
echo "foobar1: ", foobar(banan)
echo "foobar2: ", foobar(otherbanan)
echo "bazinga: ", bazinga(banan, otherbanan)
main()
@Krux02: Since I get that you are not interested in publishing this, would you mind if I did?
Still need to improve tests, add to the package repository, make a better README and so on - I will do that in the next few days
This thread is a bit dusty, but I just want to mention, that I made some improvements to andrea's implementation, in case somebody is still interested in interfaces like this, while vtrefs are still unimplemented.
Mainly I fixed it(previously the generated AST was invalid) and added a way to export interfaces. I've already made a pull request.