Hey guys, I'm using an ECS(-ish) setup for my game engine. Currently I have "entities" which are integers, "components" which are plain old data types, and "systems" which are modules containing callback functions.
What I want to do is refactor my game engine so each system module can be compiled into its own DLL, and be loaded dynamically at runtime. On first think it seems like I just need to figure out how to expose the callback functions in the DLL to the rest of the engine, and use shared engine resources in the DLL.
A typical system module:
systemOf kinematics: (DynamicBody, Transform)
proc onEngage(entity: Entity) {.kinematics.} =
# Do stuff when a new entity with DynamicBody and Transform components spawn
proc onUpdate(context: Context) {.kinematics.} =
var id: Entity
if engines.physics.checkUpdate(id):
# Synchronize with the shared physics engine
proc onMutate(entity: Entity, transform: Transform) {.kinematics.} =
# Update DynamicBody based on Transform updates
proc onMutate(entity: Entity, transform: DynamicBody) {.kinematics.} = # <-- overload
# Update Transform based on DynamicBody updates
What I plan to do is somehow generate an exported function in every system that returns a struct of function pointers that point to the functions in the module:
type SystemCallbacks = object
onEngage: proc(entity: Entity)
onUpdate: proc(context: Context)
onMutate: array[2, pointer] # huh?
proc kinematicsCallbacks*(): SystemCallbacks {.stdcall exportc dynlib.} =
# return a struct of function pointers
As you can see from the above, I am unsure about how to deal with function overloads, short of completely getting rid of them.
Currently I generate integer IDs for component types so I can do stuff with them at runtime:
type TypeId* = uint16
proc getId(): TypeId {.noinit.} =
var id {.global.} : TypeId = 0
result = id
id += 1
proc typeId*(T: type): TypeId {.noinit.} =
# generates a unique ID per instantation of the implicit generic
# source: https://stackoverflow.com/a/2486090
let id {.global.} = getId()
result = id
So I thought maybe I can use a sparse array indexed by TypeId to resolve overloads:
type SystemCallbacks = object
...
onMutate: array[0..TypeId.high, pointer] # hmm
But therein lies the problem: how do I generate a consistent TypeId that can be used in multiple DLLs?
A solution I thought of is to create a separate module where I register component types:
# componentregistrar.nim
import "./components/dynamicbody"
import "./components/transform"
# creates exported functions that return `TypeId`
register(DynamicBody)
register(Transform)
and compile that into yet another DLL, and have the engine and system DLLs use the IDs from that DLL. It's an extra step though, and adds some complexity. Is there a better way?
You can easily use overloads and .exportc: "uniqueNameHere".
What I want to do is refactor my game engine so each system module can be compiled into its own DLL, and be loaded dynamically at runtime.
But why? If you have a game engine, the next step is to write a game with it. Then you can decide during development if you really need hot code reloading and the complexity that enables it.
I see. So you're saying that there should be a way to generate functions using exportc in such a way that they can be loaded from the DLL according to parameter type? Maybe some sort of custom name mangling using the type name?
Yes, that's a good point. I actually have a very crappy game (more like a demo) going, but I'm finding it difficult to iterate and debug rapidly on it using the old edit-compile-run flow, which is why I figured I should start thinking about hot code reloading.
In my (toy) ecs i hash the typename as a component identifier, maybe a similar approach can also work for you?
https://github.com/enthus1ast/ecs/blob/341704a86b73fb14ce4f408e74a959c0e9f5f585/src/ecs.nim#L24
Thanks guys. @enthus1ast ya I was considering that, although the downside is I would need two different ways to generate type identifiers, and a way to translate between the two.
@araq currently my ECS is actually storing the callbacks in a bunch of seqs corresponding to the types of their parameters; so you're right, all I would need to do is make the memory of these seqs shared. I still need to be able to index into the seqs by some "type ID" though, so the DLLs and the main application need to agree on how the "type ID" is generated.
I just noticed that my current type ID generator module (code below) is compiled into a cpp file containing all generic instantiations of the typeId proc, and the mangling seems to be consistent between compiles:
# typeid.nim
type TypeId* = uint16
proc getId(): TypeId {.noinit.} =
var id {.global.} : TypeId = 0
result = id
id += 1
proc typeId*(T: type): TypeId {.noinit.} =
# generates a unique ID per instantation of the implicit generic
# source: https://stackoverflow.com/a/2486090
let id {.global.} = getId()
result = id
gets compiled to this:
// typeid.cpp
...
N_LIB_PRIVATE NU16 id__someApparentlyConsistentId;
...
N_LIB_PRIVATE N_NIMCALL(NU16, getId__someOtherApparentlyConsistentId)(void) {
NU16 result;
result = id__someApparentlyConsistentId;
id__someApparentlyConsistentId += 1;
return result;
}
...
N_LIB_PRIVATE N_NIMCALL(NU16, typeId__yetAnotherApparentlyConsistentId)(void) {
NU16 result;
result = id__someApparentlyConsistentId;
return result;
}
...
That leads me to believe that a typeid.cpp generated for different DLLs would contain generated functions with the same names if the same types are passed to proc typeId*(T: type): TypeId in the DLL source. Is this an accurate observation? Where in the compiler can I see how the cpp function names are generated?
Is this an accurate observation?
I think so.
Where in the compiler can I see how the cpp function names are generated?
ccgtypes.nim, proc mangleName(m: BModule; s: PSym): Rope
Thanks @Araq for your help. I found the signatureHash macro by investigating how symbol names are generated, and tried using it to create stable hashes of types across my DLLs, but for some reason it would sometimes generate a different hash for the same type used in different DLLs. I saw this PR which might help, but I'm unsure if it does what I think it does: https://github.com/nim-lang/Nim/pull/13305.
For now I ended up going with something similar to @enthus1ast's suggestion. Below is the PoC implementation. It generates sequential IDs based on a string hash of a type. It can be used across multiple DLL boundaries by assigning memory to typeRegistry.hashes allocated by the main executable.
import std/macros
type
TypeId* = uint16
TypeRegistry* = object
hashes: ptr[TypeHashArray]
TypeHashArray = array[TypeId.high, TypeHash]
TypeHash = array[32, char]
proc init*(self: var TypeRegistry) {.ctor.} =
self.hashes = createShared TypeHashArray
proc deinit*(self: var TypeRegistry) =
if self.hashes != nil:
dealloc self.hashes
self.hashes = nil
var typeRegistry*: TypeRegistry
proc setTypeRegistry*(registry: TypeRegistry) {.dllapi.} =
typeRegistry = registry
proc getTypeHash(hash: string): TypeHash {.compileTime.} =
for i, element in enumerate hash:
result[i] = element
macro typeHash(Type: type): TypeHash =
# TODO: Make this more reliable (`signatureHash` fails sometimes)
# Wait for https://github.com/nim-lang/Nim/pull/13305
newCall(bindSym"getTypeHash", newLit $Type.getTypeImpl[1])
proc getId(typeHash: static[TypeHash]): TypeId {.noinit.} =
var id: TypeId
while (let hash = typeRegistry.hashes[id] ; hash[0] != '\0'):
if hash == typeHash:
return id
id += 1
typeRegistry.hashes[id] = typeHash
return id
# fast memoized type ID; doesn't use `Tables`
proc typeId*(Type: type): TypeId {.noinit.} =
var typeId {.global.} : TypeId
var implId {.global.} = 0
var impl {.global.} = [
proc: TypeId {.inline noinit.} =
typeId = getId Type.typeHash
result = typeId
implId += 1,
proc: TypeId {.inline noinit.} = typeId
]
result = impl[implId]()
I hope someone finds something useful in that.