Hello,
I've been having issue with the following code which is part of a library to automatically compute gradients of basic operations. Full code
The idea is to do the operation (+, *, - ...) and at the same time add the gradient transformation as closures (type BackProp).
Currently I can store BackProp[T]: proc( _ : T): T It works fine for float32 for example, but I need to generalize to linalg's Matrix32, DMatrix32, Vector32, DVector32.
Here is the current working implementation :
type
BackProp[T] = proc (gradient: T): T {.noSideEffect.}
Node[T] = object
weights: array[2, BackProp[T]]
parents: array[2,int]
Context*[T] = object
nodes: ref seq[Node[T]]
proc newContext*[T]: Context[T] {.noSideEffect.} =
result.nodes = new seq[Node[T]]
result.nodes[] = @[]
let ctx = newContext[float32]
Now I need BackProp to be T -> U, T and U can be different types or sometimes the same. For example I will have a series of Nodes: BackProp float32 -> Vector32 and Vector32 -> Matrix32
I can get to there:
import linalg
type
BackProp[T, U] = proc (gradient: T): U {.noSideEffect.}
Node[T, U] = object
weights: array[2, BackProp[T, U]]
parents: array[2, int] #ref indices to parent nodes
Context*[T, U] = object
nodes: ref seq[Node[T, U]]
proc newContext*[T, U]: Context[T, U] {.noSideEffect.} =
result.nodes = new seq[Node[T, U]]
result.nodes[] = @[]
let ctx = newContext[float32, Matrix32[2,2]]
But now there is no way to add a Matrix32[2, 2] -> Matrix32[4, 4], I'm stuck with float32 -> Matrix.
I've tried the 2 following snippets which fails:
import linalg
type
Tensor[float32] = float32 or DVector32
BackProp[T] = proc (gradient: Tensor[T]): Tensor[T] {.noSideEffect.}
Node[T] = object
weights: array[2, BackProp[T]]
parents: array[2, int] #ref indices to parent nodes
Context*[T] = object
nodes: ref seq[Node[T]]
proc newContext*(T: typedesc[SomeReal]): Context[T] {.noSideEffect.} =
result.nodes = new seq[Node[T]]
result.nodes[] = @[]
let ctx = newContext(float32)
Error is Cannot evaluate at compile time: T
And I also tried removing the T generic
import linalg
type
Tensor = float32 or DVector32
BackProp = proc (gradient: Tensor): Tensor {.noSideEffect.}
Node = object
weights: array[2, BackProp]
parents: array[2, int] #ref indices to parent nodes
Context* = object
nodes: ref seq[Node]
proc newContext*(T: typedesc[SomeReal]): Context {.noSideEffect.} =
result.nodes = new seq[Node]
result.nodes[] = @[]
let ctx = newContext(float32)
Error: 'BackProp' is not a concrete type.
I tried wrapping all the types (float32, Vector32, DVector32, Matrix32, DMatrix32) in an object variant but I also get Vector32 is not a concrete type. Probably because Vector32 in linalg is actually Vector32[N: static[int]].
I also tried using a tuple (Backprop[T,U], Backprop[T,U]) instead of an array but I still can't compile with same errors.
Is there a solution to have Node[Closure[float32,Vector32]], Node[Closure[Vector32,Matrix32]] and Node[Closure[Matrix32,Matrix32]] in same seq (ClosureType being BackProp in this case)?
If interested, here is the full background of why I'm passing a closure instead of just a value.
Here is my take with Object Variant which fails with "Vector32" is not a concrete type as well. Probably because Vector is also parameterized by a static[int].
I feel like I have to recode all the base types with an unified float, vector, matrix (and why not n-dimensional array) type to get it to work:
import linalg
type
BackPropKind = enum
## List of BackProp[T,U] types
bpFF,
bpFV,
bpVF,
bpVV,
bpFM,
bpVM,
bpMF,
bpMV,
bpMM
BackProp = ref BackPropObj
BackPropObj = object
case kind: BackPropKind
of bpFF:
ff: proc (gradient: float32): float32 {.noSideEffect.}
of bpFV:
fv: proc (gradient: float32): Vector32 {.noSideEffect.}
of bpVF:
vf: proc (gradient: Vector32): float32 {.noSideEffect.}
of bpVV:
vv: proc (gradient: Vector32): Vector32 {.noSideEffect.}
of bpFM:
fm: proc (gradient: float32): Matrix32 {.noSideEffect.}
of bpVM:
vm: proc (gradient: Vector32): Matrix32 {.noSideEffect.}
of bpMF:
mf: proc (gradient: Matrix32): float32 {.noSideEffect.}
of bpMV:
mv: proc (gradient: Matrix32): Vector32 {.noSideEffect.}
of bpMM:
mm: proc (gradient: Matrix32): Matrix32 {.noSideEffect.}
Node = object
## Represent an operation
## Stores the gradient transformation for backprop in weights
## Stores indices of parent operation in parents
weights: array[2, BackProp]
parents: array[2, int] #ref indices to parent nodes
Context* = object
## Tape / Wengert list. Contains the list of applied operations
nodes: ref seq[Node]
proc newContext*: Context {.noSideEffect.} =
## Initialize a context (Tape / Wengert list)
result.nodes = new seq[Node]
result.nodes[] = @[]
let ctx = newContext()
Note: using DVector and DMatrix from linalg works fine however.
They do not have a static[int] parameter in their type like Matrix32[M,N: static[int], T].
I will go forward with this design for now.
PS: Is there a macro to automate object variants creation?
Yup, here it is
I am pretty sure there is another one called data but I cannot recall the nim package that implements it