Macro generating AST that builds table would work. But maybe there is better solution.
EDIT: I just posted compile-time string obfuscation thread, you may find some ideas there: http://forum.nim-lang.org/t/1305
This is possible in principle. You do have to deal with the problem that the layout of a Table is not directly accessible, but in principle you can write a module that does an include tables and can access the internals. Of course, if the implementation of Table ever changes, that will break your code.
A bigger concern is that the tables layout is not designed for something that can be precomputed. You will still have to put it on the heap, and so you'll only save a negligible amount of cpu time at startup (we're talking microseconds here at worst). If you do want to optimize for space, you'll probably want to define your own hash table implementation that can be put in a const value.
An alternative solution is to simply generate a case statement for a table lookup (assuming you are dealing with strings or ints). Case statements for more than eight strings will use hashing for the generated code. You can do that in a macro, or (something that I do often) use Nim to generate Nim source code. For example, run the following with nim c -r --verbosity:0 --hints:off --warnings:off:
proc main() =
# procedure head
echo "proc lookup(key: string): int ="
echo " case key"
# lookup table
for key, value in items({
"alpha" : 1,
"beta" : 2,
"gamma" : 3,
"delta" : 4,
"epsilon": 5,
"zeta" : 6,
"eta" : 7,
"iota" : 8,
"kappa" : 9,
"lambda" : 10,
"mu" : 11
}):
echo " of \"" & key & "\": " & $value
# default value
echo " else: 0"
main()
Then use include or import on the resulting source file (note that if you do this you'll need to use make or another build tool to ensure that generated source files are being built before being included or imported).
Of course, you can also use a macro with parseStmt on the result, but if you have large tables, this can slow down compilation quite a bit.
Actually tables can be precomputed just fine with 0.11.2:
import tables
const
foo = {"ah": "finally", "this": "is", "possible.": "nice!"}.toTable()
However, there is no way to generate the "links" between the tables which have to be ref/ptr.
If I may jump into this discussion, I am trying to pars a JSon into a table at compile time. Right now it is done at runtime, but it is suppose to run on a Beaglebone and creating the index at runtime is a bit slow.
You can find the code here: https://github.com/xyz32/BeagleboneBlack_GPIO_nim/blob/master/src/bone.nim
The original pin definition comes from the bonejs project and I would like to keep it as is, to make it easier to upgrade.
Thank you
@reactormonk
I'd like to use this (compile-time-created) structure at run-time:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~ ~ ~ G_base <Table> ~ ~ -------- ------------ ~ ~ allowAcc | true | | "none" | ●--- ~ ~ |--------| --> |------------| \ ~ ~ prefix [0] | "--" | / | | | \ Attribute ~ ~ |--------| / ~ .. ~ ~ | ------------ ~ ~ [1] | "-" | | | | | | | "<text>" | comment ~ ~ |--------| | ------------ \ |------------| ~ ~ reName [0] | /.../ | | <Table> \ | "<name>" | option ~ ~ |--------| | ------------ --> |------------| ~ ~ [1] | /.../ | / | | | / | None | otype ~ ~ |--------| / --> ~ .. ~ ~ / |------------| ~ ~ optAttr [0] | ●------ / | | | / | ● | values ~ ~ |--------| / |------------| / -------\---- ------------------------ ~ ~ [1] | ●-------- | "x" | ●--- -------> | "alpha" | "beta" | ... | ~ ~ -------- ------------ ------------------------ ~ ~ ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
And what about having the data at compile time in the following type:
seq[tuple[name: string, gpio: int, led: string, mux: string, key: string, muxRegOffset: string, options: seq[string], eeprom: int, universalName: string, pwm: tuple[module: string, sysfs: int, index: int, muxmode: int, path: string, name: string], ain: int, scale: int]]
Then you could write things like:
PinTuple[0].pwm.muxmode
This way the binary wouldn't contain the string "muxmode", there is no string key lookup at runtime. It would be fast, memory efficient, and good for the teeth.
All you have to do is:
proc createEmpty(s, d: expr): expr {.compiletime.} =
let k = s.kind
result = d
if k == nnkBracket:
if result.isNil:
var bracket = newNimNode(nnkBracket)
result = prefix(bracket, "@")
bracket.add(nil)
elif result.kind != nnkPrefix and result[1].kind != nnkBracket:
raise newException(Exception, "mixing list with " & $result.kind)
for i in 0..<s.len:
result[1][0] = createEmpty(s[i], result[1][0])
elif k == nnkTableConstr:
if result.isNil:
result = newNimNode(nnkPar)
elif result.kind != nnkPar:
raise newException(Exception, "mixing tuple with something else")
for i in 0..<s.len:
if s[i].kind != nnkExprColonExpr:
raise newException(Exception, "use (a: 2) format instead of (2)")
var found = false
for j in 0..<result.len:
if $(result[j][0]) == $(s[i][0]):
result[j][1] = createEmpty(s[i][1], result[j][1])
found = true
break
if not found:
result.add(newColonExpr(ident($(s[i][0])), createEmpty(s[i][1], nil)))
elif k == nnkStrLit:
result = newLit("")
elif k == nnkIntLit:
result = newLit(0)
else:
raise newException(Exception, "unexpected type " & $k)
proc createTuple(s, d: expr): expr {.compiletime.} =
let k = d.kind
if k == nnkPrefix: # for @[]
var bracket = newNimNode(nnkBracket)
result = prefix(bracket, "@")
if s.isNil or s.len == 0:
result = newCall(newNimNode(nnkBracketExpr).add(ident("newSeq")).
add(newCall(ident("type"), d[1][0])))
else:
for i in 0..<s.len:
bracket.add(createTuple(s[i], d[1][0]))
elif k == nnkPar:
result = newNimNode(nnkPar)
for j in 0..<d.len:
var found = false
if not s.isNil:
for i in 0..<s.len:
if $(d[j][0]) == $(s[i][0]):
result.add(newColonExpr(d[j][0],
createTuple(s[i][1], d[j][1])))
found = true
break
if not found:
result.add(newColonExpr(d[j][0],
createTuple(newNilLit(), d[j][1])))
elif k == nnkStrLit:
result = if s.isNil: d else: s
elif k == nnkIntLit:
result = if s.isNil: d else: s
macro jsonToTuple(s: expr): expr =
let empty = createEmpty(s, nil)
createTuple(s, empty)
let PinTuple = jsonToTuple(
[
{
"name": "USR0",
"gpio": 53,
"led": "usr0",
"mux": "gpmc_a5",
"key": "USR0",
"muxRegOffset": "0x054",
"options": [
...
}
]
)
And you can forget importing the json library.
Cheers, Peter
@mora Thank you for the idea.
Although I don't want the end user to use yet another translation table that converts from a pin name "P8_14" to a "random" ID, so the hash table is still a good idea
And also The original approach seem to be faster (after I implemented a lazy loading for the hash table):
time ./bone
(name: USR0, gpio: 53, led: usr0, mux: gpmc_a5, key: USR0, muxRegOffset: 0x054, options: @[gpmc_a5, gmii2_txd0, rgmii2_td0, rmii2_txd0, gpmc_a21, pr1_mii1_rxd3, eqep1b_in, gpio1_21], eeprom: 0, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 0)
real 0m6.348s
user 0m6.238s
sys 0m0.014s
and the old way:
time ./bone
{"name":"USR0","gpio":53,"led":"usr0","mux":"gpmc_a5","key":"USR0","muxRegOffset":"0x054","options":["gpmc_a5","gmii2_txd0","rgmii2_td0","rmii2_txd0","gpmc_a21","pr1_mii1_rxd3","eqep1b_in","gpio1_21"]}
real 0m5.172s
user 0m5.072s
sys 0m0.023s
It seems that it is still not done at compile time, or the memory allocation takes a long time?
I did a benchmark test with your code (0.049 sec) and my code (0.001 sec) on my laptop. I know that it is x86 and not arm, but I hope that you will see some improvement on arm. With my code I can generate the table at compile time. Somehow I couldn't generate the table at compile time with the json approach.
Please find it here: https://github.com/petermora/BeagleboneBlack_GPIO_nim/blob/master/src/bone.nim
I didn't make the rest of the code compile, only worked on this file. I hope that you can use this. Peter
@mora Thank you very much for the help. I ran your implementation of the code, and indeed it is A LOT faster, somewhere around 0.2 seconds. Unfortunately the table has only a had full of keys, although the tuple has all the keys.
for example:
: unhandled exception: Key not found: 'P9_1' [ValueError] this are all the keys stored in the table:
{P9_40: (name: AIN1, gpio: 0, led: , mux: , key: P9_40, muxRegOffset: , options: @[], eeprom: 68, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 1, scale: 4096), P9_37: (name: AIN2, gpio: 0, led: , mux: , key: P9_37, muxRegOffset: , options: @[], eeprom: 69, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 2, scale: 4096), P9_44: (name: DGND, gpio: 0, led: , mux: , key: P9_44, muxRegOffset: , options: @[], eeprom: 0, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 0), P9_39: (name: AIN0, gpio: 0, led: , mux: , key: P9_39, muxRegOffset: , options: @[], eeprom: 67, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 4096), P9_43: (name: DGND, gpio: 0, led: , mux: , key: P9_43, muxRegOffset: , options: @[], eeprom: 0, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 0), P9_41: (name: CLKOUT2, gpio: 20, led: , mux: xdma_event_intr1, key: P9_41, muxRegOffset: 0x1b4, options: @[xdma_event_intr1, NA, NA, clkout2, NA, NA, NA, gpio0_20], eeprom: 13, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 0), P9_42: (name: GPIO0_7, gpio: 7, led: , mux: ecap0_in_pwm0_out, key: P9_42, muxRegOffset: 0x164, options: @[eCAP0_in_PWM0_out, uart3_txd, spi1_cs1, pr1_ecap0_ecap_capin_apwm_o, spi1_sclk, mmc0_sdwp, xdma_event_intr2, gpio0_7], eeprom: 4, universalName: , pwm: (module: ecap0, sysfs: 2, index: 0, muxmode: 0, path: ecap.0, name: ECAPPWM0), ain: 0, scale: 0), P9_45: (name: DGND, gpio: 0, led: , mux: , key: P9_45, muxRegOffset: , options: @[], eeprom: 0, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 0), P9_46: (name: DGND, gpio: 0, led: , mux: , key: P9_46, muxRegOffset: , options: @[], eeprom: 0, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 0, scale: 0), P9_38: (name: AIN3, gpio: 0, led: , mux: , key: P9_38, muxRegOffset: , options: @[], eeprom: 70, universalName: , pwm: (module: , sysfs: 0, index: 0, muxmode: 0, path: , name: ), ain: 3, scale: 4096)}
Thank you again
I did some benchmark in the code (import times; var t = cpuTime(); ...; echo cpuTime()-t;). The json loader %* seems to run at compile time, that's the main slowness, and not the table. The json structure uses heap, I couldn't create a table from it at compile time.
Yes, my solution creates all the fields. You can set the default values at the end of creatEmpty, if that helps. Currently the default value for int is 0, for string is "" (maybe it would be better to have nil?).
Cheers, Peter
Key not found: 'P9_1' [ValueError] Sorry, I didn't understand your problem at first.
Well, well, well. Please open Nim/lib/pure/collections/tables.nim, find line 225, and replace the line to these:
#swap(t.data, n)
var temp = t.data
t.data = n
n = temp
I'll open a github issue, the "swap" function seems broken at compile time. Peter
Update: Here is the github issue: https://github.com/nim-lang/Nim/issues/2946