Hi all,
I'm working on a microbenchmark framework for Nim, like facebook folly and google benchmark.
I want to use the getTicks() proc from https://github.com/nim-lang/Nim/blob/devel/lib/system/timers.nim
Unfortunately this proc does not seem to be exported. Other files access it by
include "system/timers"
Maybe this was done for performance reasons, but {.inline.} annotations should work just as well.
I think the proper approach would be to make it importable, exporting the necessary procs. Not sure if that's wanted in the Nim distribution or would live outside of it as a copy.
Related, there are 2 other benchmarking suites:
If it is not advisable, how to properly import it?
Well it's for internal use only and we can change the API. We could make it a module on its own but I don't intend to do that. Currently there is no way to import it properly, sorry.
Thanks. I have worked hard on making the module fit for public, including the nicely generated nim documentation. It can be found at https://github.com/ivankoster/nimbench
Related, there are 2 other benchmarking suites
I think the nimbench module has reason for existence, besides the stopwatch and nim-benchmark modules:
Currently there is no way to import it properly, sorry.
Okey, no problem, will work on importing the necessary functions myself. Will be added in a future commit, soon.
nimbench looks really nice, good job!
The other nim-benchmark module claims to use the RDTSC instruction to measure. I have not looked at the source yet, but this is really dangerous for measuring, in case you don't account for the fact that threads can jump between cpus, making you read the wrong RDTSC value.
This would help I assume: http://nim-lang.org/docs/threads.html#pinToCpu,Thread[Arg],Natural
nimbench looks really nice, good job!
Thank you :)
This would help I assume
Possibly, but you still have clock frequency fluctuations that can throw off the measurements. There is more information to be found on https://en.wikipedia.org/wiki/Time_Stamp_Counter as to why using that instruction is not a good idea :)
Great, waiting for it to be on Nimble!
It could be on right now, however, nimbench is only tested on one platform yet: windows 64bit with vcc/msvc compiler. I don't know if nimble package listing accepts this? Nimbench still has to be ported to other platforms.
I'm not really a linux guy, but I will setup a VM with linux soon to port nimbench.
I guess Ubuntu with GCC or Clang is the most popular setup? Looking to get as much coverage as possible so more people can enjoy nimbench :)
Nimbench creates a proc in-place for each of your bench blocks and calls this function multiple times to create measurements. So we can use global variables inside these bench blocks, since they are just a nim proc:
import nimbench
var d = 1.0
bench(fpOps1, m):
for i in 1..m:
var x = float(i)
d = d + x
runBenchmarks()
echo d
Since you don't know how many iterations the framework is gonna do, the value of d doesn't seem that useful in this snippet.
I don't really know what your actual use case is. You can check the documentation for the current interface that nimbench provides. If you want more / different functionality, we can discuss this here or github and see what we can do.
You can also check the benchmark frameworks that inspire nimbench, maybe there is some functionality there that you would like, and we could port it to nimbench if there is demand. I have added links to these frameworks at the top of the documentation
Let me show you a minimal repro of the problem I'm having. The following works:
import nimbench
const n = 1024 * 256
bench(seqPush, m):
var s = newSeq[int](n)
for i in 0 .. m:
s[i] = 1
doNotOptimizeAway(s)
The following doesn't work (crashes at run-time with attempt to read from nil):
import nimbench
const n = 1024 * 256
var s = newSeq[int](n)
bench(seqPush, m):
for i in 0 .. m:
s[i] = 1
doNotOptimizeAway(s)
What am I doing wrong?
I don't know either. This works:
import nimbench
const n = 1024 * 256
var s = newSeq[int](n)
bench(seqPush, m):
var x: int
for i in 0 .. m:
s[i] = i
x = i
doNotOptimizeAway(x)
runBenchmarks()
echo s.repr()
@gmpreussner
You are making a seq with n nodes, n is 1024 * 256 = 262_144. In the benchmark you access s[i]. If a code snippet runs really fast, m can get bigger than that, depending on your CPU speed. (And s[i] = i seems to be really fast) So in
for i in 0 .. m:
With const n = 1024 * 256 i get the error randomly sometimes. With const n = 1024 * 256 * 2 it seems to be fixed.
My suggestion is to write your benchmark snippet in a different way so that it will never access out of bounds memory.