Hello,
I was wondering what were the possible solutions to generate code coverage with current Nim 1.4.0.
As far as I know there is only:
Are there other solutions I'm not aware of ?
Could this functionality be added to a future roadmap of testament ?
I've implemented this in a private nim fork implementing this and something more general: tracing.
This is very useful in many cases, eg:
This is a bit analog to dtrace, which is an incredibly useful tool for debugging: https://8thlight.com/blog/colin-jones/2015/11/06/dtrace-even-better-than-strace-for-osx.html
It works similarly to --stacktrace:on, by instrumenting code, but does so with user defined callabacks.
Instead of generating the full trace, you can collect the unique LOC generated using a HashSet, generating a code coverage instead of an execution trace.
This is interesting. Do you think this is something that has the potential to be merged in the mainstream version ?
It would be a fantastic addition to testament html test report .
Merged in mainstream would be nice, but until (and if) that happens, it would be great if we can access your private fork Timothee (when and if you're ready, no questions asked :-).
I'm new to Nim (coming from Python, and Modula-2/3 before that), and I'd love to be able to generate execution tracing for debugging, & coverage (I'm currently using Nim for personal projects for Android / arm32).
Do you think this is something that has the potential to be merged in the mainstream version ?
I believe so, here's a draft since there is interest: https://github.com/nim-lang/Nim/pull/15827
I am using a modified version of https://github.com/yglukhov/coverage. I am generating lcov.info file and then using vscode extension coverage gutters to overlay coverage info into my code. Superb result.
Yes, you have to add cov annotation but I find it useful, as project grow I have started to remove {.cov.} annotations to keep performance of the debug build reasonable. Since it hot spot, you know it is covered.
I had an idea to implement version of https://github.com/yglukhov/coverage that will use term rewriting macro feature https://nim-lang.github.io/Nim/manual_experimental.html#term-rewriting-macros. You will only to import coverage module to get coverage remort. Though some improvements to term writing macro needs to happen to make it feasible.
Though some improvements to term writing macro in compiler needs to happen to make it feasible.
=> https://github.com/nim-lang/Nim/pull/15827#issuecomment-723382446
fwiw, rr gives the possibility to replay any program execution in a debugger: https://rr-project.org/ - this gives access to all interactive features of gdb effectively, without requiring any changes to nim - breakpoints, memory inspection etc etc work out of the box. We've used it to successfully debug difficult garbage collector issues for example.
ditto with code coverage - gcc already supports generating code coverage information which can be integrated trivially with lots of existing tools - we're using it for code coverage reports in nim-libp2p for example.
I've used rr in the past but I didn't know it could generate code coverage.
Or is it using gcov under the hood when replaying a program ?
Today I've faced my first experience with code coverage outside python world, and it went nice by mixing some examples I've found online into a nimble task. There are probably better ways to achieve the same, but this is quick and easy to understand.
It does the same coco does, but streamlined. It is necessary to adjust the "cleanup" phase according to the desired output, but it is ok to leave it catch all if vscode colors all you want.
let fileinfo = "lcov.info"
let lcov_args = "--rc lcov_branch_coverage=1"
let generated_not_to_break_here = "generated_not_to_break_here"
task coverage, "run code coverage":
# Based on:
# https://github.com/binhonglee/coco/blob/master/coco.nim
# https://github.com/samdmarshall/kisuru/blob/d7f039aae43bd8da9444d13b76849160f7af21ed/kisuru.nimble
# Compile tests in coverage mode
for file in listFiles("tests/"):
let (dir, file, ext) = splitFile(file)
if file.startsWith("test") and ext == ".nim":
exec &"""nim --hints:off --nimcache:nimcache/{file} c --debugger:native --passC:--coverage --passL:--coverage --outdir:{binDir} tests/{file}{ext}"""
# Reset counters
exec """lcov --base-directory . --directory nimcache/ --zerocounter """
# Run tests
for binfile in listFiles(binDir):
let (dir, file, ext) = splitFile(binfile)
if file.startsWith("test"):
exec binfile
# Generate coverage
exec &"touch {generated_not_to_break_here}"
exec &"""lcov {lcov_args} --capture --base-directory . --directory nimcache/ --output-file {fileinfo}"""
rmFile generated_not_to_break_here
# Cleanup tests
let currentFolder = absolutePath("")
exec &"""lcov {lcov_args} --extract {fileinfo} "{currentFolder}/src/*" -o {fileinfo}"""
#exec &"""lcov {lcov_args} --remove {fileinfo} "{currentFolder}/tests" -o {fileinfo}"""
#exec &"""lcov {lcov_args} --remove {fileinfo} "{currentFolder}/{generated_not_to_break_here}" -o {fileinfo}"""
# Generate HTML report
exec &"""genhtml --branch-coverage --legend --output-directory coverage/ {fileinfo}"""
I'm then using the results in vscode with https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters