Hi, I am trying to interface Nim with a C library (libmill) that heavily depends on C macros as its API. So, it seems like the only way for me to do that is via the emit pragma and generate C code directly to use the macros. But, I ran into a couple problems, namely how do I emit C code as a part of an arguments to existing nim AST? I tried something like:
proc newEmitPragma(s: string): NimNode {.compileTime.} =
result = newNimNode(nnkPragma)
result.add(newColonExpr(newIdentNode("emit"), newStrLitNode(s)))
macro mill_chmake*(T, bufsz: expr): s =
result = newNimNode(nnkStmtList)
result.add(newCall("mill_chmake_int",
newCall("sizeof", T),
bufsz,
newEmitPragma(""" __FILE__ ":" mill_string(__LINE__))""")))
but that doesn't work b/c emit pragma isn't a proper AST NimNode. Does anyone have any suggestions for how to do this? Or am I missing something else I can obviously do?
Wait, I don't understand what does changing #define to #def do for me?
Also, I am not sure if I should be changing an upstream C library include file to resolve a nim FFI problem...
Sorry, I should have said "if you use c2nim". But I was assuming that libmill was a shared/static library.
(If I understand libmill correctly)
libmill is for writing c programs. Make your Nim code a library, or follow the documentation on interacting with C, but the main executable will be the C program, not the Nim code, so the Nim code won't be calling libmill stuff, but your c code will be calling the Nim stuff.
Hi @terry, what exactly are you trying to achieve with Nim & libmill? It's not at all clear from your post.
Please be specific as to whether you are intending to compile an executable containing a Nim main function, a shared library from Nim procs, an executable containing a C main function, or something else. And do you intend to call C from Nim, or call Nim from C (which is what @jlp765 seems to be assuming)?
I've never used the emit pragma myself, but according to the Nim Manual, it is used to insert specific code into the source files that have been generated in the intermediate language to which Nim is being transpiled, in the midst of transpiled Nim code. It's not entirely clear to me that this is what you need -- but then again, I'm not clear on what you're trying to achieve.
Hi, sorry. Indeed, I wasn't being clear.
I would like to call the libmill C library from Nim. The goal is to add a simple lightweight go-like CSP library to the Nim language. The problem is that the libmill C library's API are all based on C MACROs (see here) which I can not figure out how to invoke directly from Nim (importc didn't work for me on a macro).
I tried c2nim, but it choked on the complex defines in libmill.h.
Unfortunately, the macros are key to libmill's capability as it relies on C's __FILE__ and __LINE__ plus setjmp / longjmp to achieve the coroutine like effects.
Thanks for your help!
Hi @terry, thanks for the clarification. That is a lot clearer now.
I just double-checked for myself, and I can confirm that Nim can importc a C preprocessor macro to define a proc:
/* header.h */
#define foo(x) ((x) + 1)
# body.nim
proc foo*(x: int): int {.importc: "foo", header: "header.h", cdecl .}
echo foo(5)
To compile & run:
$ nim c --cincludes:. body.nim
$ ./body
6
That said, it may be that some C preprocessor macros are un-importable. (Or maybe __FILE__ & __LINE__ are confusing things?)
When I started work on Pymod, I didn't know that C preprocessor macros could be importc-ed, so I just wrapped each of them in a C function, which I then importc-ed. For example: https://github.com/jboy/nim-pymod/blob/master/pymodpkg/private/pyarrayobject_c.c#L61
But it sounds like that fallback wouldn't help you, because you need very specific C preprocessor magic.
Can you post an example of what you expect the Nim code would look like, when you use this hypothetical go-like CSP library in Nim?
Perhaps this would suggest an implementation approach.
For assertion macros with __FILE__ and __LINE__ in them,
you probably will need to replace them with Nim templates, like described in this post
Also, there is a known bug with multi line C macros with do/while loops. The c2nim translation has an incorrect indentation. It is easy to remedy by manually fixing the indent in the c2nim output file.
Thanks for the help jboy / jlp765. After a couple more hours of messing with libmill. I finally managed to get it to work.
Here is what my final main prog looks like:
import libmill
{.pragma: millcoro, codegenDecl: "coroutine $# $# $#".}
proc feeder(ch : chan, val: cint) : void {.millcoro.} =
while true:
chs(ch, val)
myield()
proc main(): void =
var ch1 = chmake(cint, 0)
var ch2 = chmake(cint, 0)
go(feeder(chdup(ch1), 111))
go(feeder(chdup(ch2), 222))
for i in 0..100:
choose:
var val {.nodecl.} : cint
mill_in(ch1, int, val):
echo "ch1, val = ", val
mill_in(ch2, int, val):
echo "ch2, val = ", val
myield()
chclose(ch1)
chclose(ch2)
main()
Here is my work-in-progress libmill.nim
import macros
{.passC: "-I/home/tchen2/.tools/include ".}
{.passL: "/home/tchen2/.tools/lib/libmill.a -lanl -lrt" .}
{.pragma: millcoro, codegenDecl: "coroutine $# $# $#".}
var FILENAMELINE {.nodecl, importc:"__FILE__ \":\" mill_string(__LINE__)" .}: cstring
var static_count {.compiletime.} = 0
const
mill_header = "<libmill.h>"
type
size_t = int
chan* = pointer
proc newEmitPragma*(s: string): NimNode {.compileTime.} =
result = newNimNode(nnkPragma)
result.add(newColonExpr(newIdentNode("emit"), newStrLitNode(s)))
proc mill_chmake(sz : size_t, bufsz: size_t, created: cstring): chan
{. importc: "mill_chmake", header: mill_header, noconv .}
proc mill_chs(ch : chan, val: pointer, sz: size_t, current: cstring): void
{. importc: "mill_chs", header: mill_header, noconv .}
proc mill_chclose(ch : chan, current: cstring): void
{. importc: "mill_chclose", header: mill_header, noconv .}
proc mill_chdup*(ch : chan, created: cstring): chan
{. importc: "mill_chdup", header: mill_header, noconv .}
proc myield*(): void {. importc: "yield", header: mill_header, noconv .}
template chmake*(T: expr, bufsz: int): expr = mill_chmake(sizeof(T), bufsz, FILENAMELINE)
template chclose*(ch: chan): expr = mill_chclose(ch, FILENAMELINE)
template chdup*(ch: chan): expr = mill_chdup(ch, FILENAMELINE)
template chs*[T](ch: chan, v: T): stmt =
var mill_val = v
mill_chs(ch, addr mill_val, sizeof(mill_val), FILENAMELINE)
template go*(fn: expr): stmt =
var mill_sp : bool
{.emit: """{
void* mill_sp = mill_go_prologue(__FILE__ ":" mill_string(__LINE__));
if(mill_sp) {
int mill_anchor[mill_unoptimisable1];
mill_unoptimisable2 = &mill_anchor;
char mill_filler[(char*)&mill_anchor - (char*)(mill_sp)];
mill_unoptimisable2 = &mill_filler;""".}
fn
{.emit: "mill_go_epilogue(); } } ".}
macro mill_in*(ch: expr, T: expr, name: expr, actions: stmt): stmt {.immediate.} =
var ch = repr(ch)
var name = repr(name)
var T = repr(T)
var c = repr(static_count)
static_count += 1
result = newNimNode(nnkStmtList)
result.add(newEmitPragma("break; } goto mill_concat(mill_label, " & c & "); }"))
result.add(newEmitPragma("char mill_concat(mill_clause, " & c & ")[MILL_CLAUSELEN];"))
result.add(newEmitPragma("mill_choose_in( &mill_concat(mill_clause, " & c & ")[0], (" & ch & "), sizeof(" & T & "), " & c & ");"))
result.add(newEmitPragma("if(0) { " & T & " " & name & ";"))
result.add(newEmitPragma("mill_concat(mill_label, " & c & "):"))
result.add(newEmitPragma("if(mill_idx == " & c & ") { " & name & " = *(" & T & "*)mill_choose_val(sizeof(" & T & "));"))
result.add(newEmitPragma("goto mill_concat(mill_dummylabel, " & c & "); mill_concat(mill_dummylabel, " & c & "):"))
result.add(actions)
macro choose*(actions: stmt): stmt {.immediate.} =
result = newNimNode(nnkStmtList)
result.add(newEmitPragma("""{ mill_choose_init(__FILE__ ":" mill_string(__LINE__)); int mill_idx = -2; while(1) { if(mill_idx != -2) { if(0) {"""))
result.add(actions)
result.add(newEmitPragma("break; } } mill_idx = mill_choose_wait(); } }"))
Cheers to the awesome Nim community
Congratulations on getting it to work, @terry!
You should put it up on Github and post a link back here. :)
Hi Araq,
Thanks for that, I wasn't aware of -d:nimCoroutines (I didn't see in the docs for some reason...). I will take a closer look at it.
Can you (or anyone else) explain a little more to me why libmill's light weight threading is a deadend when interacting with Nim's GC?
I looked at the C code generated by nim inside the nimcache folder and at first glance it doesn't seem like there will be any issues (although I still don't know what magic goes on inside the nimfr/popFrame functions).
The only issue that I can think of may be the fact that I can not pass Nim's ref into libmill's channels b/c the ownership of Nim allocated memory won't allow the GC to function properly. But if I stick to just using alloc/free would I be able to circumvent this problem. Or possibly, can I wrap libmill's chs (channel send) and chr(channel recv) with GC_ref / GC_unref to resolve this problem?
Are there other obvious issues associated with the GC that I am missing?
Thanks!