I've now got some slightly less trivial async code I'm trying to get to run, using the 'news' WebSockets module. The compiler complains that my HTTP handler isn't gcsafe, and suggests "Annotate the proc with {.gcsafe.} to get extended error information."
However,
import blip
import news, asyncdispatch, asynchttpserver
when isMainModule:
echo("Starting Blip server on port 9001...")
var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
if req.url.path == "/blipsync":
var ws = await newWebsocket(req)
echo "Creating new Blip"
var blip = newBlip(ws)
await blip.run() # <-- this line triggers the gcsafe error
echo "...Closed Blip"
await req.respond(Http404, "Nope", newHttpHeaders())
waitFor server.serve(Port(9001), cb)
If I take out the line await blip.run(), the error goes away. But neither that function nor anything it calls appears to involve any global state. (It's too much code to post here in the forum, unless someone really wants me to!)
Did adding the {.gcsafe.} pragma silence a false alarm? Or is it suppressing a genuine problem that will cause thread-safety issues?
Thanks!
—Jens
I've added some more code, and now I'm getting actual errors that seem to explain the cause of the GC-safe violation ... if I could understand them.
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic instantiation of `async` from here
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(318, 24) Warning: 'receiveLoopNimAsyncContinue' is not GC-safe as it accesses 'nameIterVar`gensym21340263' which is a global using GC'ed memory [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic instantiation of `async` from here
/Users/snej/Code/nim/blip/src/blip.nim(126, 6) Warning: 'receiveLoop' is not GC-safe as it calls 'receiveLoopNimAsyncContinue' [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic instantiation of `async` from here
/Users/snej/Code/nim/blip/src/blip.nim(136, 6) Warning: 'run' is not GC-safe as it calls 'receiveLoop' [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic instantiation of `async` from here
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(278, 31) Warning: 'cbIter' is not GC-safe as it calls 'run' [GcUnsafe2]
receiveLoop is pretty simple (see below), but receiveLoopNimAsyncContinue and nameIterVar`gensym21340263 must be internal functions created by the macros behind the {.async.} pragma, so I don't really have a clue what's going on...
proc receiveLoop(blip: Blip) {.async.} =
while blip.socket.readyState == Open:
let packet = await blip.socket.receivePacket()
case packet.kind
of Binary: blip.handleFrame(cast[seq[byte]](packet.data))
of Close: return
else: continue
In short, no declaration of non-primitives global variable. There seems something declared at module level in blip that you didn't show.
When a proc is annotated with gcsafe, any proc called inside the proc has to be gcsafe also, regardless of whether you are compiling with --threads. And if any proc marked as gcsafe ever touched a module level non-primitive variable. An error will throw. The first proc annotated with gcsafe here is the callback function in serve().
Other people on this forum will probably explain better me. But here my 3-yo attempt, if a proc is annotated with {.gcsafe.}, the proc is supposed to be run in multi-threaded environment. And in current nim memory management model, each thread has its own gc and they work on their own, no knowledge of each other. This raises a problem: which gc should deal with the shared memory allocation and freeing? Nim current answer is to avoid the problem at its first place: not allow existence of non-primitives global variable touched at compile time.
gcsafe was one of the most annoying in my nim-experience... I had a long time didn't know why my single threaded web server has to be complained with non-gcsafe. Generally speaking, avoid global variable at all, this make testing easier and common practice also.
btw, if you need to pass some state to callback in server(), wrap it will closure. Nim current approach force threads to talk explicitly.
I recall my memory that I have written a database connection pool and logger for a web server, which they are supposed to be globally accessible. And then I have to fight with the unexpected gcsafe for a long time... dealing with threads in nim is another story, I encountered unexpected exits without any messages if I did something wrong in ITC...
If I add the gcsafe pragma to my callback proc, it now compiles without errors! Where's my "extended error information"?
There is a phase ordering problem in today's compiler so that in rare cases .gcsafe isn't inferred the way it should be. (Known problem.)
The good news is that when the code compiles with .gcsafe, it really is GC safe and nothing to worry about.
There is also https://github.com/nim-lang/RFCs/issues/142 for how to evolve the "GC safety" issues.
In short, no declaration of non-primitives global variable. There seems something declared at module level in blip that you didn't show.
My code has zero global variables.The global referred to in the error is nameIterVar`gensym21340263 which is obviously a compiler-generated name, not anything I did myself.
I've also run into the error Warning: 'send' is not GC-safe as it performs an indirect call here [GcUnsafe2] referring to this code:
type
MessageBuf* = object
# ...
sendProc: SendProc
SendProc* = proc(msg: sink MessageBuf): Future[MessageIn]
proc send*(buf: sink MessageBuf): Future[MessageIn] {.gcsafe.} =
return buf.sendProc(buf)
So it appears that calling a closure isn't GC-safe?!
Nim current approach force threads to talk explicitly.
I'm not using threads. This is all with asynchttpserver, which AFAIK is single-threaded.
I am not saying that you are using thread. This problem is that in standard library the callback function in asynchttpserver.serve() is marked as gcsafe, which (in my understanding) force all its inner-descendent function call to follow the gcsafe rule, no matter you compile with --threads or not.
In your case, add {.gcsafe.} at the end of type SendProc should solve your problem.
Weirdly, after I did some more work on my code I tried to reproduce the original problems I described (by backing out my tweaked versions of news and asynchttpserver) ... and they went away. Adding one single gcsafe pragma to the top-level HTTP server callback is sufficient.
This is very strange, but I'm happy that the problem is gone.