In a previous post today, I mentionned I got "unexplanable" crashes, that were not always reproducible. It turn out, the crashes seem to be caused by runForever().
I have async code in a single module, the same one where I call runForever(). If I comment out the code calling runForever(), the crashes go away. If I comment out all the async code, but just keep the call to runForever(), it still crashes.
As I added asyncdispatch code in my module as an "afterthought", I used this code to run the event loop:
# ...
proc runForeverThread() {.thread.} =
## Executes runForever() in a separate thread.
runForever()
proc initProcess*(pid: ProcessID, processMapper: ProcessIDToAddress): void =
# ...
#asyncCheck initCluster(port, address)
var thread: Thread[void]
createThread[void](thread, runForeverThread)
I'm assuming this is somehow wrong. The description of "runForever()" in asyncdispatch doesn't actually say how to call it; maybe that should be explicitly documented.
@Hlaaftana Thanks. That was rather "obscure"; I wouldn't have thought there were two versions of createThread(). Unfortunately, I changed the code as you suggested, and it does not make the crash go away.
I think there is a single example of calling runForever() in the documentation. It is just called at the end of a module. If this is the only correct way to use it, it would be good to document this, as it could also just be "incidental" in this example.
But the bigger question is, what if the whole "async" thing is an optional feature of my module? I'm creating an alternative API to allow threads to send messages to each other. Sending messages transparently (as much as possible) across the cluster is just an "optional feature" for my API, so it doesn't make sense to force the user into "locking the main thread" with runForever(), unless the user wants the application to become part of a cluster, which could very well be decided throught a run-time parameter, or even interactively (off then on then off ...) throught the user-interface.
I think I can. I'm not sure if it is the same problem as in my main module, but this crashes for me:
import asyncdispatch
import os
proc runForeverThread() {.thread.} =
## Executes runForever() in a separate thread.
runForever()
proc initProcess(): void =
echo("In initProcess()")
#
var thread: Thread[void]
createThread(thread, runForeverThread)
proc doStuff(): void =
echo("In doStuff()")
# ...
initProcess()
sleep(500)
# ...
echo("After initProcess()")
doStuff()
Result:
Hint: ../bin/runforever [Exec]
In doStuff()
In initProcess()
Error: execution of an external program failed: '../bin/runforever '
The terminal process terminated with exit code: 1
As seen, the program never gets to output "After initProcess()" ...
Thanks for reproducing. It doesn't crash for me (macOS), what OS are you on and what Nim version are you using?
btw, the : void is redundant for proc return types.
Windows 10, x64
Microsoft Windows [Version 10.0.16299.192]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\Users\Sebastien Diot>nim --version
Nim Compiler Version 0.17.2 (2017-09-07) [Windows: amd64]
Copyright (c) 2006-2017 by Andreas Rumpf
git hash: 811fbdafd958443ddac98ad58c77245860b38620
active boot switches: -d:release
Visual Studio 2017 Developer Command Prompt v15.0.26228.13
Copyright (c) 2017 Microsoft Corporation
[vcvarsall.bat] Environment initialized for: 'x64'
I tested using GCC 7.2 msys2 on Windows 10. Nim 0.17.2
When compiled with no release, it simply crashed/segfault.
When compiled with release, it gave: Error: unhandled exception: No handles or timers registered in dispatcher. [ValueError]
Interesting! Does anyone know what "No handles or timers registered in dispatcher" actually mean? That is, why is it an error?
Regarding the compiler, I hadn't thought about trying different compilers to work around crashes. But unfortunately, I want to use this in UE4, and UE4 pretty much requires VC under Windows, so I'll need a work around. I guess I could just add a dummy timer, for a start.
EDIT: OK, I added a "timer" (based on this thread); it made no difference:
import asyncdispatch
import os
proc doNothing() {.async.} =
while true:
await sleepAsync(100)
proc runForeverThread() {.thread.} =
## Executes runForever() in a separate thread.
runForever()
proc initProcess(): void =
debugEcho("In initProcess()")
asyncCheck doNothing()
var thread: Thread[void]
createThread(thread, runForeverThread)
proc doStuff(): void =
debugEcho("In doStuff()")
# ...
initProcess()
sleep(500)
# ...
debugEcho("After initProcess()")
doStuff()
OK, I've got it! It has nothing to do with runForever() or asyncdispatch.
Here another example that crashes:
import os
proc whatever() {.thread, nimcall.} =
echo("TEST")
proc initProcessX(): void =
echo("In initProcess()")
var thread: Thread[void]
createThread(thread, whatever)
echo("initProcess() done")
proc doStuff(): void =
echo("In doStuff()")
# ...
initProcessX()
sleep(500)
# ...
echo("Crashes before getting here!")
doStuff()
And here, one that doesn't crash:
import os
var thread: Thread[void]
proc whatever() {.thread, nimcall.} =
echo("TEST")
proc initProcessX(): void =
echo("In initProcess()")
#var thread: Thread[void]
createThread(thread, whatever)
echo("initProcess() done")
proc doStuff(): void =
echo("In doStuff()")
# ...
initProcessX()
sleep(500)
# ...
echo("Crashes before getting here!")
doStuff()
It looks like the issue is that I don't have a "hard reference" to the thread itself, so I assume it gets garbage-collected. Should I always keep a hard-reference to threads, until they are done? Or is that a bug?
EDIT: My "real" code is still crashing after the fix I discovered, but now I'm getting a stack-trace, so I think I should eventually work it out too:
First msg sent from Thread 0.
Thread 1 initialised.
Traceback (most recent call last)
kueues.nim(1032) runForeverThread
asyncdispatch.nim(278) runForever
asyncdispatch.nim(283) poll
Error: unhandled exception: No handles or timers registered in dispatcher. [ValueError]
Message received after 0 'timestamp units'
Error: execution of an external program failed: '../bin/test_kueues '
The terminal process terminated with exit code: 1
EDIT #2: OK, I just have to call "asyncCheck" in the same thread that calls runForever(). I guess that makes sense. :)
@mikra "After the fact", it totally makes sense to me that I would need to keep a reference to a thread. But as I was writing the code, it wasn't obvious at all, because I'm used to Java, where threads are "allocated on the heap", and so they don't get destroyed by leaving the scope where they were created.
I've wasted a lot of time on this, along with other people here trying to reproduce the issue or find out what I was doing wrong (or if it was a bug). The information available at the point of the crash was less than helpful.
I think something should be done here, so that others don't waste their time like I did, but I can't see what could be changed. There are surely use-cases for creating thread objects in the local heap or on the stack, so I don't think this should be forbiden. The only idea I came up with, was to use a thread destructor that spits out an error, if the thread is still running. But destructors aren't reliable yet(?), and so idk if this is possible solution.
At the very least, it wouldn't hurt to "state the obvious" in the createThread() method documentation, if nothing else can be done.
@monster yea threading could be painful :->. my 2cts: if you go for threads and you have strange spurious side effects/crashes the likelihood turns almost to 1 that it has to do with threading.
I don't know how this could be handled. Maybe the compiler could check that some references going out of scope? remember, you are operating with the low-level thread api. The easiest way is to store the nim-thread-handles within a container. This is also the reason why I love Nim and dislike Java. The syntax is very handy and powerful, no hierogliphes, you have 100 percent focus on the problem and not the language; if you like you can dive into bits and bytes but you don't have to (almost:-) and so on :-)
If you use the C/Cpp backend the cpp-reference is very very helpful: http://en.cppreference.com/w/cpp/thread
btw, you could use the threadpool api but if you are using endless-loops it's better to model the loop outside of the worker method.
@mikra "After the fact", it totally makes sense to me that I would need to keep a reference to a thread. But as I was writing the code, it wasn't obvious at all, because I'm used to Java, where threads are "allocated on the heap", and so they don't get destroyed by leaving the scope where they were created.
I do not understand this: as far as I know Nim still doesn't have destructors and RAII semantics so allocation and deallocation should be completely managed by the GC which should keep the thread object alive even if it goes out of scope as long as the thread is running (exactly like in Java). I tried to write this simple piece of code:
type P2d = object
x,y : float
proc foo() : auto =
var pt = P2d(x:4, y:5)
result = proc() : P2d = pt
pt.x = -1
let f = foo()
echo f()
@woggioni, how about adding threading to your example, like
# compile with --threads:on --threadAnalysis:off
# because this thread need to modify `thef` value
type
P2d = object
x, y: float
P2dp = proc(): P2d
var thef: P2dp = nil
proc foo(): auto =
var pt = P2d(x: 4, y: 5)
result = proc(): P2d = pt
pt.x = -1
proc giveFoo() {.thread.} =
thef = foo()
var f = foo()
echo f()
var t: Thread[void]
t.createThread giveFoo
joinThread t
if not thef.isNil:
echo thef()
else:
echo "nothing"
As we can see, thef is not nil but the P2d defined from other thread is. (I just followed the example, but we actually need ref object instead of object )
@mashingan this confirms that even local value-typed variables are GC managed.. But using ref doesn't solve the problem:
type
P2d = object
x, y: float
P2dp = proc(): ref P2d
var thef: P2dp = nil
proc foo(): auto =
var pt : ref P2d
new[P2d](pt)
pt.x = 4
pt.y = 5
result = proc(): auto = pt
pt.x = -1
proc giveFoo() {.thread.} =
thef = foo()
var f = foo()
echo f()[]
var t: Thread[void]
t.createThread giveFoo
joinThread t
if not thef.isNil:
echo thef()[]
else:
echo "nothing"
@woggioni The GC is involved in your example because the proc(): auto = pt is a closure; it captures pt as part of its environment.
Closures in Nim are bound to the thread that creates them because the closure's environment is allocated on the thread-local heap (managed by the thread's GC). The local variable pt which has a value type (not ref/ptr) would normally be stack-allocated (or even register-allocated) but because it is captured by the closure, it gets directly allocated on the heap in the closure's environment instead.
@mashingan FYI, your example is unsafe because it stores a GC'ed value -- the closure returned by foo() into a global variable. (The compiler reports as such for me.)