For some functionality i was using fork-join like pattern and noticed a large gradual spike in Virtual Ram usage. Also i could notice the large number of handles in system monitor too.
Following code should reproduce the behaviour. It seems to work correctly if i call closeHandle on thread system handle.
import std/locks
when defined(windows):
import std/winlean
var
thr: array[5, Thread[tuple[a,b,: int]]]
L: Lock
proc threadFunc(interval: tuple[a,b: int]) {.thread.} =
for i in interval.a..interval.b:
acquire(L) # lock stdout
# echo i
release(L)
proc some_function()=
initLock(L)
for i in 0..high(thr):
createThread(thr[i], threadFunc, (i*10, i*10+5))
joinThreads(thr)
deinitLock(L)
# when defined(windows):
# for i in 0..high(thr):
#echo closeHandle(cast[Handle](thr[i].sys))
for i in 0..<100000000:
some_function()
I am wondering if i am supposed to call closeHandle manually after joinThread to deallocate threads' resources, on windows atleast ?
@Araq could you please take a look?
Some suggestion would be good enough !
Yes. Generally, joinThread will not clean up thread-variable resources for you. Depending on what you use, you will need to deallocate their resources too. That includes the per thread GC-related variables from ORC.
In my (in progress) multithreading lib I have to do so as well. See below the proc I wrote for myself that cleans up the ORC GC as well as the asyncDispatcher (because I use async) and std/times variables that I use during logging.
proc clearThreadVariables*() =
## Internally, this clears up known thread variables
## that were likely set to avoid memory leaks.
## May become unnecessary if https://github.com/nim-lang/Nim/issues/23165 gets fixed
when not defined(butlerDocs):
{.cast(gcsafe).}:
times.localInstance = nil
times.utcInstance = nil
`=destroy`(getThreadDispatcher())
when defined(gcOrc):
GC_fullCollect() # from orc.nim. Has no destructor.
I am wondering if i am supposed to call closeHandle manually after joinThread to deallocate threads' resources, on windows atleast ?
You are not supposed to call "closeHandle". joinThread should probably do that for you. But createThread is for writing a thread pool, for anything else use Weave or Malebolgia.
Generally i am quite vigilant about what i allocate and what resources i might be using, and define destructors accordingly. But in this case i was only using createThread and joinThread standard routines , i only discovered the issue while benchmarking .
I also came across some thread variables related memory leak in issues . Thanks for illustrating your point, i would try to understand possible scenarios with thread variables in greater detail.
Thanks for replying!
I know about @mratsim weave and yours Malebolgia.
But i work on a BLAS backend in pure nim, and for me it is better to use createThread directly even though it would have some overhead , this fork-join pattern suits well for some of the routines i have implemented.
Is there some guide or code examples you could point me to understand better the behavior of thread variables or some gotchas i should look out for when using ` createThread` directly with ARC/ORC ?
Most of my multithreading works fine and i have occasional allocation (using allocShared) for which i save pointer to call deallocShared when done.