Hi. How how to define shared sequence correctly and access from different threads? I tried this:
import locks, os, terminal, strformat, threadpool
var sblock: Lock
var sharedbuff {.guard: sblock.} : ptr seq[string]
initLock(sblock)
{.locks: [sblock].}:
sharedbuff = cast[ptr seq[string]](allocShared0(sizeof(seq[string])))
#GC_ref(sharedbuff[])
sharedbuff[] = @[]
var begin = true
proc setProc (thrName:string, pause: int) =
var i = 0
while begin:
{.locks: [sblock].}:
for j in 1..5:
sharedbuff[].add(fmt"{thrName}-{i}-{j}")
sleep(pause)
i += 1
proc outProc (pause: int) =
var i = 0
while begin:
{.locks: [sblock].}:
echo "\n OUT N", i
while sharedbuff[].len > 0:
echo " ", sharedbuff[].pop()
sleep(pause)
i += 1
echo "PRESS ANY KEY TO STOP AND EXIT..."
spawn setProc("Thread One", 250)
spawn setProc("Thread Two", 500)
spawn outProc(2050)
discard getch()
begin = false
sync()
{.locks: [sblock].}:
sharedbuff[] = @[]
#GC_unref(sharedbuff[])
deallocShared(sharedbuff)
Thanks for answer! Yes, I'm using lockers and prointers. But I doing something wrong in lines (to write a new data after pointer derefernce):
sharedbuff[].add(...
sharedbuff[].pop()
sharedbuff[] = @[]
is the culprit, it uses the thread local heap and your ptr indirection does nothing except making Nim's type checker happy. ;-)
I would suggest using the standard library's SharedList.
As it's been mentioned above, you should not share any heap-allocated managed objects (refs, strings, seqs) across threads because Nim uses thread-local heaps (and garbage collection). It's generally unsafe to share them because the GC is oblivious to the use across threads and may deallocate the object even if it's still in use (by another thread).
If you need something else than a singly-linked list, you can use the example above as a good reference on how to do it.
Hi r3d9u11,
finally I got your code running but there are some pitfalls. the sequence itself seems not to be threadsafe so I don't recommend using it in conjunction with multithreading. please see my comments within the codesnippet. Better to use the mentioned SharedList or something else.
Also there is some strange behaviour (I think that could be a compiler issue); hopefully @Araq or someone who knows the internals could comment it (I didn't checked the emitted c source yet): if you pass the pointer of the sequence by parameter the code compiles and works - if you access the pointer directly (global) the code compiles but crashes. I never investigated that because I always pass some init-context always by parameter.
Hope this helps you a bit - and I learned something new :-)
import locks, os, terminal, strformat, threadpool
var sblock: Lock
# possible bug: if the sequence does realloc outside the owning thread code will crash
# var sharedbuff : seq[string] = newSeq[string](2)
var sharedbuff : seq[string] = newSeq[string](50)
var ptrsharedbuff : ptr seq[string] = cast[ptr seq[string]](sharedbuff)
initLock(sblock)
var begin = true
# without the next call the code will crash.
# possible some init done inside the first add?
sharedbuff.add("test")
proc setProc (thrName:string, pause: int, buff : ptr seq[string] ) =
var i = 0
while begin:
{.locks: [sblock].}:
for j in 1..5:
buff[].add(fmt"{thrName}-{i}-{j}")
# if the sequence does reallocation the code will crash
# sequence impl could check if called outside the owning context
sleep(pause)
i += 1
proc outProc (pause: int, buff : ptr seq[string] ) =
var i = 0
while begin:
{.locks: [sblock].}:
echo "\n OUT N", i
while buff[].len > 0:
# accessing the global ptr directly result in crash. possible compiler bug?
# echo " ", ptrsharedbuff[].pop()
echo buff[].pop() # works
sleep(pause)
i += 1
echo "PRESS ANY KEY TO STOP AND EXIT..."
spawn setProc("Thread One", 250,cast[ptr seq[string]](sharedbuff.addr))
spawn setProc("Thread Two", 500,cast[ptr seq[string]](sharedbuff.addr))
spawn outProc(2050,cast[ ptr seq[string] ](sharedbuff.addr))
# discard getch() # if you remove the comment code crashes. possible bug within getch()?
begin = false
sync()
deinitLock(sblock)
#without the next call the code will crash.
sharedbuff.add("test")
possible some init done inside the first add?
Yes, will reserve a new chunk of elements, after the initial 50 blank ones, and will place "test" in 1st pos of the new chunk. Subsequent .add() works because realloc reserves in chunks and not one by one.
Seq data realloc will use new from the memory thread heap local, so back to square one regarding mem access errors.
My 2 cents: Parallelism nowadays tends to favour the share mem by communicating approach -- using channels or FlowVar for data return with a main controller for results management.
import locks, os, terminal, strformat, threadpool
var sblock: Lock
var sharedbuff : seq[string] = newSeq[string](50)
var ptrsharedbuff : ptr seq[string] = cast[ptr seq[string]](sharedbuff.addr)
initLock(sblock)
var begin = true
# seq's owning thread needs initial add on empty seq
sharedbuff.add("test")
proc setProc (thrName:string, pause: int, buff : ptr seq[string] ) =
var i = 0
while begin:
{.locks: [sblock].}:
for j in 1..5:
# ptrsharedbuff[].add(fmt"{thrName}-{i}-{j}")
buff[].add(fmt"{thrName}-{i}-{j}")
sleep(pause)
i += 1
proc outProc (pause: int) =
var i = 0
while begin:
{.locks: [sblock].}:
echo "\n OUT N", i
while ptrsharedbuff[].len > 0:
# echo " ", ptrsharedbuff[].pop()
echo ptrsharedbuff[].pop() # works
sleep(pause)
i += 1
echo "PRESS ANY KEY TO STOP AND EXIT..."
spawn setProc("Thread One", 250,ptrsharedbuff)
spawn setProc("Thread Two", 500,ptrsharedbuff)
spawn outProc(2050)
# discard getch() if you remove the comment code crashes. possible bug within getch()?
begin = false
sync()
deinitLock(sblock)
Thanks to all !!! This works fine:
import locks, os, terminal, strformat, threadpool
var sblock: Lock
var sharedbuff {.guard: sblock.}: seq[string] = newSeq[string](50)
var ptrsharedbuff {.guard: sblock.}: ptr seq[string] = addr(sharedbuff)
initLock(sblock)
var begin = true
sharedbuff.add("test")
proc setProc (thrName:string, pause: int, buff : ptr seq[string]) =
var i = 0
while begin:
{.locks: [sblock].}:
for j in 1..5:
buff[].add(fmt"{thrName}-{i}-{j}")
sleep(pause)
i += 1
proc outProc (pause: int, buff : ptr seq[string]) =
var i = 0
while begin:
{.locks: [sblock].}:
while ptrsharedbuff[].len > 0:
echo buff[].pop()
sleep(pause)
i += 1
echo "PRESS [ENTER] TO STOP AND EXIT..."
spawn setProc("Thread One", 1000, ptrsharedbuff)
spawn setProc("Thread Two", 2000, ptrsharedbuff)
spawn outProc(3000, ptrsharedbuff)
discard stdin.readLine()
begin = false
sync()
deinitLock(sblock)
Hi! Sure, seq (and many others collections/structures/datatypes from std) doesn't intended for usage in multithreading ;-( But for me this example is a valuable work experience with std-data types, GC, multithreading and pointer's in Nim.
What is the "buffer pool", did you mean a shared array with fixed size + locks?