Hi all
I have a proc which calls execCmdEx in a loop. The command being executed is isolated enough to be able to call it in parallel, so I'd like to see whether doing this would speed up the overall process.
What would be the most idiomatic way of calling execCmdEx in parallel? parallel keyword? async/await? spawn?
Thanks!
you don't need threads
Maybe not need, but want?
From my current understanding when he uses multiple execCmdEx these are executed in sequence, that is not parallel.
Try this:
import threadpool, osproc
let a = spawn execProcess("ls -la .")
let b = spawn execProcess("echo $HOME")
let c = spawn execProcess("./my_other_script")
# wait for all the threads to finish
sync()
# the carrot gets the value out of the FlowVar
echo ^a
echo ^b
echo ^c
execCmdEx currently doesn't work with spawn... possibly a bug. :-)
But for basic use, this should do what you want.
I'm not particularly bothered whether it's threads or async, I just want to execute the command in parallel rather than sequentially.
Of course we know that async mean "non blocking", but not really parallel, see
@dawkot, unfortunately your implementation has a big performance flaw.
the sleepAsync 5 creates a minimum bound of 5 ms on the execution time + the time it takes to run through the event loop. It's the naive solution to the problem.
The more correct way to accomplish this with async is to register the file handle for stdout of the forked process with the asyncdispatch epoll handler.
There really should be a 'asyncExecProcess` proc in the std lib. Unfortunately this does not exist in the std lib at the moment.
There is an excellent third party library that does have a good implementation of this. @cheatface's aysnc tools: https://github.com/cheatfate/asynctools/blob/master/asynctools/asyncproc.nim#L122
For @OldhamMade's usecase, I'm not sure all the extra machinery of async is necessary. The difference between Threads vs. Async does not make much of a difference for this use case, and the spawn solution is much less code to get something that just works(tm).