To do some simple parallel processing I have two nim programs, where one simply runs a number of the other through osproc's startProcess, waits for them to end, and then does some final stuff with the outputs. The child program can be run by itself, with exactly the same parameters as given by the parent without any issues, and always terminates correctly on its own.
I have run into a strange behavior that I simply cannot get my mind around. I have a simple waiting loop that counts the number of ended processes with p.running() and then processes the outputs after they have all stopped. Now, coding sloppily on my own I have no idea what caused this, but at one point a problem began where this parent process wouldn't end. After each child process should have ended naturally (because they have a time limit), using either running or peekExitCode would return true or -1 respectively. This happens seemingly at random, to a random number of child processes each run. So I tried making it end the child processes instead of waiting, and then looked at the output to see where things went wrong. But from the output it is very clear that each of the processes had in fact concluded, especially after I added assert false at the very end to make sure. The child processes, which each use a significant amount of cpu while running, each drop to 0% usage after the allotted time as well.
Now, just to make this extra strange, I tried to see what happened to exit code for the running processes as I interacted with the outputs, and found that peeking the code after running through a for line in p.lines() loop, they are then changed to the expected 0 (or 1 with the assert). So I echoed the lines as I read them, and then echoed the peeked code after each line. The baffling result (to me at least) is that some way through outputting the lines, the peeked code changes.
I have no idea where to even look at this point. Am I just misunderstanding something about running processes? Are there extra steps take to "update" the status of them?
To do some simple parallel processing I have two nim programs, where one simply runs a number of the other through osproc's startProcess,
You should probably use threads instead of external processes.
At your leisure, you can go for a lower-level approach using os threading API , coupled with Channel (or the messaging queue of your choice) for synchronisation.
You can also use threadpool + Future / FlowVar with either
Finally, ti's probably worth taking a look at https://github.com/mratsim/weave/ for parallization.
Thing is, I have no use for any of the additional functionality that come with those other solutions. The processes are completely separate, with no sort of communication/messaging needed. It would take a significant rewrite to implement, and I have exactly no experience with any of them.
Other than to get around this problem, if it truly can't be helped, I just don't think it's worth it.
Maybe this post help you: https://forum.nim-lang.org/t/4454#27937
Or your problem might be similar to this: https://github.com/nim-lang/Nim/issues/8648
If a child process writes large data to a pipe and the pipe buffer got full, then write to the pipe get blocked until parent process read the pipe. So if your child processes write something larger than pipe buffer size, your parent process have to read outputs from them in order to keep them running.
For more info:
For linux user:
Search Pipe capacity in: https://www.man7.org/linux/man-pages/man7/pipe.7.html
For ms windows user:
https://docs.microsoft.com/en-us/windows/win32/ipc/anonymous-pipe-operations
This library might works better than osproc in standard library: (but I have never used it) https://github.com/cheatfate/asynctools
If you want to run your child processes on other machines, socket would be better than pipe for inter process communication. https://nim-lang.org/docs/asyncnet.html