i have some experience programming Nim, but can't succeed redirecting / catching stdout. i'm trying to interact with a chess engine binary and use Owlkettle, providing Nim widgets to build a GUI, see my discussion at https://github.com/can-lehmann/owlkettle/discussions/134 how to fill a text widget with such data ?
i found forum entries like :
Redirect stdout for certain execution https://forum.nim-lang.org/t/6909
..but i can't make those code parts work - can anyone give some basic code, or point me to a working example ? i see techniques like async are used, but for me these are often hard to understand ..
There is typically a system call which spawns a process and gives you the file descriptors for that process's stdin and stdout. This is usually called popen. Then you have to read from it regularly like it were any other stream and write to it (& flush) when you want to push data. That's the basic of it.
The reason you would use async is because you want to be able to do other things while waiting for that process to finish doing something. If you were to push chess moves to a process and then try to read a reply: your program would block until the chess engine was done. With async streams it will still wait for a response but you can control what is going on until that response comes in.
I'd recommend ignoring async et all until you have a grasp on how to spawn a process and access its pipes. Async is very useful but its a whole extra bag of worms you need to learn separately.
The trickier parts of any approach in this vein arises when keeping alive the "companion process" to your reader and IPC is a big topic { Here are some slides and cligen/osUt shows how to wrap popen for Windows and buffering as well as blocking can be a hazard with coprocess designs. }. I agree with @icedquinn that starting simple is best, though. There is a very simple approach here - IF you can do "one shot" / "fire & forget" kind of output generation then you can just do:
import std/osproc
let (outputStr, exitStat) = "nextChessMove --white boardState".execCmdEx
and the be on with your GUI prototyping. You just need to make sure the program is in your PATH. The overhead of this is less than a few milliseconds (usually 30..100 microseconds for me on Linux) which may be much less than the chess engine for some number of "moves ahead" of board state being examined. { If that turns out to be a lot of overhead in context then you can learn more IPC. }
To be fully concrete, I am also imagining "boardState" as some flattened 64 byte representation of the 8x8 board with some kind of white-black piece encoding (like capital letters for white, lowercase for black and p=pawn, r=rook, n=knight, b=bishop, q=queen, k=king, '.'=no piece so there is no need for command parameter quoting).
You could even do quit "checkmate", 1 inside nextChessMove which otherwise spits to outputStr something like the chess notation for moves or even the whole new board state. { This all assumes "move history" does not matter, of course. Otherwise the string could be that instead of board state. And maybe this is all nearly standardized in the context of chess engines the way chess moves are among chess nerds. Just trying to help. }
i found a solution at https://glenngillen.com/learning-nim/executing-external-commands/ and i created this script, which works !
import osproc, streams
let process = startProcess("stockfish", args = [], options = {poUsePath})
let (fromp, top) = (process.outputStream, process.inputStream)
top.write "uci\n"
top.flush
echo fromp.readLine
top.write "ucinewgame\n"
top.flush
echo fromp.readLine
top.write "go infinite\n"
top.flush
echo fromp.readLine
while hasData(process):
try:
echo fromp.readLine()
fromp.flush()
except OverflowDefect:
echo "EXCEPTION 2 OverflowDefect"
except ValueError:
echo "EXCEPTION 2 ValueError"
except IOError:
echo "EXCEPTION 2 IOError"
except CatchableError:
echo "EXCEPTION 2 CatchableError"
discard process.waitForExit
process.close
eg. using the "stockfish" binary as chess engine (open source and default on most Linux distros), i feed 3 standard command (chess UCI protocol) and i get the output as expected :
Stockfish 14.1 by the Stockfish developers (see AUTHORS file)
id name Stockfish 14.1
id author the Stockfish developers (see AUTHORS file)
option name Debug Log File type string default
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 33554432
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 10 min 0 max 5000
option name Slow Mover type spin default 100 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name UCI_ShowWDL type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
option name Use NNUE type check default true
option name EvalFile type string default nn-13406b1dcbe0.nnue
uciok
info string NNUE evaluation using nn-13406b1dcbe0.nnue enabled
info depth 1 seldepth 1 multipv 1 score cp 38 nodes 20 nps 20000 tbhits 0 time 1 pv d2d4
info depth 2 seldepth 2 multipv 1 score cp 82 nodes 51 nps 51000 tbhits 0 time 1 pv e2e4 a7a6
info depth 3 seldepth 3 multipv 1 score cp 55 nodes 154 nps 154000 tbhits 0 time 1 pv e2e4 c7c6 d2d4
info depth 4 seldepth 4 multipv 1 score cp 22 nodes 807 nps 269000 tbhits 0 time 3 pv g1f3 d7d5 d2d4 g8f6
[...]
the code might not be perfect, you can give comments ..
discard process.waitForExit indeed waits for program to exit, but stockfish doesn't exit after the last command, it merely waits for another.
waitForExit() is useful for programs that fire once and exit when they're finished, like for example sleep or grep.
You can test how stockfish behaves, by simply opening program in the terminal and typing in commands manually. It won't exit after go movetime 10000<Enter>, it'll wait indefinitely for the next command. If you want to quit stockfish - give it a command to quit.
If you like, see https://github.com/crashappsec/cap10 which is built on top of a flexible single-threaded IO multiplexing library, which includes a subprocess API that's even more flexible than Python's subprocess library. The one-shot call (below) gives you a much more flexible interface where you can make orthogonal choices to capture any stream, use a pseudo-terminal, allow people to interact or not, choose to show output or not, etc.
The builder-style API is still pretty simple, and allows for things like callbacks when file descriptors have data (in addition to capture, passthrough, etc). The cap10 code itself is built on this API and is pretty tiny for what it is.
proc runCommand*(exe: string,
args: seq[string],
newStdin = "",
closeStdIn = false,
pty = false,
passthrough = SpIoNone,
passStderrToStdin = false,
capture = SpIoOutErr,
combineCapture = false,
timeoutUsec = 1000,
env: openarray[string] = [],
waitForExit = true): ExecOutput