Mirror of forum.nim-lang.org

9023 :: Read output only while input is not asked

• Limits (orginal) [2022-03-19T15:53:06+01:00] view original

  a.nim:

  import std/osproc, std/streams
  
  let myprocess = startProcess("./b")
  var i = 0
  while true:
    var s: string = ""
    while myprocess.outputstream.readLine(s):
      echo s
    echo i
    i += 1
    myprocess.inputstream.write($i)
  Run

  b.nim

  import std/strutils
  
  echo "START B"
  
  while true:
    var s = stdin.readLine
    for i in 1..parseInt(s):
      echo "Hello " & $i
  Run

  a.nim starts a process with b.nim... It prints out b.nim's output and then gets stuck.

  Is there a way for a.nim to read output only while input is not required ?

  Regards,

  Limits

---

• hamidrb80 (orginal) [2022-03-21T09:05:53+01:00] view original

  I guess you should use Stream thingy in a.nim. take a look at: https://nim-lang.org/docs/osproc.html#outputStream%2CProcess

---

• ynfle (orginal) [2022-03-21T13:49:04+01:00] view original

  Your issue is that b is waiting to receive input and you are waiting to read a line until the stream closes. Try https://nim-lang.org/docs/streams.html#peekChar%2CStream

---

• demotomohiro (orginal) [2022-03-21T22:05:51+01:00] view original

  Both readXYZ and peekXYZ procs in stream module block until it get enough data to return. stdin.readLine in your b.nim blocks your code until it get a line.

  You can read all lines from b.nim and write stdin for b.nim if you know many lines b.nim writes:

  a1.nim:

  import std/[osproc, streams]
  
  let myprocess = startProcess("./b")
  var i = 0
  block mainLoop:
    while true:
      var s: string
      for j in 0..<max(i, 1):
        if myprocess.outputstream.readLine(s):
          echo s
        else:
          break mainLoop
      echo i
      i += 1
      myprocess.inputstream.writeLine($i)
      myprocess.inputstream.flush()
    myprocess.close()
  Run

  b.nim:

  import std/strutils
  
  echo "START B"
  
  for i in 0..10:
    var s = stdin.readLine
    for i in 1..parseInt(s):
      echo "Hello " & $i
  Run

  If you don't want to use the number of lines b.nim writes, you need to know if there are any data to read in outputStream without blocking.

  In posix, hasData(p: Process): bool in osproc module blocks until it get data to read. https://www.man7.org/linux/man-pages/man2/select.2.html

  In windows, it returns immediately in a single-threaded application but blocks in a multi-threaded application: https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-peeknamedpipe

  Following code can works without stucking, but it implements hasDataNoBlocking that works only on posix:

  a2.nim:

  import std/[osproc, streams, posix]
  
  proc hasDataNoBlocking(p: Process): bool =
    var rd: TFdSet
    
    FD_ZERO(rd)
    let m = max(0, int(p.outputHandle))
    FD_SET(cint(p.outputHandle), rd)
    
    var tv: Timeval
    
    result = int(select(cint(m+1), addr(rd), nil, nil, addr tv)) == 1
  
  let myprocess = startProcess("./b")
  var i = 0
  block mainLoop:
    while true:
      while true:
        var s: string
        if myprocess.outputstream.readLine(s):
          echo s
        else:
          break mainLoop
        if not myprocess.hasDataNoBlocking():
          break
      echo i
      i += 1
      myprocess.inputstream.writeLine($i)
      myprocess.inputstream.flush()
    myprocess.close()
  Run

  b.nim:

  import std/strutils
  
  echo "START B"
  
  for i in 0..10:
    var s = stdin.readLine
    for i in 1..parseInt(s):
      echo "Hello " & $i
  Run

  asynctools might better in this case: https://github.com/cheatfate/asynctools

---

• Limits (orginal) [2022-03-22T17:06:33+01:00] view original

  I've found a way to do it using timeouts a.nim:

  import std/osproc, std/streams, std/times, std/threadpool, std/os
  
  proc readLineTimeout(readStream: Stream, timeout: float): int =
    var
      FlowVar = spawn readStream.readLine
      t = cpuTime()
    while cpuTime() - t < timeout:
      if FlowVar.isReady():
        result += 1 #number of lines read
        FlowVar = spawn readStream.readLine
        t = cpuTime()
  
  let myprocess = startProcess("./b")
  for i in 1 .. 5:
    echo "START ", i
    myprocess.inputstream.writeLine($i)
    myprocess.inputstream.flush
    echo "LINES: ", readLineTimeout(myprocess.outputstream, 1)
  Run

  b.nim:

  import std/strutils, std/os
  
  while true:
    let n = parseInt(stdin.readLine)
    for i in 1..n:
      stdout.flushFile
      echo n, " : ", i
      stdout.flushFile
  Run

  Current output:

  START 1
  LINES: 1
  START 2
  LINES: 1
  START 3
  LINES: 2
  START 4
  LINES: 3
  START 5
  LINES: 4
  Run

  Expected Output:

  START 1
  LINES: 1
  START 2
  LINES: 2
  START 3
  LINES: 3
  START 4
  LINES: 4
  START 5
  LINES: 5
  Run

  It works fine on the first time (START 1) but is off by one on the rest. Any ideas ?

---

• demotomohiro (orginal) [2022-03-24T02:43:33+01:00] view original

  When readLineTimeout got timeout as it read output from b.nim and b.nim called stdin.readLine, readStream.readLine thread you spawned keep alive even when readLineTimeout returned and it can read output from b.nim. I that case, output read by that readStream.readLine call is not read by next readStream.readLine call.

  And also, readLineTimeout wait for 1 second to determine if b.nim wrote all output and it is waiting for new input is not efficient.

  It seems your b.nim doesn't know how many lines of input it needs to process. And a.nim doesn't know how many lines of output come from b.nim for an input. So waiting for a second is the only way to know if b.nim finished output for an given input?

  Adding a number of lines b.nim going to output to its output or adding an end of line block marker doesn't work for you? https://github.com/cheatfate/asynctools doesn't work?

---