Mirror of forum.nim-lang.org

4651 :: Newbie question about reading asynchronous text file line by line

• mrhdias (orginal) [2019-02-11T20:07:40+01:00] view original

  I'm trying to read a text file asynchronous line by line but the following code fails to compile

  import asyncdispatch, asyncfile, os
  
  proc main() {.async.} =
    
    var filename = getTempDir() / "test.txt"
    var file = openAsync(filename, fmWrite)
    await file.write("test1\ntest2")
    file.close()
    
    var file = openAsync(filename, fmRead)
    while true:
        let line = await file.readLine()
        if line.len == 0: break
        echo line
    
    file.close()
    removeFile(filename)
  
  waitFor main()
  Run

  Error messages

  Exception message: index out of bounds
  Exception type: [IndexError]
  Error: execution of an external program failed: '/home/hdias/Documents/Nim-Cookbook/async_openfile '

  In Nim is possible make same thing like this with templates?

  for var line = await file.readLines():
    echo line
  Run

  // Javascript
  for await (const line of readLines(filePath)) {
    console.log(line)
  }
  Run

  https://medium.com/@wietsevenema/node-js-using-for-await-to-read-lines-from-a-file-ead1f4dd8c6f

---

• nepeckman (orginal) [2019-02-11T20:37:04+01:00] view original

  The code doesn't compile as presented because of the redefinition of file. If you change the second invocation of openAsync to file = openAsync(filename, fmRead) the code will compile. The index error you get from running the code is another matter though. I think its a potential bug in the Nim standard library, but someone who knows more than me should verify that. Here's what I do know:

  This is the implementation of readLine:

  proc readLine*(f: AsyncFile): Future[string] {.async.} =
    ## Reads a single line from the specified file asynchronously.
    result = ""
    while true:
      var c = await read(f, 1)
      if c[0] == '\c':
        c = await read(f, 1)
        break
      if c[0] == '\L' or c == "":
        break
      else:
  result.add(c)
  Run

  The index error comes from accessing the first element of c when c is the empty string. The documentation from read says: "If the file pointer is past the end of the file then an empty string is returned." So I feel that the fact that readLine errors on read returning the empty string, means that this is a bug. I added an extra check to make sure c isn't empty in my local version of Nim, and the program now runs correctly.

  Again, I'm not a Nim dev, so I'm not really sure if this is a bug or not.

---

• mrhdias (orginal) [2019-02-11T21:22:56+01:00] view original

  Sorry the second the redefinition of file in invocation of openAsync it's my mistake.

  This compile but I get the same error:

  import asyncdispatch, asyncfile, os
  
  proc main() {.async.} =
    var filename = "test.txt"
    var file = openAsync(filename, fmRead)
    while true:
      let line = await file.readLine()
      if line.len == 0: break
      echo line
    
    file.close()
  
  waitFor main()
  Run

  Exception message: index out of bounds
  Exception type: [IndexError]
  Error: execution of an external program failed: '/home/hdias/Downloads/tmp/test '

---

• tim_st (orginal) [2019-02-11T22:05:04+01:00] view original

  I didn't run your code but had a very quick look at the implmentation of asyncfile.readLine to find out what can raise an IndexError. This line if c[0] == '\L' or c == "": should be checked before the other and it should be implemented like this: if c.len == 0 or c[0] == '\L': I assume this fixes IndexError.

---

• mrhdias (orginal) [2019-02-12T16:49:28+01:00] view original

  I have created an issue 10641

---

• moerm (orginal) [2019-02-12T17:49:27+01:00] view original

  The problem is that there seems to be no check for EOF.

  This here works:

  import asyncdispatch, asyncfile, os
  
  proc main() {.async.} =
    var filename = "test.txt"
    var file = openAsync(filename, fmRead)
    let fileSize = file.getFileSize()
    while file.getFilePos() < fileSize:
      let line = await file.readLine()
      if line.len == 0: break
      echo line
    
    file.close()
  
  waitFor main()
  Run

---