Mirror of forum.nim-lang.org

2594 :: crashing closure iterator for file/directory searching object

• jordan (orginal) [2016-10-28T16:38:14+02:00] view original

  I try to apply the os.walkDir() method within a closure and got the feeling that I might have misunderstood something. In particular, running

  1  import os
  2
  3  proc walk(obj: string): iterator(): string =
  4    result = iterator(): string {.closure.} =
  5      for k,p in walkDir(obj):
  6        yield p
  7
  8  var g = walk(".")
  9  while not finished(g):
  10   echo "---> ", g()

  crashes with the error message

  ...
  Hint: operation successful (21553 lines compiled; 1.025 sec total; 34.004MiB; Debug Build) [SuccessX]
  Hint: /home/jordan/Desktop/Devel/deadpan/src/crash . [Exec]
  ---> ./nim.sh
  Traceback (most recent call last)
  crash.nim(10)            crash
  crash.nim(6)             :anonymous
  SIGSEGV: Illegal storage access. (Attempt to read from nil?)
  Error: execution of an external program failed: '/home/jordan/Desktop/Devel/deadpan/src/crash '

  I went through a session with the GDB but have not much experience with NIM, yet. It seems that it crashes in the GC. I tried to compile with different GCs (including none) without luck so it might crash for another reason? Interestingly, replacing the while loop 8..10 above with

  8  for w in walk(".")():
  9    echo "---> ", w
  10

  prints an infinite series of the same file (repeatedly restarting walk(".")()?), in my case

  ---> ./nim.sh
  ---> ./nim.sh
  ...

  while

  8  var g = walk(".")
  9  for w in g():
  10   echo "---> ", w

  crashes similar to the original while loop.

  Has somebody come across successfully encapsulating walkDir() into a closure?

---

• jordan (orginal) [2016-10-28T17:59:56+02:00] view original

  I found the problem in the file lib/pure/os,nim and it does not look right to me. Looking at the implementation there is the defer clause on line 974.

  972    var d = opendir(dir)
  973    if d != nil:
  974      defer: discard closedir(d)
  975      while true:
  976        var x = readdir(d)
  977        if x == nil: break
  978        var y = $x.d_name

  Adding a print statement to line 774

  972    var d = opendir(dir)
  973    if d != nil:
  974      defer: echo "===>", repr(d); discard closedir(d) # <<<<<<<<<<<<<<<
  975      while true:
  976        var x = readdir(d)
  977        if x == nil: break
  978        var y = $x.d_name

  I get the output

  Hint: operation successful (21553 lines compiled; 1.095 sec total; 34.004MiB; Debug Build) [SuccessX]
  Hint: /home/jordan/Desktop/Devel/deadpan/src/crash  [Exec]
  ===>ref 0x559091cfdc70 --> []
  
  ---> ./nim.sh
  Traceback (most recent call last)
  crash.nim(10)            crash
  crash.nim(6)             :anonymous
  SIGSEGV: Illegal storage access. (Attempt to read from nil?)
  Error: execution of an external program failed: '/home/jordan/Desktop/Devel/deadpan/src/crash '

  which looks like an early invocation of the defer statement. Commenting out line 974 I get what I was looking for originally:

  ...
  Hint: /home/jordan/Desktop/Devel/deadpan/src/crash  [Exec]
  ---> ./nim.sh
  ---> ./version
  ---> ./Makefile
  ---> ./Makefile.in
  ...
  ---> ./crash.nim
  ---> ./main.nim
  ---> ./version.ndb
  ---> ./version.nim~
  ---> nil

  But of course, defer is not called at all, now. Can somebody advise me how to wrap this defer issue? At least it looks like more serious than originally anticipated. When I tested os.walkDir() as simple statement without closure it seemed to work.

---

• Dippo (orginal) [2016-10-29T12:32:01+02:00] view original

  I am trying to understand your code, because there are 3 options: a) you want a file listing; b) you want a folder listing; c) you want A and B. In every folder (with the exception of the root folder) two files exists that are actual are not files, that is "." and "..". So far i can tell, you are trying to acess these objects also. So you must add a extra criteria to only access files or folders (or both).

---

• jordan (orginal) [2016-10-29T13:28:09+02:00] view original

  At this stage I only used a minimally wrapped os.walkDir() for reproducing the problem.

  What I really need is os.walkDirRec() which uses os.walkDir() recursively. It works similar to command-line find.

  As background information I can tell that this an opportunistic forensic search tool I wrote in the past where every data object (eg. ZIP, NTFS, ESE, TAR file) is sort of a directory. In C I had no closures so I fell back to a more driver like architecture not unlike the Posix TLI.

  Anyway, I believe I found the problem which has nothing to do with the data objects returned by the operator. It looks like the defer mechanism is triggered too early. Maybe there is some caching in the background I am not aware of which holds all the rest of the iterator results already. Then the defer mechanism works correctly. But then again the cache was flushed too early before all values had been yielded.

---