Mirror of forum.nim-lang.org

4890 :: Struggling with writing an iterator

• jlhouchin (orginal) [2019-05-29T23:03:07+02:00] view original

  I am writing an iterator which calls walkDir and iterates over all of the files in the dir creating objects from those files. I have a proc which reads the file and creates a seq of the objects. It works fine. Amazingly so, on my machine Nim will read one of these files and create almost 400k objects in 40ms.

  I want an iterator which iterates over each of these files, serving each of its objects one at a time.

  iterator pricestream*(source: string, instrument: Instrument): Price =
    ## Iterator for all historic prices for the source, and instrument.
    var paths: seq[string]
    for file in walkDir(datapath / source / $instrument / "Prices"):
      paths.add(file.path)
    paths.sort(system.cmp[string])
    for path in paths:
      for price in pricesFromCsv(path):
        yield price
  
  # If I call like this:
  
  var stream = pricestream("oanda", EUR_USD)
  for p in stream:
    ...
  
  # I get this:
  # ntplayground.nim(46, 27) Error: attempting to call undeclared routine: 'pricestream'
  
  # If I call like this:
  var stream = pricestream
  for p in stream("oanda", EUR_USD):
    ...
  
  # I get this:
  # ntplayground.nim(47, 18) Error: type mismatch: got <Price>
  # but expected one of:
  # iterator items[T](a: set[T]): T
  # ...
  Run

  Any help in understanding how to write and call an iterator (this iterator) would be greatly appreciated.

---

• jyapayne (orginal) [2019-05-29T23:34:51+02:00] view original

  Looks like you just need a first class iterator. https://nim-lang.org/docs/manual.html#iterators-and-the-for-statement-first-class-iterators

---

• jlhouchin (orginal) [2019-05-29T23:56:47+02:00] view original

  I have read that before. I don't quite understand all of it in relation to how to write an iterator and call it.

  However, I went and put back the {.closure.} in my iterator and it works if I call like this:

  var stream = pricestream
  for p in stream("oanda", EUR_USD):
    ...
  Run

  What I don't understand is something like this:

  iterator walkDir*(dir: string; relative=false): tuple[kind: PathComponent, path: string] {.
    tags: [ReadDirEffect].} =
    ...
    when nimvm:
      for k, v in items(staticWalkDir(dir, relative)):
        yield (k, v)
    ...
  
  #And  walkDir  is called just like just like a proc   and it contains no {.closure.}
  
  
  for file in walkDir(datapath / source / $instrument / "Prices"):
      paths.add(file.path)
      ...
  Run

  Thanks for at least getting me working even if lack understanding.

  I don't understand why the intermediate step of assigning the iterator to a variable and then providing the variable with the parameters.

  Thanks again.

---

• jyapayne (orginal) [2019-05-30T16:51:30+02:00] view original

  The problem is that iterators default to being inline. This means the compiler transforms the iterator into a block of code that is executed only when it is in an iteration construct like a for loop.

  So in

  for file in walkDir(datapath / source / $instrument / "Prices"):
      paths.add(file.path)
      ...
  Run

  This would actually be transformed into (I believe) something like this:

  ...
    when nimvm:
      for k, v in items(staticWalkDir(datapath / source / $instrument / "Prices", relative)):
        let file: tuple[kind: PathComponent, path: string]  = (k, v)
        
        paths.add(file.path)
        ...
    ...
  Run

  So then the compiler isn't actually calling anything per se, but detecting when an iterator is used in a for loop and then making the transformation inline.

  When using the {.closure.} pragma, the compiler will make the iterator an actual function call (or something similar), making the iterator symbol a so called "first class citizen" that can be used how you are wanting to in your second example.

  To get it to work in your first example, you can return a closure iterator from a pricestream proc like so:

  import os
  import algorithm
  
  type
    Instrument = object
      name: string
    Price = float
  
  proc `$`(ins: Instrument): string =
    ins.name
  
  var datapath = "."
  
  proc pricesFromCsv(path: string): seq[Price] =
    # Just hardcoded for example
    return @[1.0, 2.0, 3.0, 4.0, 5.0]
  
  proc pricestream*(source: string, instrument: Instrument): iterator(): Price =
    # When returning an iterator type, {.closure.} is inferred.
    return iterator(): Price =
      ## Iterator for all historic prices for the source, and instrument.
      var paths: seq[string]
      
      for file in walkDir(datapath / source / $instrument / "Prices"):
        paths.add(file.path)
      
      paths.sort(system.cmp[string])
      
      for path in paths:
        for price in pricesFromCsv(path):
          yield price
  
  
  var EUR_USD = Instrument(name: "EUR_USD")
  
  var stream = pricestream("oanda", EUR_USD)
  for p in stream(): # brackets are needed here to invoke the iterator
    echo p
  Run

---