Mirror of forum.nim-lang.org

7876 :: read chunks from file or stream with custom delimiter

• cmc (orginal) [2021-04-28T11:05:50+02:00] view original

  I'm looking for a way to do the equivalent of

  for token in "fooXbarXfuzXbuz".split($X):
    echo token
  Run

  but with a file or a stream. It's conceptually similar to readLine, I want to read chunks from a stream until a specified delimiter occurs without reading everything into memory as with the example above.

  I know I could write my own iterator based on readData, but is a short and sweet way to do it, preferably but not necessarily within stdlib? Thanks!

---

• cblake (orginal) [2021-04-28T11:25:41+02:00] view original

  If you are on POSIX what you want is getdelim which can work directly upon a Nim File. As an example, I wrap this in cligen <https://github.com/c-blake/cligen/cligen/osUt.nim.

---

• cblake (orginal) [2021-04-28T11:29:39+02:00] view original

  Oh, and if it is always a file (as opposed to a run-time-decided file|stream) then the stdlib memfiles module (e.g. iterator memSlices) can also help you which uses memchr which is usually fast/vectorized in libc.

---

• cblake (orginal) [2021-04-28T11:33:41+02:00] view original

  You can, of course, port something like this C code to Nim, but it will be slower due to the character-at-a-time processing. A good getdelim for a buffered IO library should be able to just do a memchr/equivalent on each read buffer, skipping forward at vectorized strides.

---

• cmc (orginal) [2021-04-28T22:17:37+02:00] view original

  Thanks for your very comprehensive answer!

  Of course it would be ideal to have a pure nim getdelim, but...

  iterator memSlices

  This will do nicely!!!

---