Mirror of forum.nim-lang.org

9603 :: memory footprint of parsejson

• setop (orginal) [2022-11-13T23:43:21+01:00] view original

  Hi,

  Newbie to Nim, happy to be there.

  I've been caught by the "Python but typed with meta-programming and compiling to standalone exe" kinda.

  I give a try to Nim to process large json files.

  I the doc of parsejson I saw 'This module implements a JSON parser. It is used and exported by the JSON standard library module, but can also be used in its own right.' Nice.

  In run the following program on an 71MB input file, basically an array of 35k objects.

  import streams
  import parsejson
  
  var strm = newFileStream(system.stdin)
  
  var p = JsonParser()
  p.open(strm, "stdin")
  
  while true:
      p.next()
      case p.tok
      of tkEof:
          echo "end"
          break
      else: discard
  Run

  Sorry for your bleeding eyes, it is my very first program in Nim.

  It is pretty fast : 1.5s ; faster than jq - a least for the moment since it does nothing else than reading input.

  But take a lot of RAM : 250MB. I was hopping a streaming parser to take a fix small amount of memory.

  What did I miss ?

  Thanks for your help

---

• Araq (orginal) [2022-11-14T07:44:47+01:00] view original

  Report it on github please. It shouldn't use 250MB but then I don't know how you measure it.

---

• jasonfi (orginal) [2022-11-14T08:46:11+01:00] view original

  The jsony lib is faster and should use less memory: https://github.com/treeform/jsony

---

• setop (orginal) [2022-11-14T12:13:51+01:00] view original

  @Araq

  Report it on github please

  I'll fill a proper in issue on GH if useful.

  but then I don't know how you measure it

  In short, I use /usr/bin/time -f '%M %e' ./exe < input.json. It gives

  • M: Maximum resident set size of the process during its lifetime, in Kilobytes.
  • e: Elapsed real (wall clock) time used by the process, in seconds.

  For a small file I get 1.5MB, for a multi-MB file, I get multi-MB consumed. It is pretty linear : about 3x the size of the input.

  I also tested with no mm, I get a constant overhead of 20%.

---