Mirror of forum.nim-lang.org

13149 :: nim-json-serialization streaming arrays

• arnetheduck (orginal) [2025-06-27T10:45:25+02:00] view original

  We're looking into refreshing the n-j-s API to enable stream-writing of arrays - however, owing to a design choice long ago, we're stuck with two options:

  • create a somewhat ugly, but backwards-compatible API that preserves the semantics of writing writeValue overloads
  • create a nice and easy-to-use API but introduce a small compile-time backwards-compatibility break for some of said overloads

  The changes are best explained with an example of how they affect writing nested objects/arrays vs what it's like to access the inner stream:

  
  # nesting
  writer.writeObject:
    writer.writeMember("status", "ok")
    writer.writeArrayMember("data"):
      for i in 0 ..< 2:
        writer.writeObjectElement:
          writer.writeMember("id", i)
          writer.writeMember("name", "item" & $i)
  
  # custom serialization directly to stream:
  proc writeValue(w: var JsonWriter[StringyJson], value: seq[byte]) =
    w.stream.write('"')
    w.stream.write(toHex(value))
    w.stream.write('"')
  Run

  vs

  
  # nesting
  writer.writeObject:
    writer.writeMember("status", "ok")
    writer.writeName("data")
    writer.writeArray:
      for i in 0 ..< 2:
        writer.writeObject:
          writer.writeMember("id", i)
          writer.writeMember("name", "item" & $i)
  
  # direct stream access:
  proc writeValue(w: var JsonWriter[StringyJson], value: seq[byte]) =
    w.streamElement(s):
      s.write('"')
      s.write(toHex(value))
      s.write('"')
  Run

  Because the inner stream becomes protected in the latter version, we can catch any breakage at compile-time which is nice. Leaning towards this but happy to hear feedback and/or better ideas for a streaming JSON API.

  Since it would be boring to leave it at that, a bonus sneak peak for streaming via closure iterator (that could read from a file or database), once the above is settled:

  
  type Item = tuple[id: int, name: string]
  
  proc makeItems[Flavor](items: int): JsonArrayIterator[Item, Flavor] =
    iterator (writer: var JsonWriter[Flavor]): Item =
      for i in 0 ..< items:
        yield (i, "item" & $i)
  
  writer.writeValue((status: "ok", data: makeItems[writer.Flavor](2)))
  Run

---