Mirror of forum.nim-lang.org

10109 :: Which might be the best way to store complex data in MySQL

• nimnam (orginal) [2023-04-18T09:24:35+02:00] view original

  I wrote a library in TypeScript which stores patch sets (changes of values based on keys). So a document is a final states (key value array) and a changelog. I use JSON(P) because because PG and TS can easily handle it.

  Now Nim can handle JSON but you loose all typing guarantees. Using Nim datatypes means you need to teach PG how to work with it. That can be done cause you can load shared libraries.

  But what would be the most efficient way to store the data ?

  So when doing a select sum( my_nim_pg_code_get_value(data, "value") ) ?

  I have like two solutions in my mind:

  1. use binary data where pointer reference itself. Something like first 4 bytes point to start structure. Then continue treating binary data as if it was memory (but use memory pointers relative to start of the binary dump).
  2. just use a binary dump and unserialize (which might take more time and require more allocations/freeing)

  I noticed that my simple key value store is somewhat limited cause I need sets and change operations which are more complex such as 'append to list' for chat logs whatever.

  so my simple model would eat up much more memory than needed fast.

  The main question is: If NIM cannot type JSON (like TS). What's the equivalent of document storage ? Create objects and unserialize/serialize from JSON ?

---

• cmc (orginal) [2023-04-18T09:54:50+02:00] view original

  Personally I usually do stuff like that normalized to a database. Structured data you can mutate one-by-one is kind of the definition of what a relational database is supposed to do. I like tiny_sqlite a lot because it neatly maps type information without doing strings, and it's in-process- no seperate server to maintain.

  To have structured data in Nim, what you are looking for is indeed a Nim object. You can scratch JSON completely for internal use, I'd say making it that central to data storage is an artifact of the JavaScript ecosystem. But if you need a JSON equivalent of your object to send to an API or something, then you would serialize the object to JSON. I used an ad-hoc proc for that but there might be library meanwhile.

  I use my own LimDB, sometimes combined with the flatty serializer for using it with objects, when I just need to dump data somewhere that doesn't need to be mutated, such as for caching. It's also in-process and goes very well with tiny_sqlite.

---