Mirror of forum.nim-lang.org

13954 :: jsoncanon - canonical JSON (Nim, python, pandas accessor)

• luciano (orginal) [2026-06-01T19:44:30+02:00] view original

  

  https://github.com/lucianofedericopereira/json-canon

  Lint, re-encode, and much more

  Take JSON produced by anything — CPython json, pandas to_json, Nim std/json, JS, hand-edited files with comments and trailing commas — and emit one deterministic byte-stream. Then cmp, sha256, and git diff on the output actually mean something, no matter who produced the input.

  Two independent implementations (Python and Nim) that are byte-identical by construction.

  MIT

---

• nimzwo (orginal) [2026-06-17T16:11:27+02:00] view original

  Just out of curiosity: Why doesn't the project implement RFC 8785: JSON Canonicalization Scheme (JCS) <https://www.rfc-editor.org/info/rfc8785/>?

---

• luciano (orginal) [2026-06-21T21:21:48+02:00] view original

  The short answer: big ints.

  I treat numbers as string tokens so you can handle 30-digit integers without losing precision (avoiding floating-point errors). 4.50e1 to 45 and 4.0 to 4 directly via string manipulation.

  RFC 8785 explicitly mandates the ECMA-262 (JavaScript) float-to-string rules, which are bounded by IEEE 754 double-precision variables.

  Under RFC 8785, integers larger than 2^53 may lose precision or turn into exponential notation.

  I guess the best way to add support is some kind of RFC 8785 canonicalizer, like what jcs does, in python, If someone is interested in RFC8785 or have improvement ideas I will try to implement them. Thanks

---

• luciano (orginal) [2026-06-22T19:30:53+02:00] view original

  Added RFC 8785 / JCS mode (--jcs). Numbers are serialized by the ECMAScript Number.prototype.toString algorithm and object keys are sorted by UTF-16 code unit. Includes a from-scratch pure-Nim/pure-Python port of Ulf Adams' Ryu d2s (shortest round-tripping binary64), with its 128-bit power-of-five tables generated by tools/gen_ryu_tables.py. Validated byte-for-byte against node's String(x) over 10M+ random doubles (tools/fuzz_ryu.py).

  JCS precision warnings. In --jcs mode, every number whose value changes when routed through binary64 (e.g. 9007199254740993 → …992, or overflow) is reported with its JSON path. --log FILE writes the report; -q/--quiet suppresses stderr warnings but requires --log (a precision change is never silently lost).

  Full JSON5 unquoted keys (SPEC §1.2). Unicode identifier keys ({café:1, π:2, 中文:3}) and uXXXX-escaped keys. The ID_Start/ID_Continue ranges are generated once by tools/gen_id_tables.py and embedded in both implementations so classification can't drift between Unicode databases.

  --ijson — I-JSON (RFC 7493) conformance report: numbers outside the binary64 safe-integer range or precision, duplicate keys, unpaired surrogates, plus any non-RFC-8259 lexical feature. Exit 1 if non-conformant.

  --sha256 — emit the SHA-256 hex digest of the canonical bytes. Ships a dependency-free pure-Nim SHA-256 (nim/src/sha256.nim); matches Python's hashlib.

  --diff FILE — structural diff of two inputs after canonicalization (~/+/- lines by JSON path), so whitespace, key order, and number spelling are normalized away before comparing.

  --json-seq — parse a stream of top-level values: RFC 7464 JSON Text Sequences (RS-framed), whitespace-separated, and directly concatenated, via one lenient reader.

  CBOR (--from cbor / --to cbor). RFC 8949 Core Deterministic Encoding (§4.2): shortest integer/float (16/32/64) encoding, bignum tags for integers beyond 64-bit, and §4.2.1 map-key ordering. The Nim half-precision codec is verified against Python over all 65 536 bit patterns; encode is byte-identical between implementations.

  --geojson — GeoJSON (RFC 7946) conformance report: type/member/coordinate /ring-closure checks, polygon right-hand-rule winding (§3.1.6), bbox shape, and the removed crs member. Geometries nested in Features/collections are checked recursively.

  --validate FILE — JSON Type Definition (RFC 8927) schema validation. All eight forms (empty, ref, type, enum, elements, properties, values, discriminator) plus nullable and root definitions; errors reported as the RFC 8927 §3.3.2 instance/schema JSON Pointers.

  --from jdata — JData / NeuroJSON: expand _ArrayType_/_ArraySize_/ _ArrayData_ annotated objects into nested JSON arrays (row-major), then canonicalize. Uncompressed arrays only.

  --cddl FILE — CDDL (RFC 8610) schema validation (practical subset: type rules, choices, primitives, literals, ranges, arrays/maps with occurrences, refs).

  --schema FILE — JSON Schema validation, dialect autodetected from $schema (Draft 2020-12 default, Draft-07 supported). Core/applicator/ validation keywords, $ref into $defs/definitions, allOf/anyOf/oneOf/ not, if/then/else, boolean schemas. pattern/patternProperties use a self-contained regex subset (nim/src/regexlite.nim + python/_regex.py), validated against Python re and Nim==Python over 40k+ pattern/text pairs; format is annotation-only.

  --force — best-effort salvage of malformed input: drop bad object members / array elements (resync to the next delimiter), close unterminated containers, drop trailing garbage, top-level junk → null; recovery report on stderr. Deterministic and byte-identical across implementations.

  JSON Schema --format — opt-in assertion of the format vocabulary (date-time/date/time, email, hostname, ipv4/ipv6, uri, uuid, json-pointer, regex) via deterministic hand-written checks in both languages.

  MessagePack --from/--to msgpack — deterministic encoding (smallest int/float/header; map keys ordered by encoded bytes) + decode; mirrors CBOR.

  --pointer EXPR — extract a sub-value by JSON Pointer (RFC 6901).

  --patch FILE (JSON Patch, RFC 6902) and --merge-patch FILE (JSON Merge Patch, RFC 7386) applied before canonicalizing.

  JSON Schema $anchor (#name refs) and unevaluatedProperties/ unevaluatedItems with annotation collection across allOf / anyOf / oneOf / if-then-else / $ref.

  Housekeeping git init + .github/workflows/ci.yml running the full gate (mypy, pyright, nim check, both suites, parity, Ryu fuzz vs node) on push/PR. tools/check.sh now builds the CLI itself (no longer relies on a pre-built binary) and spans ~58 parity rows. Added CHANGELOG and a Nim-artifacts section to .gitignore. Documented out of scope: remote $ref/$dynamicRef, and full single-document streaming (NDJSON/json-seq already stream per record). Notes

  --jcs and --to cbor deliberately route numbers through IEEE-754 binary64, departing from the project's default lossless decimal-string number engine. This is opt-in and, for --jcs, loudly warned about (see SPEC §5.1).

  tools/parity.sh now spans ~46 byte-parity rows; tools/check.sh runs the full gate (mypy --strict, pyright, nim check, both test suites, parity).

---