I'm currently working on loading glTF files. Part of this spec is parsing a json formatted 'chunk' (their term). I have created a repo with the simplest compilable example that demonstrates the issue. When I run the code as it is currently in the repo it runs successfully. If I uncomment lines 8,9,10 and comment line 11:
var data = newString(bytesCount)
for i in 0 .. bytesCount:
data.add(stream.readChar)
#var data = """{"accessors":[{"bufferV.....I receive the error json parsing error: input(1, 0) Error: { expected.
The json string is the same in the var data = """"... line as in the file.
Is this a problem with the json parser or with me? :)
Never heard of Valgrind before. So a quick download and perusal of the 'Quick Start' guide and here's the output:
colin@colin-ThinkPad-T14:~/Documents/devel/nim/jsonerr$ valgrind --leak-check=yes --leak-check=full --show-leak-kinds=all src/main
==12563== Memcheck, a memory error detector
==12563== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12563== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12563== Command: src/main
==12563==
json parsing error: input(1, 0) Error: { expected
==12563==
==12563== HEAP SUMMARY:
==12563== in use at exit: 16,856 bytes in 2 blocks
==12563== total heap usage: 19 allocs, 17 frees, 44,637 bytes allocated
==12563==
==12563== 472 bytes in 1 blocks are still reachable in loss record 1 of 2
==12563== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12563== by 0x490D92D: __fopen_internal (iofopen.c:65)
==12563== by 0x490D92D: fopen@@GLIBC_2.2.5 (iofopen.c:86)
==12563== by 0x10C944: open__stdZsyncio_u414 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x124C42: newFileStream__pureZstreams_u1210 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12F097: loadFile__main_u4 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12F6D2: NimMainModule (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12F5F6: NimMainInner (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12F60B: NimMain (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12F649: main (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563==
==12563== 16,384 bytes in 1 blocks are still reachable in loss record 2 of 2
==12563== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12563== by 0x10E3D4: allocImpl__system_u1747 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x10E3FA: allocSharedImpl (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x10E774: init__system_u3378 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x110B90: registerCycle__system_u3503 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x110D82: rememberCycle__system_u3543 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x123C07: nimDecRefIsLastCyclicDyn (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x125334: eqdestroy___pureZstreams_u725 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x125BC2: ssReadDataStr__pureZstreams_u855 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12510D: readDataStr__pureZstreams_u84 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x126BD6: fillBuffer__pureZlexbase_u16 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563== by 0x12771F: open__pureZlexbase_u144 (in /home/colin/Documents/devel/nim/jsonerr/src/main)
==12563==
==12563== LEAK SUMMARY:
==12563== definitely lost: 0 bytes in 0 blocks
==12563== indirectly lost: 0 bytes in 0 blocks
==12563== possibly lost: 0 bytes in 0 blocks
==12563== still reachable: 16,856 bytes in 2 blocks
==12563== suppressed: 0 bytes in 0 blocks
==12563==
==12563== For lists of detected and suppressed errors, rerun with: -s
==12563== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)If there's something else you want me to try just yell out. In the meantime I'll continue troubleshooting/debugging.
The problem is not with the json parsing but with newString. If I declare:
var str = newString(10)
str.add('x') # x will be placed at index 11
# however if I do:
str[0] = 'x' # x will be placed at index 0This behaviour seems counter to the documentation
How so? It tells you to use s[i] = ..., it doesn't tell you to use add.
"Returns a new string of length len. One needs to fill the string character after character with the index operator s[i].
This procedure exists only for optimization purposes; the same effect can be achieved with the & operator or with add."
Guess it's just the way I interpret that sentence.
This all has to do with how you read data into the string. Your code from above:
var data = newString(bytesCount)
for i in 0 .. bytesCount:
data.add(stream.readChar)
#var data = """{"accessors":[{"bufferV.....This creates a string data filled with bytesCount zero bytes. You then iterate through the bytesCount and _adds character by character to the string. This means that the end result is a string filled with bytesCount zero bytes followed by bytesCount of data. This is why the JSON parser complains, it expects a { but finds a null byte. In order to fix this you can use newStringOfCap which creates a new string with a _capacity of bytesCount, but a length of 0. This means that add will increase the length, but it won't cause the string to reallocate because the string is already long enough.
A better option would probably be to use readStr instead of readChar though. This would allow the stream implementation to be more efficient. The above code could then be simplified to:
let data = stream.readStr(bytesCount)That sentence means
var s = newString(5)
for i in 0..4:
s[i] = 'a'and
var s = ""
for i in 0..4:
s.add('a')result in the same string, but that the former is more efficient since it only has to allocate once.