Hi,
I am relatively new to nim and having some problems troubleshooting a simple program that works with nim v0.19.6, but fails if I upgrade to a more recent version. I have isolated the section of code that fails, but I am not sure I am doing something wrong (more than likely) or if there is another issue.
My program reads a gzipped file using newGZFileStream then parses it using the parsecsv module. With v0.19.6 this runs as expected, however with v1.0.0 it compiles fine but I get the following runtime error (full trace below):
Error: unhandled exception: /Users/bars/.choosenim/toolchains/nim-1.0.0/lib/pure/lexbase.nim(74, 13) `s < L.buf.len` [AssertionError]
I tried a few different input files and found that this error only occurs if the input gzip file is larger than some (unknown) size. I also attempted to isolate if the issue was with the gzip decoding or the csv parsing, but I can only reproduce the fault when I use them in combination. I believe I am using uptodate versions of everything (?).
The following example will hopefully demonstrate what I am seeing
# Use v1.0.0
$ choosenim update stable
Updating stable
Info: Already up to date at version 1.0.0
$ nim -v
Nim Compiler Version 1.0.0 [MacOSX: amd64]
Compiled at 2019-09-23
Copyright (c) 2006-2019 by Andreas Rumpf
git hash: f7a8fc46c0012033917582eb740dc0343c093e35
active boot switches: -d:release
# Check I have the latest version of zip
$ nimble install zip
Downloading https://github.com/nim-lang/zip using git
Verifying dependencies for [email protected]
Installing [email protected]
Prompt: [email protected] already exists. Overwrite? [y/N]
Answer: y
Success: zip installed successfully.
Generate some test files. One short, one long.
> test_short.txt
for i in $(seq 200)
do
echo "this,is,a,test" >> test_short.txt
done
gzip -c test_short.txt > test_short.txt.gz
> test_long.txt
for i in $(seq 2000)
do
echo "this,is,a,test" >> test_long.txt
done
gzip -c test_long.txt > test_long.txt.gz
Test program - it is read_gz_csv that is causing the problem:
import zip/gzipfiles
import parsecsv
proc read_gz(file_path:string): int =
echo "Read gzip file and parse with readLine"
let strm = newGZFileStream(file_path)
var line:string
while strm.readLine(line):
result += 1
echo "Success! line count = ", result
proc read_txt_csv(file_path:string): int =
echo "Read plain text file and parse as CSV"
var csv_parser: CsvParser
csv_parser.open(newFileStream(file_path), file_path, separator = '\t')
while csv_parser.readRow():
result += 1
echo "Success! line count = ", result
proc read_gz_csv(file_path:string): int =
echo "Read gzip file and parse as CSV"
var csv_parser: CsvParser
csv_parser.open(newGZFileStream(file_path), file_path)
while csv_parser.readRow():
result += 1
echo "Success! line count = ", result
## read short input file
discard read_gz("./test_short.txt.gz")
discard read_txt_csv("./test_short.txt")
discard read_gz_csv("./test_short.txt.gz")
## read longer input file
discard read_gz("./test_long.txt.gz")
discard read_txt_csv("./test_long.txt")
discard read_gz_csv("./test_long.txt.gz")
All three procs run fine with the short input file, but read_gz_csv will fail with the long input file:
$ nim -v
Nim Compiler Version 1.0.0 [MacOSX: amd64]
Compiled at 2019-09-23
Copyright (c) 2006-2019 by Andreas Rumpf
git hash: f7a8fc46c0012033917582eb740dc0343c093e35
active boot switches: -d:release
$ nim c -r read_gzip.nim
Hint: used config file '/Users/bars/.choosenim/toolchains/nim-1.0.0/config/nim.cfg' [Conf]
Hint: system [Processing]
Hint: widestrs [Processing]
Hint: io [Processing]
Hint: read_gzip [Processing]
Hint: gzipfiles [Processing]
Hint: os [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: bitops [Processing]
Hint: macros [Processing]
Hint: algorithm [Processing]
Hint: unicode [Processing]
Hint: pathnorm [Processing]
Hint: osseps [Processing]
Hint: posix [Processing]
Hint: times [Processing]
Hint: options [Processing]
Hint: typetraits [Processing]
Hint: zlib [Processing]
Hint: streams [Processing]
Hint: parsecsv [Processing]
Hint: lexbase [Processing]
CC: read_gzip.nim
Hint: [Link]
Hint: operation successful (38633 lines compiled; 1.155 sec total; 63.348MiB peakmem; Debug Build) [SuccessX]
Hint: /Users/bars/tests/read_gzip [Exec]
Read gzip file and parse with readLine
Success! line count = 200
Read plain text file and parse as CSV
Success! line count = 200
Read gzip file and parse as CSV
Success! line count = 200
Read gzip file and parse with readLine
Success! line count = 2000
Read plain text file and parse as CSV
Success! line count = 2000
Read gzip file and parse as CSV
/Users/bars/tests/read_gzip.nim(42) read_gzip
/Users/bars/tests/read_gzip.nim(30) read_gz_csv
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/pure/parsecsv.nim(277) readRow
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/pure/lexbase.nim(119) handleLF
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/pure/lexbase.nim(97) fillBaseLexer
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/pure/lexbase.nim(74) fillBuffer
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/system/assertions.nim(27) failedAssertImpl
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/system/assertions.nim(20) raiseAssert
/Users/bars/.choosenim/toolchains/nim-1.0.0/lib/system/fatal.nim(39) sysFatal
Error: unhandled exception: /Users/bars/.choosenim/toolchains/nim-1.0.0/lib/pure/lexbase.nim(74, 13) `s < L.buf.len` [AssertionError]
Error: execution of an external program failed: '/Users/bars/tests/read_gzip '
If I run the same thing with v0.19.6 all three run fine with both input files:
$ choosenim 0.19.6
Switched to Nim 0.19.6
$ nim -v
Nim Compiler Version 0.19.6 [MacOSX: amd64]
Compiled at 2019-05-10
Copyright (c) 2006-2018 by Andreas Rumpf
git hash: c6f601d48ec81e0d6e052ba0d19a195b55cc68f2
active boot switches: -d:release
$ nim c -r read_gzip.nim
Hint: used config file '/Users/bars/.choosenim/toolchains/nim-0.19.6/config/nim.cfg' [Conf]
Hint: system [Processing]
Hint: read_gzip [Processing]
Hint: gzipfiles [Processing]
Hint: os [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: bitops [Processing]
Hint: algorithm [Processing]
Hint: unicode [Processing]
Hint: times [Processing]
Hint: options [Processing]
Hint: typetraits [Processing]
Hint: strformat [Processing]
Hint: macros [Processing]
Hint: posix [Processing]
Hint: ospaths [Processing]
Hint: zlib [Processing]
Hint: streams [Processing]
Hint: parsecsv [Processing]
Hint: lexbase [Processing]
CC: read_gzip
Hint: [Link]
Hint: operation successful (34871 lines compiled; 0.646 sec total; 48.246MiB peakmem; Debug Build) [SuccessX]
Hint: /Users/bars/tests/read_gzip [Exec]
Read gzip file and parse with readLine
Success! line count = 200
Read plain text file and parse as CSV
Success! line count = 200
Read gzip file and parse as CSV
Success! line count = 200
Read gzip file and parse with readLine
Success! line count = 2000
Read plain text file and parse as CSV
Success! line count = 2000
Read gzip file and parse as CSV
Success! line count = 2000
Thanks