I made a small program using the 'coffeepots/odbc' and 'achesak/nim-csv' packages.
The program works well and builds my csv files from the database views as intended, but I can't compile it in release mode?
With a regular build, it runs. In release, I get this:
fatal.nim(49) sysFatal Error: unhandled exception: index out of bounds, the container is empty [IndexDefect]
How do I even find out where the problem is? I tried it using the regular compiler and vcc and get the same error.
Consider this program named bug1.nim:
import os, strutils
let n = [1, 2]
echo n[paramStr(1).parseInt]
And some runs of it (hiding some options that only make the output less noisy):
$ nim r bug1 4
bug1.nim(3) bug1
fatal.nim(49) sysFatal
Error: unhandled exception: index 4 not in 0 .. 1 [IndexDefect]
$ nim -d:release r bug1 4
fatal.nim(49) sysFatal
Error: unhandled exception: index 4 not in 0 .. 1 [IndexDefect]
The normal build pinpoints the line in the program that leads to the bug. The release build doesn't do this.
So how can we get that information into the release build?
Answer: add --stackTrace:on --lineTrace:on to the build options:
$ nim -d:release --stackTrace:on --lineTrace:on r bug1 4
bug1.nim(3) bug1
fatal.nim(49) sysFatal
Hopefully your bug still shows up in the presence of these features.
With a regular build, it runs.
Of course we may wonder why a program can run well in debug mode but crash when compiled with -d:release.
One reason for your database application can be timing issues. "index out of bounds, the container is empty" message may indicate that your Nim code in release mode is so fast that it tries to access data which is not yet available.
As @moigagoo requested, I am posting the whole program. It's pretty straightforward. As the first nim programming I have tried, I am happy that it works at all (even if only in a debugging build).
import odbc, csv, threadpool
setMaxPoolSize(8)
type DataFile = object
name: string
exportable: bool
filename: string
proc getConnection*(): ODBCConnection =
var con = newODBCConnection()
con.host = r"dbhostname"
con.driver = "ODBC Driver 17 for SQL Server"
con.userName = "user"
con.password = "Passw0rd"
con.database = "dev"
con.integratedSecurity = false
con.reporting.level = rlErrorsAndInfo
con.reporting.destinations = {rdEcho}
return con
proc getExportList*(): seq[DataFile] =
var con = getConnection()
if not con.connect():
echo "Couldn't connect to db"
let r = con.executeFetch("SELECT * FROM data_to_extract")
for i in 0 ..< r.len:
result.add(
DataFile(
name: r[i][0].strVal,
exportable: r.data(1, i).boolVal,
filename: r[i][2].strVal
)
)
con.disconnect()
proc writeSqlDataSet*(data: DataFile): void =
var con = getConnection()
if not con.connect():
echo "Couldn't connect to db"
let r = con.executeFetch("SELECT * FROM " & data.name)
var ds: seq[seq[string]] = @[newSeq[string](r.colFields.len)] # need this structure for csv writing
#Write the header names
var idx: int
for field in r.fields:
ds[0][idx] = field.fieldname
idx.inc
#Write the data rows
for i in 1 ..< r.len:
ds.add(newSeq[string](r[i].len))
for j in 0 ..< r[i].len:
ds[i][j] = r[i][j].asString
#echo r[i][j].asString
# For now, just load the entire dataset in memory rather than page the SQL server query but re-eval if perf problem
if ds.len <= 4000:
discard writeAll(data.filename & ".csv", ds)
else:
var i = 0
var j = 1
while ds.len - i > 4000:
discard writeAll(data.filename & $j & ".csv", ds[i .. i+4000])
i += 4000
j.inc
#write out remainder
if i != ds.len:
discard writeAll(data.filename & $j & ".csv", ds[i ..< ds.len])
let list = getExportList()
for extract in list:
if extract.exportable: spawn writeSqlDataSet(extract)
sync()
All calls are blocking, so it shouldn't be possible for me to use something before it's there, right?
name: r[i][0].strVal,
exportable: r.data(1, i).boolVal,
filename: r[i][2].strVal
You do access r[i][0] and similar elements without any checks if that elements exists. Generally debugging starts there is case of errors, and if I remember correctly the error message was matching to such an invalid acess. I can not test myself, I think I have not installed odbc and I have no experience with database access.
You do access r[i][0] and similar elements without any checks if that elements exists.
In theory, yes, though since I control the schema in the db, I know that the elements are there, even if that's a little sloppy.
But either way, what about this point would cause this to be fine when compiled with nim c but not when the -d:release is added? I am trying to understand the difference, since I have very little experience with it.
I told you already that it may be timing issues. Unfortunately I do know nothing about odbc. I just tested if I can import it, but I can not. And nimble search odbc seems to find it not.
Timing issues can occur when threading or async is active.
Debugging is generally testing for things for which you strongly assume that they are OK. You can split complex statements into simpler ones and print all the values with echo. Have you tried the "nim -d:release --stackTrace:on --lineTrace:on" suggestion from Mr jrfondren above? Well maybe you will find someone who will do the debugging for you. For me unfortunately that never works, I have to do it myself always. In a few forums using a nice female user name helps, but I never tried that. :-)
Hi @protium, I'm the author of odbc. As others have said, I'd assume something like r[i][0] or similar is what's failing. For example, if the database can't retrieve the data due to any number of reasons (eg; security) then you'd get an index error when trying to read the results, as they would be empty.
Do you get the output you expect when you echo the executeFetch results (or at least the len of the results) whilst running in release mode?
I did try doing a simple test of your code and didn't get any errors. I set up a little table with three columns, the name of another table in the first column, and a filename in the last, and when run with -d:release and --nilseqs:on (required for the csv lib) it output a CSV with data from the table referenced in the first column.
Thanks for your feedback, everyone. This one is proving very vexing. I added additional defensive coding to check the row counts before proceeding. It doesn't make a difference.
If I try @jrfondren 's suggestion to run compile with nim -d:release --stackTrace:on --lineTrace:on c src/BuildEcDataLoadFiles.nim, the entire program runs lickety-split with no problems. That's awesome. But if I remove the stackTrace and lineTrace options, I get the same failure. The failure is instant. I don't even get the usual ODBC messages from leaving reporting on, so I am reasonably sure that it's not related to any of the later work with accessing SQL results. I don't even get to that point.
Really struggling to understand what semantics would be causing the change in outcomes if the stackTrace symbols are left in.
How curious. Does the compiler finish and produce an executable that fails, or is the issue during compilation?
What version of Nim are you using to compile and what's your compile target (C, CPP, ObjC, JS)? Since you mention vcc I'm assuming you're on Windows?
For reference I'm currently using Nim 1.4.2 on Windows and compiling to C seemed to work okay with -d:release. That makes it difficult to help directly as I can't reproduce the issue.
When you run the program in it's problem state, does it still fail in the same way if you add a quit(0) or echo statement right at the start, or just after imports? Just wondering if it's possible to isolate where the issue is occurring in the program, or if it's during in an import, or perhaps even before that in a pre set up phase.