My dll crashes when just displaying what is in a file (it should actually produce a hash table from that file).
Quick explanation of situation: It's a program that creates text based on conditions in a game (3rd party, so this is an addon). This will then go to a text-to-speech system for in game voice. I'm converting to a dll based system. Assuming the user can choose different vehicles, based on which one the gamer is using a different dll would be loaded. This could be 'bike.dll', 'car.dll', 'boat.dll', etc..
Every dll only has 2 procs for entry by the main program : initModule and speechModule, and based on what vehicle, different local procs to read the data files and generate the sentences to speak based on parameters. This 'text to speak' relies on files with possible sentences (so it'll be translatable into other languages).
A) Main program: Running thread to monitor game process based on output to 3 files (updated by the game)
It's in loading this text data file where it goes wrong. If the file is NOT loaded, the dll works fine, but cannot pick the words to say (in debug mode just outputs what it would pick).
That text data should go into a hash table, but even just iterating over the file contents makes the program crash. I've tried different memory models (normally --mm:ord) but with the same result. The proc loadSpeechData is in an imported nim source (import LoadData)
proc loadSpeechData*(project:string = "", folder: string, language: string = "en") =
debugEcho("---------------------------------------In loadSpeechData")he file
var
section, info, index: string
speechData: string = ""
sects: seq[string] # will hold the key in [0] and the data in [1]
parts: seq[string]
resourceFilename: string = "VoiceData.zip"
debugEcho(fmt("Loading speech data for {project:s} from {resourceFilename}\n"))
# read in the data file from the zipped data file (TODO : depending on language)
# Open the zip archive. This only reads the records metadata,
# it does not read the entire archive into memory.
try:
let reader = openZipArchive(resourceFilename)
try:
# Extract the file contents from the archive.
speechData = reader.extractFile("data-en.txt")
except:
logInfo = logInfo & fmt("- ERROR : could not load speech data\n")
debugEcho("- ERROR: Could not load speech data")
finally:
reader.close()
except:
debugEcho(fmt("- ERROR : could not find speech data file {resourceFilename}"))
debugEcho("DUMP speechData:")
debugEcho(" - size: ", (sizeof(speechData)))
debugEcho(" - length: ", (speechData.len))
debugEcho(" - lines: ", (speechData.countLines))
debugEcho(speechData)
# clear the pronounciation and project data
phonList.setLen(0)
corrList.setLen(0)
trigList.setLen(0)
textTable.clear
dataTable.clear
for d in speechData.splitLines():
inc countR
let dataLine = d.uncomment
debugEcho(fmt("READ {countR:4d}|{dataLine}"))
debugEcho(speechData) shows the whole file, but when iterating with d, (uncomment just strips // comments), it stops halfway the file. The traceback reports prepareDealloc and SIGSEGV: Illegal storage access. (Attempt to read from nil?) on the garbage collector, so I tried other memory models but with (nearly) the same result. Breaking my head over this for hours :-/. Strange the file shows up fine with echo, but when iterating over the lines it just crashes.
Here is the code (same code works fine in the pre-dll version, where it was statically linked. However because of the various 'vehicles' that statically linked version is becoming too big and won't allow plugin of new/updated 'vehicles'.
This is the output (removed lines or it would become too long):
DUMP speechData:
- size: 8
- length: 38264
- lines: 959
///////////////////////////////////////////////////////////////////////
// This file contains data for the speech system
... (some 950 more lines)
TXT_ALLZEROS:all zeros
TXT_TENSECONDS:Ten seconds
TXT_TARGET:target
READ 1|///////////////////////////////////////////////////////////////////////
READ 2|// This file contains data for the speech system
...
READ 455|=00000;Zero
READ 456|=_;
READ 457|=[;
Traceback (most recent call last)
C:\Users\Ivan\GFR\ReModuleG.nim(216) speechModule
C:\Users\Ivan\GFR\ReModuleG.nim(161) processPlayer
C:\Users\Ivan\GFR\ReLangData.nim(132) loadSpeechData
C:\Users\Ivan\AppData\Local\Programs\nim-1.4.8_x64\nim-1.4.8\lib\pure\strformat.nim(484) formatValue
C:\Users\Ivan\AppData\Local\Programs\nim-1.4.8_x64\nim-1.4.8\lib\pure\strformat.nim(411) formatInt
C:\Users\Ivan\AppData\Local\Programs\nim-1.4.8_x64\nim-1.4.8\lib\system\gc.nim(439) newObj
C:\Users\Ivan\AppData\Local\Programs\nim-1.4.8_x64\nim-1.4.8\lib\system\gc_common.nim(423) prepareDealloc
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Thank you both for the prompt reply. I'll try it out tonight after work.
PMunch, that article on your blog together with your answer on reddit from some years back https://www.reddit.com/r/nim/comments/ds9n7k/dll_dependency_on_nimrtldll/ makes things clear.
Indeed I was not using NimMain... nor nimrtl.dll
Well, I was very hopeful, but alas...
After a few hours at this (again) still no progress. It is a nim dll for a nim exe, so no c declarations that can/should go wrong.
Compiles: nim c -d:useNimRtl --app:lib --noMain -mm:orc ReModule
proc NimMain() {.importc.}
proc initModule(md: ModuleData): bool {.exportc, dynlib.} =
NimMain() # Be sure to call this! It will set up garbage collector and initialize any global memory
# first base data (paths to files/folders, language, etc..)
basedata = md
proc endModule() {.exportc, dynlib.} =
GC_FullCollect()
proc speechModule*(sd: SimData): SimData {.exportc, dynlib.} =
simdata = sd # copy to a global for this module
initTalk(simdata)
... more code
return
The calling program:
type
SpeechModuleProc = proc(sd: ReData.SimData): ReData.SimData {.nimcall}
InitModuleProc = proc(mdl: ModuleData): bool {.nimcall}
EndModuleProc = proc() {.nimcall}
var
libModule: LibHandle
initModuleAddr: pointer
initModule: InitModuleProc
speechModuleAddr: pointer
speechModule: SpeechModuleProc
endModuleAddr: pointer
endModule: EndModuleProc
moduleFile: string
moduleLoaded: string = ""
moduleAvailable: bool = false
proc loadModule(moduleName: string): bool =
result = false
# create path to module (working dir if development version)
moduleFile = "ReModule" & moduleName & ".dll"
# try loading the module
try:
libModule = loadLib(moduleFile)
result = true
except:
logI(fmt("ERROR: could not load module {moduleName}."))
result = false
# get address(es) of the proc(s) in the dynamic lib
var moduleErrors: string = ""
if libModule != nil:
initModuleAddr = libModule.symAddr("initModule")
speechModuleAddr = libModule.symAddr("speechModule")
endModuleAddr = libModule.symAddr("endModule")
if initModuleAddr != nil:
initModule = cast[InitModuleProc] (initModuleAddr)
else:
moduleErrors = moduleErrors & ("- initModule not found\n")
if speechModuleAddr != nil:
speechModule = cast[SpeechModuleProc] (speechModuleAddr)
else:
moduleErrors = moduleErrors & ("- speechModule not found\n")
if endModuleAddr != nil:
endModule = cast[EndModuleProc] (endModuleAddr)
else:
moduleErrors = moduleErrors & ("- endModule not found\n")
else:
moduleErrors = moduleErrors & ("- something wrong with module")
if moduleErrors > "":
logP(fmt("ERRORS: loading {moduleName} gave problem(s):\n") & moduleErrors)
result = false
if result:
moduleLoaded = moduleName
# get data needed for initialization of module
var mdl: ModuleData
mdl.lang = "en"
mdl.appname = APPNAME
mdl.appversion = VERSION
discard initModule(mdl) # initialize !
else:
moduleLoaded = ""
proc unloadModule(): bool =
if moduleLoaded != "":
endModule()
unloadLib(libModule)
proc projectModule(simd: ReData.SimData): ReData.SimData =
var
simdata = simd #: ReData.SimData
# not same lib module: unload old and load correct one
if moduleLoaded != simdata.mission.projectName:
discard unloadModule()
if loadModule(simdata.mission.projectName):
logP(fmt("Module {moduleLoaded} loaded successfully..."))
else:
logI(fmt("WARNING: Could not load the project {simdata.mission.projectName} module: {moduleFile}."))
if libModule != nil:
try:
simdata = speechModule(simdata)
except:
logP("ERROR: Call to project module failed!")
else:
logP(fmt("ERROR: Module not loaded or defined! ({moduleFile})"))
return simdata
But the app keeps crashing (even earlier than before, when I got into the loadSpeechData proc called from the initModule
Now it crashes earlier. I'm using Fidget (100% pure nim) as the UI package.
When compiled with : nim c -mm:orc --threads:on -d:useNimRtl ReVoice (with useNimRtl) The crash occurs even before the UI appears, at the loading of the needed fonts (6 in total, crash on 3rd) Loading fonts Loading font IBM Plex Sans (IBMPlexSans-Regular.ttf) Loading font IBM Plex Sans Bold (IBMPlexSans-Bold.ttf) Loading font Typewriter (TT2020StyleE-Regular.ttf) SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Compiled with : nim c -mm:orc --threads:on ReVoice The the crash occurs much later, only on the first call to projectModule in the dll Loading module file ModuleTest.dll Loaded module Test.
Traceback (most recent call last) C:UsersIvanGFRReVoice.nim(1385) ReVoice C:UsersIvanGFRReVoice.nim(1213) main C:UsersIvan.nimblepkgsfidget-0.7.8fidgetopenglbackend.nim(442) startFidget C:UsersIvan.nimblepkgsfidget-0.7.8fidgetopenglbase.nim(216) updateLoop C:UsersIvan.nimblepkgsfidget-0.7.8fidgetopenglbase.nim(167) drawAndSwap C:UsersIvan.nimblepkgsfidget-0.7.8fidgetopenglbackend.nim(360) :anonymous C:UsersIvanGoForReEntryReVoice.nim(1019) drawMain C:UsersIvanGoForReEntryReVoice.nim(281) projectModule C:UsersIvanGoForReEntryReVoice.nim(257) loadModule SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Every DLL needs to use nimrtl.dll so that they all agree on the used memory manager. (This is not Nim specific, it's just the way the world works.)
I think we need a tutorial on DLLs, on the Nim website, even @PMunch article doesn't mention that.
It's actually not true btw, if you don't heap allocate or use your own memory management (no ref, seq, strings) you don't need nimrtl.dll / nimrtl.so / nimrtl.dylib
It's actually not true btw, if you don't heap allocate or use your own memory management (no ref, seq, strings) you don't need nimrtl.dll / nimrtl.so / nimrtl.dylib
And how does that additional knowledge helps newcomers? "Use DLLs, C style memory management and malloc, it works". - "Actually that has the same problems because the DLLs need to agree on the malloc implementation..."
OK, I've tried to get a reasonably compact version for testing.
There are 3 nim files: TESTmain, TESTmodule and Testlang.
# TESTmain : should load and call TESTmodule procs - crash on call to initModule
# compile with : nim c -mm:orc --threads:on -d:useNimRtl TESTmain.nim
# also tried without threads
import os
import strformat, strutils
import std/times
import dynlib
import random
import TESTdata
###################################################################################################
###################################################################################################
# DYNAMIC LIB MODULE LOAD & CALL
#
# all modules only have 3 procs which have a standard call
# - initModule (called with Settings, returns true or false)
# - speechModule (called with SimState and returns SimState = SimData + speech queue)
# - endModule void
###################################################################################################
###################################################################################################
type
InitModuleProc = proc(s: Settings): bool {.nimcall}
SpeechModuleProc = proc(sd: SimState): SimState {.nimcall}
EndModuleProc = proc() {.nimcall}
# used .nimcall. as per https://forum.nim-lang.org/t/1400 that got me farthest on my first tries
# plus, since it's the default for a nim proc and all is pure nim, this seems the optimal choice
var
libModule: LibHandle
initModuleAddr: pointer
initModule: InitModuleProc
speechModuleAddr: pointer
speechModule: SpeechModuleProc
endModuleAddr: pointer
endModule: EndModuleProc
moduleFile: string
moduleLoaded: string = ""
moduleAvailable: bool = false
simstate: SimState
proc loadModule(moduleName: string): bool =
result = false
# create path to module (working dir if development version)
when defined(release):
moduleFile = os.joinPath(pathBase, "Modules", moduleName & ".dll")
else:
moduleFile = moduleName & ".dll"
debugEcho(fmt("Loading module file {moduleFile:s}"))
# try loading the module
try:
libModule = loadLib(moduleFile)
result = true
except:
debugEcho(fmt("ERROR: could not load module {moduleName}."))
sleep(250)
result = false
# get address(es) of the proc(s) in the dynamic lib
var moduleErrors: string = ""
if libModule != nil:
debugEcho(fmt("Loaded {moduleName} module."))
initModuleAddr = libModule.symAddr("initModule")
speechModuleAddr = libModule.symAddr("speechModule")
endModuleAddr = libModule.symAddr("endModule")
if initModuleAddr != nil:
initModule = cast[InitModuleProc] (initModuleAddr)
else:
moduleErrors = moduleErrors & ("- initModule not found\n")
if speechModuleAddr != nil:
speechModule = cast[SpeechModuleProc] (speechModuleAddr)
else:
moduleErrors = moduleErrors & ("- speechModule not found\n")
if endModuleAddr != nil:
endModule = cast[EndModuleProc] (endModuleAddr)
else:
moduleErrors = moduleErrors & ("- endModule not found\n")
else:
moduleErrors = moduleErrors & ("- something wrong with module")
if moduleErrors > "":
debugEcho(fmt("ERRORS: loading {moduleName} gave problem(s):\n") & moduleErrors)
result = false
if result: # seems to have loaded fine : init
moduleLoaded = moduleName
else:
moduleLoaded = ""
return result
proc unloadModule() =
if moduleLoaded != "":
endModule()
unloadLib(libModule)
proc projectModule(simstate: SimState): SimState =
if simstate.simdat.projectName == "":
debugEcho(fmt("WARNING: No project {simstate.simdat.projectName}."))
return # nothing to do TODO: default module, so speech enabled for debugging ?
var
sims = simstate # copy to modify
debugEcho(fmt("In projectModule {sims.simdat.projectName} (loaded={moduleLoaded}).--------------------------------------------------"))
# not same lib module: unload old and load correct one
if moduleLoaded != sims.simdat.projectName:
unloadModule()
if loadModule(sims.simdat.projectName):
debugEcho(fmt("Module {moduleLoaded} loaded successfully..."))
# set data needed for initialization of module
var settings: Settings
# set some data for testing
settings.pathFile = "C:/Users/ivan"
settings.pathData = "C:/Users/ivan/Data"
settings.aBool = true
settings.anInt = 1504
settings.aFloat = 15.04
settings.aString = "Nim dll test"
settings.lang = "en"
debugEcho("Calling initModule with:")
debugEcho(settings)
discard initModule(settings) # initialize !
debugEcho("Returned from call to initModule")
else:
debugEcho(fmt("WARNING: Could not load the project {sims.simdat.projectName} module: {moduleFile}."))
if libModule != nil:
try:
sims = speechModule(sims)
debugEcho(fmt("Returned from speechModule (with {sims.speakQ.len} sentences)"))
except:
debugEcho("ERROR: Call to project module failed!")
else:
debugEcho(fmt("ERROR: Module not loaded or defined! ({moduleFile})"))
return sims
proc createTestfile() =
# create a 500 line test text file
var txt: string
for i in 1..500:
let r = rand(i)
txt = txt & (fmt("TXT_{i:3d}:Random {(r)} out of {i}.\n"))
if i in @[7, 12, 33, 88, 100, 105, 161, 163, 200, 300, 400, 488]:
txt = txt & (fmt("// Just comment {i}\n")) # fake some comment in the file
try:
writeFile("TEST-en.txt", txt)
except:
debugEcho("Could not write test file")
proc main() =
debugEcho("Creating test text file")
createTestfile()
simstate.simdat.projectName = "TESTmodule"
debugEcho("Testing calls to speechModule")
let testData = @["Ivan:Hi bro!:71", "Johan:Hi you:11", "Ivan:Are you coming?:7", "Johan:No, I'm going.:3", "Ma:Oh no!:1"]
for i in 0..testData.high:
let randomItem = rand(testData.high)
simstate.simdat.actionlist.add( (0, testData[i]) )
simstate.simdat.actionlist.add( (0, testData[(randomItem)]) )
simstate = projectModule(simstate)
debugEcho("Back from calling module with this to say:")
for spk in simstate.speakQ:
debugEcho(fmt("{spk.who:7s}|{spk.speaktime:8.1f}|{spk.sentence}"))
sleep(1234)
when isMainModule:
main()
# plugin to be loaded and called by TESTmain
# FIRST create NimRtl.dll with "nim c -mm:orc -o:NimRtl.dll <path-to>NimRtl.nim
# compile with : nim c -mm:orc -d:useNimRtl --noMain --app:lib TESTmodule.nim
import os
import std/times
import strformat, strutils
import TESTdata
import TESTlang
var
data: int = 0
timeNow: float
timeStart: float = epochTime()
moduleSettings: Settings
sims: SimState
proc processMessage(m: string) =
debugEcho(fmt("Processing Message {m}"))
let mesg = m.split(":")
if mesg.len == 3: # otherwise invalid data expecting "<name>:<sentence>:<lookupkey>"
var spk: SpeakData
spk.who = mesg[0]
spk.sentence = mesg[1] & " " & getTXT(mesg[2])
spk.speaktime = epochTime() - timeStart + 2.0
spk.expiretime = spk.speaktime + 10.0
sims.speakQ.add(spk)
proc processFile(f: string) =
debugEcho(fmt("File to process = {f}"))
proc NimMain() {.nimcall, importc.}
#{.push dynlib exportc.}
proc initModule(settings: Settings): bool {.exportc, dynlib, nimcall.} = # also tried .dynlib, exportc.
debugEcho("Arrived in initModule")
NimMain() # Be sure to call this! It will set up garbage collector and initialize any global memory
debugEcho("NimMain has been called")
moduleSettings = settings # copy to dll global var
debugEcho("initModule finished")
return true
proc endModule() {.exportc, dynlib, nimcall.} =
debugEcho("Closing down, these are the settings:\n", moduleSettings)
GC_FullCollect()
proc speechModule*(simstate: SimState): SimState {.exportc, dynlib, nimcall.} =
sims = simstate # copy to a global for this module
# load project specific data
loadSpeechData(simstate.simdat.projectName, "")
let alist = sims.simdat.actionlist
debugEcho(fmt("Action List contains {(sims.simdat.actionlist.len):02d} items"))
# just iterate over (dummy) text file loaded with loadSpeechData
for act in alist:
moduleSettings.anInt += 1
if act.what == ACT_MSG:
debugEcho(fmt("- PROCESSING: {act.what:s}|{act.info}"))
processMessage(act.info)
elif act.what == ACT_FIL:
debugEcho(fmt("- TO PROCESS: {act.what:s}|{act.info}"))
processFile(act.info)
# done processing
sims.simdat.actionlist = @[] # clear the actionlist
return sims
#{.pop}
# Text data : can load different languages, but now only en English is supplied
import times
import tables
import strformat
import strutils
import os
#import zippy/ziparchives
#import random
var
# lookup table with simple speech (words or sentences) found in data-<language> currently only 'en' is implemented
# in data file starts with TXT_ to extract to this table
textTable* = initOrderedTable[string, string]()
countR, countT: int
index: string
proc uncomment*(s: string): string =
# strip comments (start with //) from string
var commentpos = s.find("//")
if commentpos == 0: # found at start
return ""
elif commentpos > 0:
return s[0..commentpos-1].strip()
else:
return s
proc loadSpeechData*(project:string = "", folder: string, language: string = "en") =
# project is not used in this TEST version, just fixed TEST-en.txt (later to become <project>-en.txt)
debugEcho("---------------------------------------In loadSpeechData")
var
speechData: string = ""
resourceFilename: string = "TEST-en.txt"
debugEcho(fmt("Loading speech data from {resourceFilename}\n"))
# read in the data file from the zipped data file (TODO : depending on language)
# Open the file (no zip for TEST)
try:
speechData = readFile(resourceFilename)
except:
debugEcho(fmt("- ERROR : could not find speech data file {resourceFilename}\n"))
debugEcho("---------------------------------------")
debugEcho("DUMP speechData:")
debugEcho(" - size: ", (sizeof(speechData)))
debugEcho(" - length: ", (speechData.len))
debugEcho(" - lines: ", (speechData.countLines))
debugEcho(speechData)
# clear the data
textTable.clear
for d in speechData.splitLines():
inc countR
let dataLine = d.uncomment
debugEcho(fmt("READ {countR:4d}|{dataLine}"))
if dataLine.startsWith("TXT_"):
# simple text (words or phrases)
let s = dataLine.split(':')
if s.len > 1:
index = s[0].replace("TXT_", "")
if not textTable.hasKey(index):
# does not exist : create one
textTable[index] = s[1]
inc countT
debugEcho(fmt("- checked {countR-1:d} items and used {countT}."))
return
proc getTXT*(index: string): string =
# get text from textTable
if textTable.hasKey(index):
return textTable[index]
else:
# returned string may sound weird in TTS, but otherwise tracing what goes wrong will be very difficult
return fmt("|{index} not found| ")
My tries two days ago got me a crash when iterating over the loaded text file. Now I don't even get that far.
TESTmain
Creating test text file
In projectModule TESTmodule (loaded=).--------------------------------------------------
Loading module file TESTmodule.dll
Loaded TESTmodule module.
Module TESTmodule loaded successfully...
Calling initModule with:
(pathFile: "C:/Users/ivan", pathData: "C:/Users/ivan/Data", lang: "en", aBool: true, anInt: 1504, aFloat: 15.04, aString: "Nim dll test")
Traceback (most recent call last)
C:\Users\Ivan\GFR\TESTmain.nim(178) TESTmain
C:\Users\Ivan\GFR\TESTmain.nim(169) main
C:\Users\Ivan\GFR\TESTmain.nim(132) projectModule
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
I think we need a tutorial on DLLs, on the Nim website, even @PMunch article doesn't mention that.
I agree that there needs to be a complete tutorial on dynamic libraries. From A to Z, what we have now kind of stops at W, leaving one in the dark for the remaining X, Y, Z to make it work.
Just for the heck of it I scanned my Windows drive :
That means of Windows executable code 80% is dll, 20% is exe. Maybe some dll are resources, so it may be a bit less, but clearly it's a dll world
That would only encourage more people to waste days of debugging effort on their lovely poorly thought out "plugin systems".
With great power comes great responsibility and all that jazz.
The whole appeal of Nim is to provide people with the power and expressivity to realize what they want. Just like macros power can lead to poorly thought out and unmaintainable code, yes so does plugins.
However, if we wanted to put shackles in the language we could have done Go v2 instead of targeting Lisp expressivity.
"The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt." – Rob Pike
"It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical." – Rob Pike
There are legitimate use to a plugin system, the solution is to educate not to restrict.
CFO asks CEO: “What happens if we invest in developing our people and then they leave us?” > CEO: “ What happens if we don’t, and they stay?”
A programming language cannot fix system designs and architecture issues.
That completely ignores the ongoing costs for us in ensuring the DLL support remains healthy and good.
There is nothing wrong with saying "DLLs are a 2nd tier target, we try our best they work but encourage you to use something else, especially when you're new to Nim".