Can somebody please help me to understand why the following example crashes?
import threadpool
import "nimborg/py/high_level"
import "nimborg/py/low_level"
proc testSpawn(path:string) =
var pyos = pyImport("os")
var dir = pyos.listdir("/home/kuba")
var first = $dir[0]
echo first
for i in countup(0, 3):
spawn testSpawn($i)
sync()
########################
test1.nim(8) testSpawn
high_level.nim(265) pyImport
SIGSEGV: Illegal storage access. (Try to compile with -d:useSysAssert -d:useGcAssert for details.)
Traceback (most recent call last)
threadpool.nim(288) slave
test1.nim(14) testSpawnWrapper
test1.nim(8) testSpawn
high_level.nim(265) pyImport
I'm essentially trying to load a python library using excellent nimborg module. Everything works fine as long it takes place on the main thread, but crashes when I move the pyImport("os") statement to a separate thread. My gut feeling tells the problem lays in loading library on one thread but using it on the other. Having said that though, I have no idea if there is any feasible way to work around it...
The plan is to run load python library in complete isolation which should satisfy restrictions of spawn call. The thing I'm not sure about are GC and the import module statements which apparently can only be executed on the main thread.
I wonder if there is any obvious thing I missed or the problem rather lays in how library was implemented and is not suitable for running on other threads.
I appreciate any suggestions.
Thank you.
kuba: I really doubt that's possible - I don't think the CPython runtime was written to support multiple instances of the interpreter in one program. Things like global variables would likely not work right. See this question.
I suggest you either just run separate processes of the CPython executable, or possible construct two programs of your own - one to manage a CPython VM, and the a 'main controller' program, and use pipes/stdout/stdin to communicate.
Hi @kuba, I'm also interested in connection between Nim & Python.
Can I ask which Python libraries you wanted to use with Nim, and what sort of I/O, and for what sort of computation?
Sure, in general I'm interested in geometry computation/simulation where for various performance reasons there are zillions binary, highly optimized formats. The example libraries I'm considering are usually a python wrappers of C++ APIs such as:
https://github.com/ImageEngine/cortex/
http://www.sidefx.com/docs/houdini14.0/hom/
I can see Nim to fit really well here in terms of speed and expressiveness, especially in the early prototyping phase. Hopefully it will scale well too. The only down side is the lack of support for binary formats I mentioned earlier. Since I'm trying to avoid dealing with C++ API, I found python quite usable.
So the idea is to pay the price for slow loading only once at the start, convert data to Nim structures, do the job and spit out asynchronously the geometry for subsequent frames/steps in the background. Unfortunately at this point the last step is not possible and what I have found out so far it comes down to the lack of threading support in python.
Hi @kuba, that's interesting, thanks for the information.
I can't help you with async processing in Python, but over the past 3 months, one of the Nim projects I've been working on is a library to auto-generate Python wrappers (as Python C-API extension modules) for any Nim procs you want to export.
So if you can work out the binary structure of the data (as it is contained in Python), you could write Nim procs that accept PyObject instances that encapsulate this data, then my library could auto-generate the C-API wrappers for you. My library produces standard Python extension modules, which are shared libraries that you can then import into Python as modules.
So perhaps (and again, I'm no expert on async processing), you could:
What OS do you use? I've only tested with CPython on Linux...
jb
Perhaps I still don't understand how this machinery suppose to work. In my (probably not very typical python use case) I end up with a large Nim sequence which needs to be transformed to equivalent Python data structure. Ideally in a non-blocking way. Kicking of a separate python process will require to store that Nim seq temporarily somewhere.
The exported Nim proc will be run as an external separate python process, correct?