I have this code that works on Windows, but not on Ubuntu MATE on the Raspberry Pi 3:
#
# nim_module.nim
#
type
PySizeT* = clong
# typedef int (*traverseproc)(PyObject *, visitproc, void *);
TraverseProc* = proc (ob: PyObjectPtr,
prc: VisitProc, p: pointer): cint{.cdecl.}
# typedef int (*inquiry)(PyObject *);
Inquiry* = proc (ob: PyObjectPtr): cint{.cdecl.}
# typedef void (*freefunc)(void *);
FreeFunc* = proc (p: pointer){.cdecl.}
# typedef int (*visitproc)(PyObject *, void *);
VisitProc* = proc (ob: PyObjectPtr, p: pointer): cint{.cdecl.}
# typedef struct _typeobject PyTypeObject; /* opaque */
PyTypeObjectPtr* = ptr PyTypeObject
PyTypeObject* {.final.} = object # Defined in "Include/object.h"
# typedef struct _object {
# _PyObject_HEAD_EXTRA
# Py_ssize_t ob_refcnt;
# struct _typeobject *ob_type;
# } PyObject;
#
# #define _PyObject_HEAD_EXTRA
PyObjectPtr* = ptr PyObject
PyObject* {.final.} = object # Defined in "Include/object.h"
obRefcnt*: PySizeT
obType*: PyTypeObjectPtr
# struct PyMethodDef {
# const char *ml_name; /* The name of the built-in function/method */
# PyCFunction ml_meth; /* The C function that implements it */
# int ml_flags; /* Combination of METH_xxx flags, which mostly
# describe the args expected by the C func */
# const char *ml_doc; /* The __doc__ attribute, or NULL */
# };
# typedef struct PyMethodDef PyMethodDef;
PyMethodDefPtr* = ptr PyMethodDef
PyMethodDef* {.final.} = object # Defined in "Include/methodobject.h"
mlName*: cstring
mlMeth*: PyCFunction
mlFlags*: cint
mlDoc*: cstring
# typedef PyObject *(*PyCFunction)(PyObject *, PyObject *);
PyCFunction* = proc (self, args: PyObjectPtr): PyObjectPtr{.cdecl.}
# typedef struct PyModuleDef{
# PyModuleDef_Base m_base;
# const char* m_name;
# const char* m_doc;
# Py_ssize_t m_size;
# PyMethodDef *m_methods;
# struct PyModuleDef_Slot* m_slots;
# traverseproc m_traverse;
# inquiry m_clear;
# freefunc m_free;
# }PyModuleDef;
PyModuleDefPtr* = ptr PyModuleDef
PyModuleDef* {.final.} = object
m_base*: PyModuleDefBase
m_name*: cstring
m_doc*: cstring
m_size*: PySizeT
m_methods*: PyMethodDefPtr
m_slots*: PyModuleDefSlotPtr
m_traverse*: TraverseProc
m_clear*: Inquiry
m_free*: FreeFunc
# typedef struct PyModuleDef_Base {
# PyObject_HEAD
# PyObject* (*m_init)(void);
# Py_ssize_t m_index;
# PyObject* m_copy;
# } PyModuleDef_Base;
# #define PyObject_HEAD PyObject ob_base;
PyModuleDefBasePtr* = ptr PyModuleDefBase
PyModuleDefBase* {.final.} = object
ob_base*: PyObject
m_init*: proc (): PyObjectPtr {.cdecl.}
m_index*: PySizeT
m_copy*: PyObjectPtr
# typedef struct PyModuleDef_Slot{
# int slot;
# void *value;
# } PyModuleDef_Slot;
PyModuleDefSlotPtr* = ptr PyModuleDefSlot
PyModuleDefSlot* {.final.} = object
slot*: cint
value*: pointer
# PyObject* PyModule_Create2(struct PyModuleDef* module, int module_api_version)
proc moduleCreate2*(module: PyModuleDefPtr;
module_api_version: cint): PyObjectPtr {.cdecl,
importc: "PyModule_Create2" dynlib: "libpython3.5m.so.1".}
proc moduleCreate*(module: PyModuleDefPtr): PyObjectPtr {.cdecl.} =
result = moduleCreate2(module, 3)
var
nim_method_array: array[1, PyMethodDef] = [
PyMethodDef(mlName:nil, mlMeth:nil, mlFlags:0, mlDoc:nil)
]
nim_methods* {.exportc.}: PyMethodDefPtr = cast[PyMethodDefPtr](
addr(nim_method_array)
)
var nim_lexers* {.exportc.}: PyModuleDef = PyModuleDef(
m_base: PyModuleDefBase(
ob_base:PyObject(obRefcnt:1, obType:nil),
m_init:nil,
m_index:0,
m_copy:nil),
m_name: "nim_lexers",
m_doc: "",
m_size: -1,
m_methods: nim_methods,
)
# Hack needed on windows
{.emit: """N_CDECL(void, NimMain)(void);""".}
proc PyInit_nim_lexers(): PyObjectPtr {.exportc, cdecl.} =
{.emit: """NimMain();""".}
result = moduleCreate(addr(nim_lexers))
Traceback (most recent call last)
nim_module.nim(140) PyInit_nim_module
nim_module.nim(115) moduleCreate
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
I suspect it's an error in one of the types since I got the same errors when making the python3 wrapper, but I've been looking at the code for so long and cannot find anything. I copied all of the types from the python3 wrapper and added the C declarations on top of every type and proc definition for clearity, but maybe I missed something. It's
If anyone can see any mistakes or knows more about Python3, please help!
Thanks, Matic
Hey Araq,
Doesn't nim_methods point to the array with only the sentinel, or am I missing something?
Ok, I corrected the PyObject object (it has only the obRefcnt and obType fields) but now I'm getting the following error:
*** Error in 'python3': free(): invalid pointer: 0x76508050 ***
Traceback (most recent call last)
nim_module.nim(133) PyInit_nim_module
nim_module.nim(106) moduleCreate
SIGABRT: Abnormal termination.
I have no more good ideas. If anyone finds anything, please let me know!
Matic
Hey @JohnS,
Installed Valgrind and as a test I ran it with:
valgrind python3 -c "print('TEST')"
Any ideas?
I now tried the following. Python3 initializes a module by using the C function PyInit_{MODULE_NAME}, inside which I only initialize the Python3 stuff using the C Py_Initialize function like so:
# The imported Python3 initialization function
proc initialize*(){.cdecl, importc: "Py_Initialize" dynlib: "libpython3.5m.so.1".}
# The module's init function
proc PyInit_nim_module() {.cdecl, exportc.} =
{.emit: """NimMain();""".}
initialize() # <--- ERROR
Maybe you have the newer python3.6 on Ubuntu or the .so file has a different link name ? For me the second example compiles with python3.6 on openSuse Tumbleweed.
# The imported Python3 initialization function
proc initialize*(){.cdecl, importc: "Py_Initialize" dynlib: "libpython3.6m.so.1.0".}
# The module's init function
proc PyInit_nim_module() {.cdecl, exportc.} =
{.emit: """NimMain();""".}
initialize() #
Your first example does not compile , some error in nim_lexers.
Hey @qqtop,
I fixed the first example above, could you try it again? Did you try importing the second compiled module from python3? It compiles on my machine too, but when I import the module from python3 it throws the error.
Thanks, Matic
After further testing and help from jasonrbriggs, I found out that the error appears on all Debian Linux distributions. It works on Windows and Arch Linux, but on Ubuntu MATE (ARM), Lubuntu (x86) and Raspbian (ARM) it doesn't. Tried Python versions from 3.4 to 3.6, Nim was at the latest github version.
Even this simple example (already shown above) like this throws an error:
# The imported Python3 initialization function
proc initialize*(){.cdecl, importc: "Py_Initialize" dynlib: "libpython3.YOUR_VERSIONm.so.1.0".}
# The module's init function
proc PyInit_nim_module() {.cdecl, exportc.} =
{.emit: """NimMain();""".}
initialize() # <--- ERROR
Anyone know how the Python 3 installaion Debian distributions differs from other distributions or other OS'es?
@Araq,
Fiddled around in the generated C file in the nimcache directory by adding code to load the library libpython3.5m.so.1.0 in C directly and it throws the same error!
The added code (added to the generated PyInit_nim_module function) is:
N_CDECL(..., PyInit_nim_module)(void) {
NimMain();
nimln_(171, "nim_module.nim");
void* handle;
nimln_(900, "nim_module.nim");
handle = dlopen("libpython3.5m.so.1.0", RTLD_LAZY);
nimln_(901, "nim_module.nim");
void (*init)(void) = dlsym(handle, "Py_Initialize");
nimln_(902, "nim_module.nim");
(*init)(); <----------------- ERROR
nimln_(903, "nim_module.nim");
dlclose(handle);
/*
Rest of the code
*/
}
I know this question would probably be better placed in a python forum, but I'm hoping someone knows something about the problem!
Hey,
The conclusion from the python3 IRC discussion:
Debian and Debian based distributions do not load the Python 3 shared library into memory, so it is necessary to link statically with the Python.h header.
Lesson learned.
Ok, so I managed to write an extension by directly using the Python.h header. This works great on Windows x86/x64 and Linux x86/x64, but not on Linux ARM x86 (Raspberry PI 3). By doesn't work I mean it compiles and executes, but EVERY call to the Nim compiled module (.so shared library) has about a one second delay where the CPU usage of one core jumps to 100%.
Is this a known issue on ARM?
Yep, Araq, you were right :)
It's an implementation issue that has nothing to do with Nim. Seems that the Raspberry PI 3 has a bit of a hard time processing long strings.
Thanks.
Araq, a little help please.
I have a Nim procedure that takes text and a highlighting function from Python, then parses and highlights it. This works great but when parsing large pieces of text something with the Nim garbage collector causes a segmentation fault in Python. I know it has something to do with the GC, because the segmentation fault goes away if I compile the module with --gc:none. But if I do this, the memory of the running application constantly rises, as you would expect.
Is there a solution for this without disabling the GC?
Again, this is happening on Linux ARM.
Thanks jlp765!
Tried GC_disableMarkAndSweep() at the beginning and GC_enableMarkAndSweep() at the end of the procedure, still it throws the segmentation fault.
Then I tried GC_disable() and GC_enable() + GC_fullCollect() and it works!