Hi all. I'm pretty new to Nim and this is my first post on the forums. Great community here!
So, I am a Python programmer(most of the time). I'm currently developing a web application that is very data heavy. I am using Django and the Django paginator to paginate querysets that could be thousands of elements in length. The paginator can be quite sluggish on larger datasets, but has a wonderful API. There are also some sorting algorithms that perform some very costly operations. Unfortunately, I believe that I have pretty much reached the end of the optimizations I can make in Python. This is causing requests to some specific endpoints to be very slow. I've been learning Nim via the official tutorials and the the Nim in Action book. I feel like I have a strong enough grasp of the language at this point to convert the Django paginator and some of my sorting functions to Nim. I believe this will help me increase performance in a few critical areas.
The only issue I have is that it doesn't seem like there is a well defined path for writing Nim extensions to Python. I found the pymod package which seems like it will help me do what I want, but I really don't want to have Numpy as a dependency as I have no other use for it and it is quite heavy. I feel like I should be able to write a C extension module that wraps Nim code, but I can't really think of a good way to go about it.
I would like to avoid writing my extension in pure C because I'm not all that great at it. Also, if I ported the Django paginator, I could release it as a nimble package.
Does anyone have any experience with extending Python with Nim?
Thanks!
I figured I'd probably need to go down that path. I should have enough time before performance becomes a problem. I just need to make sure that the compiled nim binary exposes the same PyObject API as the python C API expects. Time to get reading I guess!
scriptkiddy,
I actually just finished up writing a Python module around a Nim dynamic library. I definitely learned a lot on my own - there are good resources out there, but they're incomplete. I'll be writing up my findings. In the meantime, this is the blogpost I used to get started: https://akehrer.github.io/posts/connecting-nim-to-python/
subsetpark,
Thanks for the link! That is exactly what I was looking for. I can see this technique becoming a lot more complex with more complex data structures, but it is a great start. It seems like the main difficulty is converting between Python, C, and Nim types. That's not surprising. I think writing Nim functions and then exporting them to C header files should help with figuring out which ctypes Python should pass to the foreign function.
wow @blvd, that looks like a high quality article. I only had a chance to skim it, but it's looking good. It's scary that I almost missed it (this thread is almost past the bottom), Nim articles are always encouraged!
btw I hope you won't mind if I tweet about it using Nim's Twitter.
@bvld
Looks great. Should definitely be helpful. As far as critique goes, I noticed a lot of grammar mistakes. If you'd like, I can do some editing for you.
Although PyMod is really great, I would like to build a lighter weight implementation that does not require Numpy. Numpy is also great, but it is a very heavy dependency that many projects don't really require. Either way, the information here is valuable and so is the PyMod project itself.
Unfortunately, I haven't had much time to work on this lately as I have been focusing more of my work on the front end implementation of my recent project in Javascript land. Still using my nights and weekends to hack on this though.
Good work on this. I would say realistically its going to be tough to compete with Cython for this use case when Cython can get C like speeds and has a more pythonic syntax than Nim. Interesting article for this here.
Although PyMod is really great, I would like to build a lighter weight implementation that does not require Numpy.
That would be brilliant. In addition, I think PyMod might not be maintained any more, so your project would really help fill that gap.
@dom96
That would be brilliant. In addition, I think PyMod might not be maintained any more, so your project would really help fill that gap.
I'll make sure to post up the link to the repo in this thread once I get a proof of concept up.