I work in a team that leans towards Python for most back-end needs. Any new project or script that anyone writes will be in Python. The CTO is also a very heavy Python user. So, anytime a vendor or external consultant is hired, they are instructed to create the application back-end or script in Python.
I understand that with nimpy, one can use Python code within Nim programs.
How far does it extend to using Python packages (say, the ones found in pypi.org)?
It would be awesome, for example, if I can use boto3 (I do DevOps, and Infra work) within a Nim program; or polars (I write ETL scripts).
If nimpy doesn't work, what are the workarounds that I can use to achieve the above?
I am exploring Nim as it is portable (and blazing fast compared to Python). Then mastering the ecosystem (Nim's) is always beneficial for the future (you'd never need another language; "one language for everything" creed).
I can't stress the portability aspect enough.
I've been told that nimpy works for pretty much anything. Please give it a try.
Disclaimer: I am not speaking from experience with it.
I'm not sure you'll have fun times using python packages in nim. You'll have to wrestle with python's dynamic PyObject all the time.
Also, if you're explicitly instructed to use python, why bring nim into the mix?
Try nimpy first, than say about PyObjects etc. Nimpy abstracts a lot of things, you even can use nimporter to do
import nimmodule
import pkg/nimpy
proc nimadd(a, b: int) {.exportpy.} =
a + bYour approach is good to try or for general use or fun, but I won't recommend it for production grade application
using Nim for Python extensions is really great but using pylibs in Nim is not something very useful for Production app.
Better it will be if you use Binding for libs like polars or use FFI as Nim has great FFI support it will be much clean.
If you do want to take a nimpy approach, here is an idea of what nim(py) based code with polars might look like (using examples from docs.pola.rs):
import nimpy
discard pyImport("sys").path.insert(0, ".venv/lib/python3.12/site-packages")
let pl = pyimport("polars")
let dt = pyImport("datetime")
proc `/`(a, b: PyObject): PyObject = a.callMethod("__truediv__", b)
proc `//`(a: PyObject, b: int): PyObject = a.callMethod("__floordiv__", b)
proc `**`(a: PyObject, b: int): PyObject = a.callMethod("__pow__", b)
proc `*`(a: PyObject, b: int): PyObject = a.callMethod("__mul__", b)
let df = pl.DataFrame(toPyDict({
"name": @["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
"birthdate": @[
dt.date(1997, 1, 10),
dt.date(1985, 2, 15),
dt.date(1983, 3, 22),
dt.date(1981, 4, 30),
],
"weight": @[57.9, 72.5, 53.6, 83.1],
"height": @[1.56, 1.77, 1.65, 1.75],
}))
echo $df
echo $df.select(
pl.col("name"),
pl.col("birthdate").dt.year().alias("birth_year"),
(pl.col("weight") / (pl.col("height") ** 2)).alias("bmi"),
)
echo $(
df.with_columns(
(pl.col("birthdate").dt.year() // 10 * 10).alias("decade"),
pl.col("name").str.split(by=" ").list.first(),
)
.select(
pl.all().exclude("birthdate"),
)
.group_by(
pl.col("decade"),
maintain_order=true,
)
.agg(
pl.col("name"),
pl.col("weight", "height").mean().round(2).name.prefix("avg_"),
)
)To the extent you isolate python and third-party dep logic to python modules the bindings will be simple.
All that to say. I think binding python libraries (particularly if you are using pre-existing code from your team) should work fine. But if there isn't a significant amount of additional nim-only business logic then it's a little questionable how much gain you get here vs friction you introduce for yourself and your team.
I can't stress the portability aspect enough.
Worth noting you'll only be as portable as your ability to distribute whatever python code you already needed to distribute.
I've had fairly good experiences with NimPy, but only when exporting a (very small) Nim API to Python.
I don't think using Python from within Nim will be particularly pleasant. The example on the NimPy homepage already demonstrates this: <https://github.com/yglukhov/nimpy?tab=readme-ov-file#calling-python-from-nim>
I couldn't get Nimporter to work. It did something (without errors) and left me with a huge file that couldn't be imported.
Since the project is primarily used on Linux, I packaged it with standard Python tools and released it on PyPI.
Here is the Python subproject <https://github.com/heuer/mancia/tree/main/python>. It contains both the Python API written in Nim (as mentioned, very small, only two functions) and Python files that use the API.
I've used nimpy extensively. It's very reliable. You may find some edge case with some of Python weirdness but they're usually easy to fix and it's a good way to improve the library.
Exporting Nim API to Python is easy. Using Python from Nim works, but you have to keep in mind in your mental model that you now have an interpreter running within your Nim program.
Just be careful of which version of libpython gets used; you may have to tinker with linker path. If you're interested in Numpy you can use the compatibility lyaer we built in Scinim : https://github.com/SciNim/scinim/blob/main/scinim/numpyarrays.nim ; this also allows to convert numpy arrays in to Arraymancer format.
Regarding :
I am getting the feeling that using FFI to use polars would be more pleasant than using it as a Python import.
Polars python is using the same Rust implementation than you would through native FFI. The overhead is going small.
I have a doubt ?
Nim -> Python runtime -> Polars is this good or
Nim -> FFI -> Target lib may be polars or other
And what is difference between this 2 approach ? If we use Python for polars then can't that be archived in Python itself ? therefore I think using Polar using Nimpy don't seem to impressme as it will not give any benefit more than Python polars lib itself.
Better will be that Try to make polars clone in Nim using lib like Arraymancer or FFI or use any other C lib like Duck DB which have good C api : https://duckdb.org/docs/current/clients/c/overview
Ad: Did someone mentioned DuckDB? :)) I have a half-baked high-level API for here https://github.com/openpeeps/duckdb-nim (or use the C API if you are a warrior).
API ref here https://openpeeps.github.io/duckdb-nim/theindex.html
I'm planning to add support for db_connector, somehow.
I also build some wrappers over Apache arrow: https://github.com/BontaVlad/narrow And duckdb: https://github.com/BontaVlad/NimDrake
Still work in progress, just a learning exercise.
Just be careful of which version of libpython gets used; you may have to tinker with linker path
I was running some AI models last fall using nimpy at work. It was awesome. The hardest part was figuring out how to setup libpython and the environment variables to get it to use the correct Python and import paths. Then I was thankful for Atlas and a simple nim.cfg file.