Mirror of forum.nim-lang.org

10103 :: Writing a wrapper for ML libraries how-to

• StrDev (orginal) [2023-04-16T13:00:44+02:00] view original

  Hi All, I am very new to Nim, but it seems highly interesting for data science. My nooby question is that is there any comprehensive tutorial or documentation (with best practices etc.), how-to write a wrapper for ML libraries, which otherwise expose a C++ API? E.g.

  XGBoost: https://xgboost.readthedocs.io/en/stable/c++.html

  Catboost: https://catboost.ai/en/docs/concepts/c-plus-plus-api

  LightGBM: https://lightgbm.readthedocs.io/en/latest/C-API.html

  I already know that there is a PyTorch wrapper (I am not sure, how up-to-date or ready it is), but I would happily learn how to write bindings for further ML/DL libraries, too. Thanks for your support!

---

• Araq (orginal) [2023-04-16T17:29:43+02:00] view original

  Copy the headers to some file you feel free to edit, run c2nim over it, use options like --nep1 and clean up/edit the headers until c2nim's result is good enough for Nim to compile. Then post your results here and ask for a more ideomatic Nim API.

---

• mratsim (orginal) [2023-04-19T22:08:58+02:00] view original

  XGBoost has a C API: https://github.com/dmlc/xgboost/blob/master/include/xgboost/c_api.h

  I've never used CatBoost though.

---

• pietroppeter (orginal) [2023-04-20T11:19:34+02:00] view original

  it seems highly interesting for data science

  I agree :)

  is there any comprehensive tutorial or documentation (with best practices etc.), how-to write a wrapper for ML libraries, which otherwise expose a C++ API?

  not that I know of.

  For xgboost there is already a wrapper, from what I can tell it binds the dll, not sure how the wrapper was generated: https://github.com/jackhftang/xgboost.nim

---

• StrDev (orginal) [2023-04-21T13:37:29+02:00] view original

  Thank you for all the answers. I already have the Nim in Action book, and ordered Araq’s book. I take a deeper look, how the dynamic linking is made (this is the better approach for this purpose, right?) and make some experiments. I guess the most important missing peace is sklearn, which is a huge library, but for example we work mostly with time series, where it would be great to have at least the most important standard libraries (statsmodels, sktime, darts etc.).

---

• mratsim (orginal) [2023-04-24T17:34:50+02:00] view original

  There was a time (6 years ago) where I was interested in reimplementing Sklearn but unfortunately I'm too busy with other projects.

  However, I would go like this.

  1. Create a robust dataframe library and implement the time-series centric features (like rolling sums and things like that).
  2. Create the key fit/regression model.

  This is something that Nvidia recently had to do with rapids.ai and CuDF. And they did that within the past 3 years so there are a lot of lessons learned there and also strategies to minimize time-to-market.

  The main issue in porting Sklearn is getting lost in the wild wild west of contributions with varying degrees of maturity.

---

• alexeypetrushin (orginal) [2023-04-24T18:09:29+02:00] view original

  do you actually need dataframe in Nim? In Python you need it to delegate data manipulation to C, but why you need dataframe in Nim, you can just use array of rows and plain select/filter/count lambdas.

---

• junelac (orginal) [2023-04-25T16:18:08+02:00] view original

  there is a dataframe library in nim https://scinim.github.io/Datamancer/datamancer.html (never tried)

---

• junelac (orginal) [2023-04-25T16:26:49+02:00] view original

  I'm currently exploring this topic. i found this page very useful as an introduction: https://livebook.manning.com/book/nim-in-action/chapter-8/18

  I'm also experimenting with automatic wrappers. Only with Futhark for the moment, but I will try c2nim and nimterop also.

  My only knowledge of C is some introduction more than 20 years ago, but I managed to create bindings for raylib and make the example here run (https://github.com/raysan5/raylib) , so it's definitively doable :)

  I think

---

• demotomohiro (orginal) [2023-04-25T16:39:17+02:00] view original

  I wrote two articles for people want to use C libraries from Nim:

  • GCC and C tutorial for Nimmer
  • C library tutorial for Nimmer

---

• junelac (orginal) [2023-04-25T18:03:39+02:00] view original

  thanks it looks very useful

---

• ITwrx (orginal) [2023-04-29T02:36:22+02:00] view original

  @StrDev IDK exactly how relevant this is, as i'm fairly ignorant of this field, but in case it's useful: https://github.com/can-lehmann/exprgrad

---