Have your eyes set on the perfect C library for your project? Can't find a wrapper for it in Nim? Look no further! Futhark aims to allow you to simply import C header files directly into Nim, and allow you to use them like you would from C without any manual intervention. It's still in an alpha state, but it can already wrap many complex header files without any rewrites or pre-processing.
import futhark
# Tell futhark where to find the C libraries you will compile with, and what
# header files you wish to import.
importc:
absPath "/usr/lib/clang/12.0.1/include"
path "../stb"
define STB_IMAGE_IMPLEMENTATION
"stb_image.h"
# Tell Nim how to compile against the library. If you have a dynamic library
# this would simply be a `--passL:"-l<library name>`
static:
writeFile("test.c", """
#define STB_IMAGE_IMPLEMENTATION
#include "../stb/stb_image.h"
""")
{.compile: "test.c".}
# Use the library just like you would in C!
var width, height, channels: cint
var image = stbi_load("futhark.png", width.addr, height.addr, channels.addr, STBI_default.cint)
if image == nil:
echo "Error in loading the image"
quit 1
echo "Loaded image with a width of ", width, ", a height of ", height, " and ", channels, " channels"
stbi_image_free(image)
This is a project I've been working on for the past couple weeks. It allows you to import C files directly in Nim code and just use all procedures and structures without any modification. It aims to bridge the gap between Nim and C and allow you to use any library without having to create any wrappers!
Great! I've just spent a month trying to use a very complex machine-generated lib with nim (multiple-pass C macros and stringify were the main issues I had to tackle). I tried with c2nim first, then i moved to nimterop, then I tried manual import + nim macro, and finally I managed to get it done via c2nim with lots of #define and #mangle + nim macro to turn the result into something nicer for nim.
I was waiting for something like this and I also know that haxscramper was also working on something similar but the added complexity of C++.
I will be happy to test Futhark on the aforementioned C library.
Thanks!
Sounds really cool !
What pre-processing feature are supported, if any ? Often, it's the C macros that makes wrapper complicated
It's very nice that finally solutions that use libclang are showing up. That said, the way c2nim does it is fundamentally the best way:
c2nim is not only a (quirky) C parser, it knows about .bycopy, .cdecl and .size: sizeof(cint) for enums and never forgets these important annotations.
Yes, you need to spend some effort on the wrapping process, but you get something in return:
IMHO we should focus on quality wrappers. Python is not successful because the C interop is so "simple" that you can hack your way around with address-of operators (Python lacks such an operator). It's successful because the wrappers are solid, they always take care of memory management issues and provide a Pythonic API.
I disagree. Using an actual C compiler to understand C will always be the best solution. Whether it is to automatically generate the definitions, or if it is to generate a wrapper that can be shared as-is. Futhark already knows about calling conventions (and not only cdecl, it also supports the other conventions that Nim supports) and tags all enums with the size of their defined underlying type (all integer sizes supported). I haven't yet done anything for bycopy, not quite sure where they are required. Futhark also supports enhancing the types, as shown in the README, and the module it generates can be copied and imported like any other module.
I agree that we should focus on quality wrappers, but Futhark is a tool that can aid with that. It also allows you to use libraries for which there are no wrappers, something that previously have required a significant amount of work and been error prone.
I disagree. Using an actual C compiler to understand C will always be the best solution.
The problem is that the C compiler doesn't understand C, C's type system is full of basic major holes and the compiler only sees the code after preprocessing.
What you say makes logical sense but I'm not buying it anymore. Usually Python has the better wrappers and yet creating a wrapper for Python is much more laborious. Much more. And c2nim is much better than Swig.
If you need to wrap code, do it properly. Bad wrappers stick around and lure our users into half-working solutions, crashes and leaks. A hacky wrapper is a liability.
A C compiler understands C as well as any automated tool can. But you're right that this isn't the same as understanding C. That's why I've written Futhark in such a way that it's very easy to supply your own type definitions to override the automatic one.
As I write in the README a wrapper will always be better. But an automatic wrapper is better than no wrapper, or worse an outdated incorrect wrapper. I've multiple times had to edit a wrapper because it was missing parts (the official OpenSSL ones) or wraps something incorrectly (the clang library I used to write Futhark).
I find nimterop to work better mainly because it has a much better C parser than c2nim, while it still knows about Nim quirks. The best feature it has it that it outputs a Nim file that you can further fine-tune.
c2nim's parser got significantly better in later releases though and it's the only tool that supports C++ anyway. I'm not concerned that other tools do a better job than my work, I'm concerned that they really don't. ;-)
Not really. May be true for tiny C libs with a very simple API. But for larger C libs I would prefer writing my whole app in C than in Nim using a low level wrapper. I tried -- wrote my Ned editor six years ago with the oldGTK3 low level wrapper. Was much uglier, more error prone and more work than writing all in plain C. I guess for low level Nim wrappers for Qt or CGAL it would be the same. Even for smaller libs like cairo or blend2d only a low level wrapper is nearly useless.
I have multiple times written Nim programs with low-level C wrappers. And I agree that for more complex libraries it isn't much better than C. But saying that it's worse is a bit of a stretch. The only downside of using Nim with a properly written low-level wrapper is that you probably have to do a lot of casting between different types which C sees as equal. That being said I find that with a couple of well written templates you can get a lot of work done even with simple wrappers and Nim. As for how error prone it is I don't think it's necessarily any worse than C, but you might get lured into a false sense of security when writing Nim and forget to handle your memory allocations properly. All of this assumes properly written wrappers though. As soon as you have a mismatch between the C types and the Nim types in the wrapper all sorts of errors crop up.
Wrapping C code is best done in two phases: one that expresses the C ABI in Nim terms - pretty much, this is what a header file does in C - you simply replace your .h file with a .nim file and change the syntax, adding appropriate Nim annotations like bycopy and so on. After this stage, the C header file is no longer needed, but should correspond 1:1 to the Nim file.
I whole-heartedly agree, and Futhark aims to automate the first phase. It allows you to import the .h files directly and then focus on writing the high-level wrapper instead.
I find nimterop to work better mainly because it has a much better C parser than c2nim, while it still knows about Nim quirks. The best feature it has it that it outputs a Nim file that you can further fine-tune.
I haven't been able to get nimterop to work sufficiently well for my purposes, which is why I wrote Futhark. It aims to do much the same thing, and will also spit out a Nim file that can be further modified (although I don't recommend this, you should rather use the retyping functionality, or the object override functionality to create better representations if you have something that mismatches).
The final point is that just like you might accidentally mismatch C headers and the compiled libraries they reference
This is the main reason for the automatic wrapping. It is all well and good to take the output from nimterop or c2nim and fine tune it to your needs, but as soon as someone shows up with a newer version it all breaks down. With Futhark it will automatically rebuild the correct definitions and the high-level wrapper can still work just fine as long as nothing has been majorly restructured.
Love the naming: futhark (runic alphabet) and opir (rune master).
Thanks, I felt pretty clever when I came up with that name :)
Well I have used c2nim successfully, on thousand of lines of C++ code.
I haven't. I have tried time and time again to use c2nim, and later nimterop, without any luck. Last time I tried I spent days fiddling with .h files and felt I was slowly inching my way closer to something that could maybe, possibly work (although how would I know, I had changed a lot of the .h files, so I might've messed up at some point). I got a lot of help on IRC as well, but as this was meant to be used for a project at work and I really wanted to show how well suited Nim was for this task I ended up writing the definitions manually instead. This has been working well for me since then, but every time the underlying library changes I have to go over all the structures again, or just try to run the software and hope it doesn't randomly SEGFAULT. The latter scenario has happened more than once already..
Futhark was written out of frustration. Frustration with what I saw as a gaping hole in the Nim ecosystem. Nim is great, and it's C interop is top-notch, but the inability to easily and automatically get up-to-date information from C libraries have meant that you either have to spend a lot of time writing wrappers by hand, or hope that someone has already sunken this time cost for you (and published it, my work project wrappings are still not publicly available).
As many here has pointed out using a C library directly in Nim might not be very pleasant or ergonomic, and it might be prone to errors. But by taking human error-prone labour out of the equation it is much more likely to provide a good result. As I mentioned I have used many wrappers for C libraries that have either not been tested properly (who has time to test all the different edge-case functions of a C library?) or which has simply been outdated, either by changes to the C sources, or by changes to the Nim language. All these scenarios require manual work to fix, manual work that isn't necessary with Futhark.
To put it this way, after spending about as much time writing Futhark as I used trying (and failing) to get c2nim and nimterop to work on wrapping the C sources I needed I replaced my 300 lines of manually written labour intensive wrappers that I spent a couple days writing and more time than I care to admit on maintaining (and they still only covered just what I needed to move on with the project) with about 10 lines of Futhark statements telling it which header files I wanted, what defines I required, and the extra type-safe re-definitions I had applied. It is now able to seamlessly work across versions of the original source without breaking, and I can finally but to rest the tiny voice in the back of my mind telling me that they might be wrong.
So to sum up, I wrote Futhark because none of the other solutions have worked for me. It is still not perfect, but I hope that it will be soon. It allows me to just import C and not have to think about it at all if I don't want to. Had the other tools worked I wouldn't have had to do it, and I hope that new and better Nim wrappers will come from it.
Wrapping libclang is great! I'll have to play around a bit with Futhark. Its nice that it's simpler than nimterop (which just confuses me).
I use c2nim and find it pretty good overall, and even patched it a bit to support some use cases that I wanted (it took a couple of hours of figuring out the codebase). c2nim has gotten much better than even a year ago and now usually it's just a few stray function attributes that cause issues. However, I do almost always need to patch at least 1 line of most C headers to get it compiling with c2nim.
Overall I do seem to fall a bit in between Futhark's and c2nim's workflows though..
I would prefer Futhark's approach of parsing directives living in a separate Nim config file (nimterop too?). c2nim's workflow of modifying the C headers directly using 'mangle's and 'def's becomes more complicated when you're wrapping a C library and not doing a one time conversion to Nim. Generally I copy a C given header to my Nim repo, tweak it to compile with c2nim (usually 1-2 lines nowadays). Then I usually need to post-process the Nim code. Then I have to decide to keep or to delete the modified header. Even if c2nim worked on 100% of C headers there's always some mangling to do (e.g. specify whether to import a given C macro as a proc or to resolve it and import the C code, etc).
On the other hand, it does seem that c2nim's general approach of outputting Nim code that's then checked into the repo is better than generating the Nim wrappers on-the-fly. Particularly if during wrapping the C/C++ doc's are copied. Then it's possible to have nimsuggest or grep work with it. It's much easier to know exactly what the Nim names are when the C ones don't work in Nim. At the end of the day, as a user of a wrapper C API I don't want to know how the C importer is configured, I want to be able to read the docs and then see suggestions pop up in VSCode. :-) Not that I can complain, I've been terrible about docs in my libs so far.
Thanks for writing Futhark, I'm curious to see where you bring it! Maybe I'll port some of it to c2nim as well for Araq. ;)
On a somewhat related note, I want to create some tooling for setting up C compiler static checks for wrapped API's. So perhaps Futhark's ability to spit out a nice JSON file could come in handy for that.
I did a small version with an RTOS project that optionally compiles any combination of IPv4/IPv6 and the Nim wrapped version would end up mismatched eventually. However I still wanted const's for things but it's a bit of an issue if the IP address struct is the wrong size! So I added some static_asserts and it saved me a lot of potential errors by asserting that certain parts of the C API and Nim's idea of that API agree at compile time. It'd seem handy to be able to statically check C/C++ wrappers at compile time rather than getting those lovely SEGFAULTS when the C api drifts.
I ended up writing the definitions manually instead.
Well you could copy the definitions that you actually need into a .h file and run c2nim on it...
I would prefer Futhark's approach of parsing directives living in a separate Nim config file (nimterop too?).
c2nim supports that with a config.c2nim configuration file.
I guess Futhark might not be good for all C libraries - but wouldn't it be ok to have it for some libraries?
Yes, indeed. Especially if the "C library" is actually a constantly moving target (and not much of a "library" then but I digress).
Futhark looks nice for somebody's personal needs, but don't push these wrappers into the Nimbleverse without QA (quality assurance) please. And I don't know how to do QA for "hey here is a header file in /usr/lib (because who cares about Windows) and some manual type overrides"...
I mean this thing is basically: "I haven't looked at the result but it worked on my machine". And that's how you get better quality out of the process? Beats me.