Compile time FFI has been a wish of mine for some time now. I noticed that it's mostly developed, but has been disabled: https://github.com/nim-lang/Nim/commit/c5dbb0379fc431a01220224097f00323df6c9ced
Is there any reason why it's still disabled after a year? Is there still work left to be done?
I've been using it extensively ever since the feature was introduced, it's a very useful feature and works well. Feel free to upvote https://github.com/nim-lang/Nim/pull/13091 which re-enables testing this feature in CI, to prevent future regressions.
all posix platforms we have in CI are supported (linux (32 and 64), osx, freebsd); win32 is supported; the only thing that needs work is win64, help welcome for that
all posix platforms we have in CI are supported (linux (32 and 64), osx, freebsd); win32 is supported; the only thing that needs work is win64, help welcome for that
That's awesome to hear! What in particular needs work on Windows 64bit?
I started playing around with it and noticed something. Right now, CT FFI tries to load an importC symbol from a DLL. Is there a way that you currently use to import a C macro from a header file at compile time?
What in particular needs work on Windows 64bit?
libffi bindings only support win32, not win64; maybe a solution is to use a dll for win64 instead of embedding the sources, see https://github.com/Araq/libffi/blob/master/libffi.nim#L14
Is there a way that you currently use to import a C macro from a header file at compile time?
not that I know of but the following works:
when defined(timn_D20191211T153325):
{.emit:"""
// or: # include "temp.h" which would contain these macros:
#define MAX(a,b) ((a < b) ? (b) : (a))
#define MY_CONST1 "asdf"
#define MY_CONST2 23
// instantiate these macros to get concrete symbols
N_LIB_EXPORT int MAX_int(int a, int b){ return MAX(a, b);}
N_LIB_EXPORT char* get_MY_CONST1(){return MY_CONST1;}
N_LIB_EXPORT int get_MY_CONST2(){return MY_CONST2;}
""".}
else:
import std/[strformat,os,strutils]
const libF = "/tmp/" / (DynlibFormat % "D20191211T153325")
const nim = getCurrentCompilerExe()
const file = currentSourcePath()
static:
let cmd = fmt"{nim} c -d:timn_D20191211T153325 --app:lib -o:{libF} {file}"
echo cmd
let (output, exitCode) = gorgeEx cmd
doAssert exitCode == 0, output
proc MAX_int(a, b: int): int {.importc, dynlib: libF.}
proc MY_CONST1(): cstring {.importc: "get_MY_CONST1", dynlib: libF.}
proc MY_CONST2(): cint {.importc: "get_MY_CONST2", dynlib: libF.}
proc main()=
doAssert MAX_int(2, 3) == 3
doAssert MY_CONST1() == "asdf"
doAssert MY_CONST2() == 23
static: main()
main()
libffi bindings only support win32, not win64; maybe a solution is to use a dll for win64 instead of embedding the sources, see https://github.com/Araq/libffi/blob/master/libffi.nim#L14
Interesting. It looks like it should support win64. I'll have to take a look later.
Thank you for the detailed example. I'll see if I can write a macro to make this easier. It seems this is more limited than I thought. Kinda sucks that the symbols have to be in a dll to be used, but this is still awesome.
It seems that the CT FFI doesn't support structs yet (as far as I can tell). You said you've been using it quite extensively, so does that mean you've found a way to use more complex data structures?
for C/C++ structs, the binary layout is different from the PNode binary layout of a compile time object instance, so you can't just deref a ptr Foo returned by a importc proc (that wouldn't make sense); and thankfully the compiler prevents it.
However you can access individual fields via accessors. Again, this can be automated to a large extent using macros that would generate such accessors.
Here's a complete working example, where I show accessors either by value (get_x1) or by pointer (get_x2_ptr):
when defined(timn_D20191211T153325):
{.emit:"""
#define MAX(a,b) ((a < b) ? (b) : (a))
#define MY_CONST1 "asdf"
#define MY_CONST2 23
typedef struct Foo {
int x1;
double x2;
} Foo;
N_LIB_EXPORT int MAX_int(int a, int b){ return MAX(a, b);}
N_LIB_EXPORT char* get_MY_CONST1(){return MY_CONST1;}
N_LIB_EXPORT int get_MY_CONST2(){return MY_CONST2;}
// example showing returning struct
N_LIB_EXPORT Foo* get_FooPtr(int x1, double x2){Foo* ret = (Foo*)(malloc(sizeof(Foo))); ret->x1 = x1; ret->x2 = x2; return ret;}
N_LIB_EXPORT int get_x1(Foo*a) {return a->x1;}
N_LIB_EXPORT double* get_x2_ptr(Foo*a) {return &a->x2;}
""".}
else:
import std/[strformat,os,strutils]
const libF = "/tmp/" / (DynlibFormat % "D20191211T153325")
const nim = getCurrentCompilerExe()
const file = currentSourcePath()
static:
when true:
let cmd = fmt"{nim} c -d:timn_D20191211T153325 --app:lib -o:{libF} {file}"
echo cmd
let (output, exitCode) = gorgeEx cmd
doAssert exitCode == 0, output
proc MAX_int(a, b: int): int {.importc, dynlib: libF.}
proc MY_CONST1(): cstring {.importc: "get_MY_CONST1", dynlib: libF.}
proc MY_CONST2(): cint {.importc: "get_MY_CONST2", dynlib: libF.}
type Foo = object
x1: cint
x2: float
proc get_FooPtr(x1: cint, x2: float): ptr Foo {.importc, dynlib: libF.}
proc get_x1(a: ptr Foo): cint {.importc, dynlib: libF.}
proc get_x2_ptr(a: ptr Foo): ptr float {.importc, dynlib: libF.}
proc main()=
doAssert MAX_int(2, 3) == 3
doAssert MY_CONST1() == "asdf"
doAssert MY_CONST2() == 23
# example showing passing/returning struct (by pointer)
let foo = get_FooPtr(123, 1.5)
doAssert foo.get_x1 == 123
doAssert foo.get_x2_ptr[] == 1.5
foo.get_x2_ptr[] = 2.0
doAssert foo.get_x2_ptr[] == 2.0
static: main()
main()
Reasons why I remain skeptical:
Unclear QA
How to ensure quality of this feature? Say I run an SDL2 game at compile-time, it crashes, now what? How do I debug that? Can I close these bug reports as "won't fix"?
Use Cases
What are the use cases? What is it you need to do at compile-time that's otherwise impossible? In the past I've seen an over-use of CTFE causing bugs and problems. The primary motivation for CTFE in Nim is Nim's macro system, the full VM was designed for macro evaluation. For the other use cases we could come up with a better staticExec mechanism so that we generate Nim source code as an intermediate step. This code can then be checked-in and is not machine specific. A mechanism like that could also make #defines importable as const, an often requested feature.
Agree with Araq here;
find a way to do it with (a possibly improved) staticExec - or alternatively, it could be solved with a bidirectional communication channel through pipes (or sockets or shared memory, though these aren't good ideas either) if Nim code with access to compiler types etc. needs to interact with external code.
But invading the compiler's memory address space is a recipe for disaster. At first it will work, but down the line people will complain that the nim compiler is buggy and unstable, but it will later turn out that the crash is caused because of a memory corruption caused by importing a library 3-layers deep which uses the compile time FFI.
But seriously, I think even staticExec+staticRead is more than enough. At some point, you have to rely on a build/make tool that can do more than your language do internally.
@Araq
What are the use cases? What is it you need to do at compile-time that's otherwise impossible?
I would simply like to use the entire std lib at compile time. For example, sometimes it's annoying to not be able to use the regex module (although nitely's pure nim library can be used as an alternative). Or to be able to read a large file at compile time (several GB), since staticRead has to read in the entire file at once.
In the past I've seen an over-use of CTFE causing bugs and problems.
That may be true, but it should be up to the programmer if they want to over-use it. But I understand where you're coming from. Debugging issues related to this would be hard and probably a huge time sink.
The primary motivation for CTFE in Nim is Nim's macro system, the full VM was designed for macro evaluation. For the other use cases we could come up with a better staticExec mechanism so that we generate Nim source code as an intermediate step. This code can then be checked-in and is not machine specific. A mechanism like that could also make #defines importable as const, an often requested feature.
I love this idea too. However, I still think CTFE is valuable for the sake of being able to use the entire std lib. It also allows someone to use C libs at compile time with other targets, specifically JS.
@PMunch I agree with you completely. Sometimes you have to "roll your own" at compile time even if there's already a C library that does what you need.
@cumulonimbus
But invading the compiler's memory address space is a recipe for disaster. At first it will work, but down the line people will complain that the nim compiler is buggy and unstable, but it will later turn out that the crash is caused because of a memory corruption caused by importing a library 3-layers deep which uses the compile time FFI.
That may be true. A compiler flag to have it turned on with some docs saying "this feature is very powerful and may cause issues if used too extensively". You can easily cause confusing memory corruption in "normal" Nim now by doing weird emit statements or playing with pointers. Just because it's dangerous doesn't mean it shouldn't be allowed. It should be up to the programmer to use the feature responsibly.
@timothee Thank you again for the detailed example. I see you're getting around the struct issue by passing back a pointer. It gets kind of messy, but I suppose I can deal with it :)
if you'd like to help, please tell me how I can download / build libffi on windows... I'm not a windows user
I'm going to give it a shot tonight. I'll let you know how it goes!
I essentially had the same use case as @PMunch in the past.
When I thought about implementing reading Keras stored NNs in Arraymancer, one of the problems was that
2. Keras networks are stored in HDF5 files and the network layout in attributes of some groups. So for the most straight forward way to implement this, I wanted to read the attributes of Keras HDF5 files at compile time. Given that HDF5 is a rather complicated file format, implementing my Nim based parser (even if only for attributes) wasn't really in the cards.
There's certainly solutions to this without requiring compile time FFI of course.
When I started out with nimterop, I was forced to leverage gorgeEx and create a separate binary toast due to this exact limitation. Nimterop uses tree-sitter which is a C/C++ parsing library for a variety of programming languages so it wasn't possible to load it during compile time.
Initially, I converted the tree-sitter AST into S-expressions which was piped in and converted back into a data structure in the VM and then converted into Nim. This was slow and clunky so the entire C => Nim processing is now done in toast.
I ended up with two advantages - toast is much faster, not by just avoiding the double conversion but also because the VM is slower. In addition, toast can now be used as a standalone tool if anyone prefers that approach, similar to c2nim.
I would certainly have preferred having the macro system available but at least in this particular case, it worked out positively.
I do agree though that having the full stdlib and FFI available during CT will be helpful for some cases but the performance implications might be worth consideration.
@timothee
I sent you build instructions for libffi on gitter, but here they are just for posterity:
I followed the instructions here:
https://proj.goldencode.com/projects/p2j/wiki/Building_and_Installing_libffi_on_Windows
Making sure to configure using the MSYS as suggested. But I used the tar.gz from https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz
Also, the command to configure for 32 bit (via MSYS terminal) is:
CC="/c/Users/jyapa/Downloads/mingw32/bin/i686-w64-mingw32-gcc.exe" sh ./configure --build=i686-w64-mingw32 --host=i686-w64-mingw32
and for 64 bit is
CC="/c/Users/jyapa/Downloads/mingw64/bin/x86_64-w64-mingw32-gcc.exe" sh ./configure --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
^^ the above two configures lets you cross compile to Win64 from Win32 and vice-versa.
Not sure if the libs are portable across Windows or not, but I could upload them if needed.
I'm not a frequent nim user, but this is a feature I have been looking forwards to since my first nim project in 2016.
My usecase - I would like to compile bpf filters at compile time to attach to sockets, and I don't want to rely on libpcap being installed on target systems.
My code looks like this:
proc compile_filter*(filter: string): Filter {.compileTime.} =
## Gets a bpf string and uses libpcap to convert it to
## an array of bpf instructions.
##
## To compile the filter we use an external compiled (nim) program that
## uses pcap. In future versions of nim we will be able to use the
## ffi at compile time and then won't even need the external program
let prog = gorge("../tools/compile_filter \"" & filter & "\"").split("\n")
result = @[]
try:
for instruction in prog:
let parts = instruction.split(" ")
result.add(sock_filter(code: parts[0].parseInt.uint16,
jt: parts[1].parseInt.uint8,
jf: parts[2].parseInt.uint8,
k: parts[3].parseInt.uint32))
except:
raise newException(InvalidFilterError, "Invalid filter " & filter)
and the important part of compile_filter minus types and boilerplate looks like this:
proc pcap_compile*(p: pcap_t; fp: ptr sock_fprog, str: cstring, optimize: int, netmask: int): cint {.importc, cdecl, header: "<pcap/pcap.h>".}
when isMainModule:
if paramCount() != 1:
echo "Usage: " & paramStr(0) & " filter"
quit(1)
var program: sock_fprog
let p* = pcap_open_dead(1, 65536)
if p == nil:
raiseOSError(osLastError())
if pcap_compile(p, addr program, paramStr(1), 1, 0) != 0:
raiseOSError(osLastError())
for i in 0..program.len-1:
echo program.filter[i]
pcap_freecode(addr program)
let filter = compile_filter("udp port 1234")
sock.attach_filter(filter)
The major advantage of having the FFI be available at compile time is not having to convert my data to a string in one program and then parsing the string again back into an object at compile time.
On second thought - not sure how well this could possibly work. When compiling my stand alone utility I pass in: --passL:"-lpcap" so that my binary will like against pcap, but if I want this code to run in compile time I need to either link the compiler to libpcap or dynamically load the libpcap into the compiler process.
I can see where @Araq's hesitation comes from
On second thought - not sure how well this could possibly work. When compiling my stand alone utility I pass in: --passL:"-lpcap" so that my binary will like against pcap, but if I want this code to run in compile time I need to either link the compiler to libpcap or dynamically load the libpcap into the compiler process. > I can see where @Araq's hesitation comes from
that's not needed; since https://github.com/nim-lang/Nim/pull/11635 you can use {.importc, dynlib: "libfoo.dylib".} with CT FFI, see examples in that PR,
If it doesn't work for you, please file a minimum reproducing example (or better, submit a PR)