Hi, I am kinda new to nim.
I am trying to make a libretro core that uses nim.
For those that are not familiar, the format of a retro-core is a DLL with a ton of specially-named exports. I started with this example and it builds fine and runs on mac x86_64 retroarch. My thinking is that I can make the main libretro wrapper DLL in C (since it has headers, and I don't have to translate anything to C) and call a few functions on the nim-side.
I added a task to my nimble file:
task libretro, "Build libretro host":
selfExec("c --threads:on --app:staticlib --out:null0-libretro.a -d:release libretro/null0_libretro.nim")
exec("gcc null0-libretro.a libretro/libretro.c -o null0-libretro.dylib -lphysfs")
The -lphysfs bit is just because I will be using that, but it seems to do the same stuff, even if I remove that.
and it seems to build ok. To test, I export a few functions:
# no input or output
proc null0_test1*(): void {.stdcall,exportc,dynlib.} =
echo("test1 working")
# int output
proc null0_test2*(): cint {.stdcall,exportc,dynlib.} =
echo("test2 working")
return 1
# int input/output
proc null0_test3*(input:cint): cint {.stdcall,exportc,dynlib.} =
echo("test3 working")
echo(input, "\n")
return 1
# string input, int output
proc null0_test4*(message:cstring): cint {.stdcall,exportc,dynlib.} =
echo("test4 working")
echo(message)
return 1
and make C aware of it like this:
void null0_test1();
int null0_test2();
int null0_test3(int input);
int null0_test4(char* message);
I get segfault on the last 2, so maybe it's input?:
null0_test1();
int a = null0_test2();
int b = null0_test3(3);
int c = null0_test4("tester");
It seems to log first on test3, so I think it is the return and/or pointer-input. What am I doing wrong?
I will try that but I'm not sure I understand this part:
or mark the C functions as __stdcall if you prefer that.
You mean where I tell C about the nim functions, or something else?
As a possibly better way, I ended up trying to solve it the other way around, by implementing all the functions that libretro looks for in nim. I tried doing it automatically with c2nim, but it didn't work at all, but it did give me a rough idea of how to implement things. I made a bunch of functions, and it seems to work, and is passing values that seem right, and the logs say things like the correct sample-rate and game-title and stuff. The only ones I am really still not getting are where retro sets a callback up. The idea is it tells you the function for "draw video" and you save that and call it when you want to update the framebuffer. Here is an example (in C):
void retro_set_audio_sample_batch(retro_audio_sample_batch_t cb) {
audio_batch_cb = cb;
}
Essentially, you make a ref globally audio_batch_cb, then later when you want to send audio-frames in a batch to the libretro host, you call that.
I did this, which seems equivalent (in my limited understanding):
type:
cbaudio_batch_t = (proc(data:array[800, int16], frames:csize_t):void) # 48000/60
var cb_audio_batch: cbaudio_batch_t
proc retro_set_audio_sample_batch*(cb: cbaudio_batch_t) {.stdcall,exportc,dynlib.} =
cb_audio_batch = cb
This segfaults.
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
[1] 71588 segmentation fault /Applications/RetroArch.app/Contents/MacOS/RetroArch -v -L justlog.wasm
This is pretty much exactly the use-case I had when developing Futhark. Essentially I was writing a dynamic library for Unbound, both DLL and so generated from the same source. The way it works is fairly simple, I have an importc block (from Futhark) which loads all the C headers from Unbound I wanted to work with. Then I have defined a dynlib pragma which adds the exportc, dynlib, and cdecl pragmas to all the procedures I need to export for Unbound to properly load my dynamic library. Then I compiled the whole thing with --app:lib and the whole thing works. But there are some caveats, most importantly that Nim has a GC which might need to be initialised (ARC doesn't require initialisation), and global variables which also needs to be initialised. This can be done by calling NimMain which can be accessed by adding proc NimMain() {.cdecl, importc.} to your code. If you're using threads you might also need to call setupForeignThreadGC in any new thread created by the host (it is safe to call it multiple times on the same thread), but again it depends on the GC.
If you're not using global variables (and no library you import uses them) and compile with the gc:arc option you will not have to call NimMain. Although if you have an initialiser procedure which the host will call exactly once then it's good practice to just put it in there anyways, it's a complete no-op if you don't need it.
Futhark looks really cool. I have already done the complicated part I was avoiding (translating all the functions & structs in a libretro plugin to nim) so I am not sure I want to auto-wrap it, at this point. I am happy with the way it's setup and it seems to run well, other than the one thing I have remaining (how to use those C callback functions.)
Am I referencing those callback functions incorrectly? I am having trouble finding examples of what I am trying to do.
Looks about right, assuming that your C wrapping is correct. It's easy to mess up in subtle ways, that's why I wrote Futhark, to get away from having to painstakingly match up every single procedure and type and ensure that the alignment, packing, calling conventions etc. all match up.
The callback types should probably also have cdecl on them, or similar. Again, Futhark should manage all that.
You also don't set up the GC, which is a must if you're using the default one. And I'm not sure why you're using stdcall and not cdecl.
Just trying passing it through Futhark with:
import futhark
importc:
path "."
"libretro.h"
And right of the bat I can see that they use cdecl, and for example the type retro_environment_t returns a bool and not void like you have it in your code. It was the first type I checked so not sure if I got "lucky" and found something that didn't match right away or if you have multiple errors like that.
that's why I wrote Futhark > You might also be interested in the genny library,
genny seems really good for my original usecase (wrap a couple nim functions for C) and futhark seems good for wrapping a C api for nim, but now I am dug into figuring this out in just nim, since I like how all the rest of the retro-core-code is in nim, now. I've got it so close, I just need to know how to do the one thing of "save a function that C host gives me to call later". It sounds like it can be subtly complicated.
And right of the bat I can see that they use cdecl, and for example the type retro_environment_t returns a bool and not void like you have it in your code. It was the first type I checked so not sure if I got "lucky" and found something that didn't match right away or if you have multiple errors like that.
This is just stubbed. I started with a single function retro_set_video_refresh, since I 100% know I will use that, and commented the rest (well, discard in function.) I just uncommented so ya'll could see what I was working with, but it's probably not a great illustration if it's non-functioning in other ways. It should be fine, even though it's not correct, though. When I first started they were all proc():void and as long as I didn't try to expose the proc to the var, It worked fine, like if I add the same function names but don't try to grab any retro callbacks. I will go through and set it all up correctly, now. I was just not bothering to write several callbacks, when I know it's not right how I am using them.
I will try cdecl. How do you view the nim code that futhark generates? It might be a good way to get some hints for how to manually those few functions. I dunno, maybe it makes more sense to to auto-wrap libretro.h for all future libretro-core-in-nim people, and just use that.
I think I have followed your advice.
In libretro.nim header:
type
retro_video_refresh_t* = proc(data:ptr UncheckedArray[cuint], width:cuint, height:cuint, pitch:csize_t):void {.cdecl.}
In core code:
proc NimMain() {.cdecl, importc.}
var buf:ptr UncheckedArray[cuint]
proc retro_init*() {.cdecl,exportc,dynlib.} =
NimMain()
var bufPtr0 = alloc0(sizeof(cuint) * WIDTH * HEIGHT)
buf = cast[ptr UncheckedArray[cuint]](bufPtr0)
log_cb(RETRO_LOG_DEBUG, "retro_init() called.")
proc retro_run*() {.cdecl,exportc,dynlib.} =
for y in 0..(HEIGHT - 1):
let index_y = uint32 bitand((y shr 4), 1)
for x in 0..(WIDTH - 1):
let b = ((y * WIDTH) + x)
let index_x = uint32 bitand((x shr 4), 1)
if bool bitxor(index_y, index_x):
buf[b] = color_r
else:
buf[b] = color_g
video_cb(buf, WIDTH, HEIGHT, (WIDTH shl 2))
proc retro_unload_game*() {.cdecl,exportc,dynlib.} =
log_cb(RETRO_LOG_DEBUG, "retro_unload_game() called.")
GC_FullCollect()
I compile with
nim c -d:release --app:lib --noMain --gc:orc --outDir=. src/example_libretro.nim
And I get checker-pattern, and no segfaults! Did I miss any steps here?
Good to hear you got Futhark installed! I realise that --passL would be required for the install, what that does is essentially tell your compiler where libclang is found so it can link against it. However when you run it the already compiled opir tool is failing to find the dynamic library. So essentially you need to use --passL when installing Futhark to tell the compiler where to find it, then you need to use LD_LIBRARY_PATH when running Futhark (or technically the helper program Opir) so that it can find and load the dynamic library. Instead of using LD_LIBRARY_PATH every time you want to use Futhark you should look at setting up your library path correctly, or symlink libclang.dynlib into your library path. Let me know if this works and then we can improve the installation section. I might also consider distributing pre-compiled libraries, this seems to affect a surprising amount of users, but that will take a while to get set up..
Your latest snippet looks much better! I did some more digging around dynamic libraries and Nim and it lead me to realise that there is no way to safely unload a Nim dynamic library when using ARC/ORC. This isn't a problem if your dynamic library lives as long as the program, but if its unloaded and reloaded continuously then memory will just keep increasing.
using LD_LIBRARY_PATH every time you want to use Futhark you should look at setting up your library path correctly, or symlink libclang.dynlib into your library path.
Ah! Yes, this makes sense.
Let me know if this works and then we can improve the installation section.
I am itching to do the thing I was originally working on (port my game engine to nim) so I am going to focus on that for a bit, but I will be sure to circle-back. Futhark seems very cool, and would definitely save a lot of time for wrapping/exposing C. I think with the libretro host the wasm-runtime and pntr ported, I just need a few other things to complete my engine (sound and some physfs functions.) I think your helpful suggestions have armed me with the knowledge to get it done! I would be totally hip to going back to Futhark and wrapping stuff that way, after I have a POC working.
I might also consider distributing pre-compiled libraries, this seems to affect a surprising amount of users, but that will take a while to get set up..
I'd be happy to help, if you like. WIth node-raylib it really cut down on installation issues, especially for windows users. Technically node requires C build-chain installed, but a lot of people never bother, since a lot of non-native stuff works without it. Linux users seem more comfortable with getting it all working, but also it can save a lot of time on pi, for example, where compile takes a while. That probably doesn't apply here, but might help people get going faster. arm was a bit tricky in GH CI because it only runs x86_64, but I setup qemu/docker to build for other CPUs.
but if its unloaded and reloaded continuously then memory will just keep increasing.
This is a great general gotcha, but I think here it is ok, similar to how libretro cores normally work, right? There should be only 1 load/unload (when the core is initialized and deinitialized by host) and the whole DLL is wiped in the process, so I think maybe the libretro host kind of takes care of that, but I could be misunderstanding.
There should be only 1 load/unload (when the core is initialized and deinitialized by host) and the whole DLL is wiped in the process, so I think maybe the libretro host kind of takes care of that, but I could be misunderstanding.
The problem is that when a dynamic library is unloaded the memory it allocated won't get automatically freed. This isn't done until the program itself quits. I've created an issue to track this, once that is implemented it should be completely safe to load and unload dynamic libraries written in Nim.
I came back to trying to use futhark on mac. I solved my original issue (installing/running it) with:
ln -s "$(xcode-select --print-path)/usr/lib/libclang.dylib" /usr/local/lib/libclang.dylib
But it still doesn't work for me. I took your pntr wrapper code above, and used a git submodule in vendor/pntr. I realized it's relative, so I changed the wrapper to look like this:
importc:
define "PNTR_PIXELFORMAT_RGBA"
define "PNTR_IMPLEMENTATION"
renameCallback renameCallback
path "../vendor/pntr"
"pntr.h"
And I get this error:
Fatal: 'stdlib.h' file not found
Tried to parse: /Users/konsumer/.cache/nim/nimterop_ideas_d/futhark-includes.h
With arguments: -I/Users/konsumer/Desktop/nimterop_ideas/vendor/pntr -I/Library/Developer/CommandLineTools/usr/lib/clang/14.0.0/include
Seems like it's screwing up something with my paths (it should definitely be able to find stdlib.h.) Again, I build lots of stuff in nim/C fine on this machine.
Do I need to do something else to make it work?
Ok, that worked, but the binding is still off. If I run this it outputs an empty image:
import pntr
var canvas = new_image(80, 80)
draw_circle(canvas, 40, 40, 30, RED)
discard save_image(canvas, "demo.png")
I did color-defs like you did:
const LIGHTGRAY * = color(anon0: colorAnon0T(r: 200.char, g: 200.char, b: 200.char, a: 255.char))
and get ` Error: undeclared identifier: 'colorAnon0T'` searching through generated code I found this:
type
unioncolor_anon0_t {.pure, inheritable.} = object
r*: cuchar
g*: cuchar
b*: cuchar
a*: cuchar
So I used defs like this:
const RED * = color(anon0: unioncolor_anon0_t(r: 230.cuchar, g: 41.cuchar, b: 55.cuchar, a: 255.cuchar))
and this compiles and runs without error, but yields an empty image. Defnitely seems closer!
Hmm, it's strange that you're getting these issues. If I where to guess though Nim doesn't like union types in const. Try defining them as let like I did and see if that helps.
And as of LOC I'm not surprised at all. Futhark, much like Nim itself, isn't made to make readable code. All those lines of code just contribute to creating a correct wrapper. You could also look at it as your hand-made wrapper missing a bunch of important lines. With that said some of the features of Futhark makes the code harder to read but ensures it doesn't try to redeclare existing things. You can read the Compatibility features and readability section of the documentation to see which flags you can pass for a "cleaner" output. Try with -d:nodeclguards that should make a huge change in the LOC count.
Did some testing and it was the const creating the issue. I did some minor fixes of Futhark so you should now be able to just require "futhark >= 0.9.0" in your nimble file and use -d:nodeclguards. Not quite sure why your types have an appended union to them, for me this doesn't happen. Also changed from cuchar to uint8 since the format is deprecated in Nim. This means that your colours are now:
let
LIGHTGRAY * = color(anon0: color_anon0_t(r: 200, g: 200, b: 200, a: 255))
GRAY * = color(anon0: color_anon0_t(r: 130, g: 130, b: 130, a: 255))
DARKGRAY * = color(anon0: color_anon0_t(r: 80, g: 80, b: 80, a: 255))
YELLOW * = color(anon0: color_anon0_t(r: 253, g: 249, b: 0, a: 255))
etc.
Of course you can clean this up with a procedure, there is a reason why it's declared as a union though.
With -d:nodeclguards the LOC count for the wrapper drops to 447 and that is mostly because Nim tries to pretty-print it.