Hi all,
I'm trying to create a Ruby extension in Nim, so I've built a Nim dynamic library (.so) that exports a number of functions, which I'm then accessing using Ruby's FFI module. What I've found, is that I can easily use functions that return ints or floats, but cannot use functions that return strings or arrays. To illustrate:
## utils.nim
proc test_int(): int {.cdecl, exportc} =
let res: int = 1
return res
proc test_string(): string {.cdecl, exportc} =
let res: string = "result"
return res
proc test_arr(): array[3, int] {.cdecl, exportc} =
let list = [3,4,5]
return list
I build this file with:
$> nim c --cc:gcc --d:release --threads:on --app:lib utils.nim
Which is successful, and I can verify that my functions have been exported with
$> nm -D libutils.so
I can then call test_int from my Ruby code and have the integer returned but when I call test_string or test_arr, I get the following error:
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
I've used the Ruby FFI before to call C-written libraries that return arrays or char* without any problem, so I don't think the problem is on that side. Any ideas on what I'm doing wrong would be greatly appreciated.
PS: I'm using Nim 0.17.0 (2017-05-17) [Linux: amd64]
I'm trying to create a Ruby extension in Nim,
Interesting, maybe you can collect your insights in a blog or forum post.
I have used Ruby too in the past, and I wrote some C extensions for using CGAL and Boost library. While I was told that extending Ruby with C was easier than extending Python with C, it was some hard work, mainly due to the CGAL and Boost C++ template stuff and due to Ruby GC.
So I am happy that I can do all in Nim now. Well, for Ruby I got the GTK3 bindings for free, and I still may need CGAL and Boost support...
To your question, returning Nim strings and using that outside of Nim can not work out of the box. Nim strings are garbage collected, so when there is no longer a reference to a Nim string in the Nim universe, than that string is freed by the GC. (I guess for your array example it is similar.). We have various possibilities to solve that problem. 1. You can call GC.ref() on the returned Nim string, so that CG can not collect it. 2. You can completely turn of GC. 3. You may compile your program without GC support. 4. You may copy the Nim string to a C string by calling alloc() to create a untraced memory location on the heap, copy the string to this location and return that location.
And, I think you do not want to return a Nim string at all, as it is not identical to a cstring, there is at least the length field that C does not have. Maybe, when you want to access the Nim string from outsite Nim, you better return addr(str[0]).
@Stefan_Salewski, @mashingan: thank you both, your suggestions were great. I was under the misapprehension that the Nim primitives were C-like. So now, using cstring instead of string means the test_string function works smoothly! For the array function I am now returning a ptr, which I can access on the other (Ruby) side without errors, but when I de-reference it I get a list of meaningless numbers
proc test_arr(): ptr array[3, int] {.cdecl, exportc} =
let
a: int = 3
b: int = 4
c: int = 5
var list = [a, b, c]
return addr list
I have tried packaging the list items using different representations (int32, uint16, etc) and matching these when I read the array on the receiving side, but still keep getting random numbers. I suspect this time the problem is in the way I unpack the array on the client side, so I need to look a bit more into that.
PS Stefan, yes, as soon as I get this going I intend to write a blog post about it. As per the GC, as long as I'm using ptrs I shouldn't need to worry about it as ptrs are not tracked so I can just release the memory from the Ruby side when I'm done. Thanks though, as I hadn't considered the Nim GC impact before, so that's definitely something to think about.
So now, using cstring instead of string means the test_string function works smoothly!
Can you show the example? I was not aware that Nim can create cstrings for FFI access out of the box.
but still keep getting random numbers.
Well, your example can not work. Returning the addr should not make it much better I guess.
Your list is allocated somewhere -- on the stack or on the heap garbage collected. In both cases is it not valid any more after return of that proc. I gave some suggestions for solution in my previous post.
Of course all what I say has no solid foundations -- only Araq known exactly what the compiler will do, as the compiler and Araq are always free to try to guess what the user really intents and to try to give him what he may wants :-)
Do you also consider freeing the returned memory somehow? Another approch may be to allocate the needed memory in the Ruby world and pass a pointer to that memory to Nim, where Nim can fill in the data.
I was not aware that Nim can create cstrings for FFI access out of the box.
I think you missed reading cstring definition in manual. It's stated there, especially using C backend.
Hi @mashingan, @Stefan_Salewski
I find it impossible to allocate memory for array, as when I try to do something like:
var p_arr = alloc(size_of(array[int, 3]))
I get a 'type expected' error.
Besides, if array is created on the stack then passing a pointer to it is meaningless anyway. So I have tried with a sequence instead,
proc test_arr(): ptr seq[int] {.cdecl, exportc} =
var list = newSeq[int](3)
list[0] = 4
list[1] = 5
list[2] = 6
GC_ref(list) # mark it as referenced
return addr list
In theory, this should work: the sequence goes on the heap, I'm preventing it from being swept by the GC and I'm returning a pointer to the client. But it doesn't, I'm still getting rubbish numbers on the client side I have tried a number of variations on this theme but no luck :(
Any suggestions / ideas will be very welcome
var p_arr = alloc(size_of(array[int, 3]))
Do
var p_arr = alloc sizeof(array[3, int])
Also pointer arithmetic post should be helpful
When you alloc something, it'll be put in heap, and untracked, so you need to call realloc when you done with it.
If you're going to have a different thread, use shared variant like allocShared and reallocShared
RedFred, note that
proc test_arr(): seq[int] {.cdecl, exportc} =
...
return list
may work better, as a seq is already a pointer. You don't want a pointer to that pointer. Of course the seq is more than a plain array internally, there is at least the size member and the capacity. I don't know the exact layout. Maybe try "return addr list[0]"
And note the typo of the cat, it should be "dealloc" of course.
I hope you buyed Dom96's book -- maybe in the final release appearing at the end of this month all this is described in detail.
ok, I finally resolved this using tuples in the following way:
type
MyData = tuple[x, y, z: int, s: cstring]
proc new_list(data: ptr MyData): ptr MyData {.cdecl, exportc} =
var data = cast[ptr MyData](alloc0(sizeof(MyData)))
data.x = 111
data.y = 222
data.z = 333
data.s = "nim rules!"
return data
proc destroy_list(data: ptr MyData): void {.cdecl, exportc} =
dealloc(data)
On the Ruby side I'm using the FFI to map the tuple to a struct-like object, making sure I get the byte offsets right (receiving pointer has offset 0)
class MyObject < FFI::Struct
layout :data1, :int, 0,
:data2, :int, 8,
:data3, :int, 16,
:name, :pointer, 24
I can now use Nim-generated data from Ruby, making sure I release the structure's memory when I'm done
my_data = NimDemo::MyObject.new(NimDemo.new_list)
puts my_data[:data1]
puts my_data[:data2]
puts my_data[:data3]
puts my_data[:name].read_string
# finished with it, so release it
NimDemo.destroy_list(my_data)
That means I can now pass a list of numeric values and strings (or combinations thereof) from Nim to Ruby. Which is pretty tidy :-)
Many thanks to @mashingan and @Stefan_Salewski for steering me in the right direction!
Great! :)
If you have some time to spare, it would be nice if you document or write your process to blog, it would be helpful for everyone ;)