I used Nim a lot quite a few years ago but have drifted away from it as most of what I needed was scientific stuff like arbitrary precision floating point numbers and matplotlib style graph plotting. However, I recently started needing to write some C++ code to integrate with an existing API and after a short period of getting frustrated with the verboseness of doing things in C++, I thought I'd give Nim another shot.
I used c2nim to convert the provided header file into a Nim file. I had to do a few tweaks before and after to get the basics working as I expected (replacing array[n, char] with cstring and fixing a very odd thing where a value in an enum was mistranslated from 0x00000010 to 0x0000000000000000'i64). That allowed me to do some very basic calls to the API functions, but I've been fighting calling functions needing arrays and pointers and have found the documentation hard to follow. Can anyone help please?
Unfortunately, the original header file for the library I'm using includes this in the header: "Proprietary and Trade Secret", so it's probably best that I don't share the details of the file. It's the API for a 3D CAD system (ZW3D Lite). I'm sure I can include snippets if there are particular bits that are of interest.
My source file is below (I can also share the full working C++ version if that helps, although that's got a bit more functionality in). I've included comments showing where some of the problems are. Some are related to difficulties in translating from C arrays to Nim arrays, some are (weirdly) related to translating from unsigned 8-bit integers (150'u8) to unsigned 8-bit integers (cuchar) in order to get unsigned 8-bit integers (unsigned char). I really don't get why that latter case needs a cast.
After a lot of messing around, it now compiles and runs, but there's still a problem with the return code. However, I'm struggling to think how to debug that and I doubt anyone here can help without a copy of ZW3D. I'm hoping that if I can fix the other issues, this one will become more solvable.
Can anyone help me with this? I was hoping Nim's much-talked-about FFI would make this a pleasure, but it feels like I'm writing considerably more code in Nim than I was writing in C++ and if it's going to be like that I think I'll just go back to C++. I'm hoping it's just because I'm very new to the FFI and I'm missing something significant (and failing to find the right documentation that explains pointers and arrays easily).
import strformat
import VxApi
proc ColourCycleNim(idData: cint): cint {.stdcall,exportc,dynlib.} =
var
entCount: cint
retCode: int
# This feels wrong (and may well be part of the issue) - I tried declaring this
# as:
# var idEnts: UncheckedArray[ptr cint]
# but that failed with:
# invalid type: 'UncheckedArray[ptr cint]' for var
var idEntsPtr: ptr ptr cint
retCode = cvxDataGetEnts(idData, 1, addr entCount, idEntsPtr);
if (retCode != 0):
# This gets called - there's clearly something wrong with the call
# above, although I'm not sure what!
cvxMsgDisp(fmt"Failed to get entities: return code is {retCode}")
return 1
var idEnts = cast[ptr UncheckedArray[ptr cint]](idEntsPtr)
# The above code is trying to replicate this (which works) in C++:
#
# int entCount;
# int *idEnts;
# int retCode = cvxDataGetEnts(idData, 1, &entCount, &idEnts);
#
# idEnts is a list of entities; entCount is how many there are
#
# Original call signature (pre c2nim):
# extern int cvxDataGetEnts(int idData, int idField, int *Count, int **idEnts);
# Post c2nim:
# proc cvxDataGetEnts*(idData: cint; idField: cint; Count: ptr cint; idEnts: ptr ptr cint): cint {.
# cdecl.}
if entCount == 0:
cvxMsgDisp("Got 0 entities")
return 1
cvxMsgDisp(fmt"Got {entCount} entities")
for i in 0 ..< entCount:
var colourStruct: svxColor
# In VxApi.h, svxColor is a structure containing three unsigned chars
# (which are, by definition, unsigned 8-bit integers). These are translated
# by c2nim into cuchar. However, 150'u8 (which is an unsigned 8-bit integer)
# cannot be applied to colourStruct.r (which is an unsigned 8-bit integer)
# without a cast. This seems a bit odd. Error message (without the cast) is:
#
# type mismatch: got <uint8> but expected 'cuchar = char'
#
colourStruct.r = cast[cuchar](150'u8)
colourStruct.g = cast[cuchar](70'u8)
colourStruct.b = cast[cuchar](8'u8)
retCode = cvxEntRgbSet(colourStruct, 1, idEnts[i])
if retCode != 0:
cvxMsgDisp(fmt"Setting RGB Colour failed at index {i}")
return 1
return 0
proc ColourCycleNimInit(format: cint, data: pointer): cint {.stdcall,exportc,dynlib.} =
discard cvxCmdFunc("ColourCycleNim", ColourCycleNim, VX_CODE_GENERAL)
return 0
proc ColourCycleNimExit(): cint {.stdcall,exportc,dynlib.} =
discard cvxCmdFuncUnload("ColourCycleNim")
return 0
Have you tried unsafeAddr entCount instead of addr entCount? I always struggle with this, so probably others can help better than me.
Regarding svxColor, I would say that one step is the wrapping and another one is making the API user friendly. I would probably do something like the following:
type
svxColor = object # This would be your wrapped code
rr:cuchar
gg:cuchar
bb:cuchar
proc `r=`(obj:var svxColor, val:uint8) =
obj.rr = val.cuchar
proc r(obj:svxColor):uint8 =
obj.rr.uint8
var colourStruct:svxColor
colourStruct.r = 150'u8
echo colourStruct.r
here's how i'd choose to do it:
import strformat
proc cvxDataGetEnts*(idData: cint; idField: cint; Count: var cint; idEnts: var ptr UncheckedArray[cint]):cint {.importc,cdecl.}
proc cvxMsgDisp(msg:cstring){.importc,cdecl.}
type svxColor = object
r*,g*,b*:cuchar
proc cvxEntRgbSet*(colorStruct: svxColor; idField: cint; idEnt: cint):cint{.importc,cdecl.}
proc ColourCycleNim(idData: cint): cint {.stdcall,exportc.} =
var
entCount: cint
idEnts: ptr UncheckedArray[cint]
if cvxDataGetEnts(idData, 1, entCount, idEnts)!=0:
cvxMsgDisp("Failed to get entities")
return 1
if entCount == 0:
cvxMsgDisp("Got 0 entities")
return 1
cvxMsgDisp(fmt"Got {entCount} entities")
for id in idEnts.toOpenArray(0,entCount):
#we can iterate over idEnts just as we would a seq or array
#now 'id' isn't the index, but the idEnt itself
var colourStruct:svxColor
# In VxApi.h, svxColor is a structure containing three unsigned chars
# (which are, by definition, unsigned 8-bit integers). These are translated
# by c2nim into cuchar. However, 150'u8 (which is an unsigned 8-bit integer)
# cannot be applied to colourStruct.r (which is an unsigned 8-bit integer)
# without a cast. This seems a bit odd.
#
# as you can see int literals are converted to cints but apparantly that's not
# the case for cuchar. maybe an oversight. but you don't need a cast:
colourStruct.r = 150.cuchar
colourStruct.g = 70.cuchar
colourStruct.b = 8.cuchar
if cvxEntRgbSet(colourStruct, 1, id) != 0:
cvxMsgDisp(fmt"Setting RGB Colour failed for id {id}")
return 1
return 0
if we check the code we can see
N_CDECL(int, cvxDataGetEnts)(int idData, int idField, int* Count, int** idEnts);
...
idEnts = (int*)0;
...
T3_ = cvxDataGetEnts(idData, ((int) 1), (&entCount), &idEnts);
if you need any more help give us a shout, I'm also hireable if this is a paid gig :)
Thanks @shirleyquirk, that's really useful and a lot cleaner than what I had before. I'm obviously going to have to do a lot more post-c2nim editing of the functions I want to use (there are 836 functions in total so I'll definitely look to see how automate-able it is though - it should be okay as the naming and type of the various arguments seems quite consistent).
I'm not quite there at the moment (I hadn't listed all the function definitions for the functions I'm using and hence you had to make some assumptions, but the assumptions weren't quite right).
I think I'm getting myself a bit confused with what the actual mapping between all the different data types works.
We have:
int **idEnts - in C in function cvxDataGetEnts - used to retrieve a pointer to a C-allocated list of ints int *idEnts - in C in function cvxEntRgbSet - used to pass a list of ints to C
c2nim maps these as:
int ** idEnts -> idEnts: ptr ptr cint int *idEnts -> idEnts: ptr cint
In your code, you've mapped the former to var ptr UncheckedArray[cint]. This feels odd to me (as someone familiar with C but unfamiliar with the FFI or Nim pointers) as I read var as being effectively a pointer (behind the seems), as is UncheckedArray, which makes var ptr UncheckedArray into *** not **. I'm sure that's down to my misunderstanding though.
The next issue is how to get the id from your code back into an UncheckedArray. The function cvxEntRgbSet takes a list of IDs (of length 1 in this case as the second parameter is the length)†.
This is the c2nim declaration for the second one:
proc cvxEntRgbSet*(Color: svxColor; Count: cint; idEnts: ptr cint): cint {.cdecl.}
To try to follow the way you've written your modification of my code, I've manually changed it to this:
proc cvxEntRgbSet*(Color: svxColor; Count: cint; idEnts: UncheckedArray[cint]): cint {.cdecl.}
but now of course I don't know how to call it!
Thanks again for your help, it's really appreciated. Oh, and sorry - this is just hobby stuff, not a paid gig!
† Obviously for this code I could get rid of the loop and call cvxEntRgbSet(colourStruct, entCount, idEnts), but when I finally get the C FFI bit working, each entity will be assigned a different colour, hence the loop.
Current version of my code:
import strformat
import VxApi
proc ColourCycleNim(idData: cint): cint {.stdcall,exportc,dynlib.} =
var
entCount: cint
idEnts: ptr UncheckedArray[cint]
retCode: int
if cvxDataGetEnts(idData, 1, entCount, idEnts) != 0:
# This gets called - there's clearly something wrong with the call
# above, although I'm not sure what!
cvxMsgDisp("Failed to get entities")
return 1
if entCount == 0:
cvxMsgDisp("Got 0 entities")
return 1
cvxMsgDisp(fmt"Got {entCount} entities")
for id in idEnts.toOpenArray(0, entCount-1):
var colourStruct: svxColor
colourStruct.r = 150.cuchar
colourStruct.g = 70.cuchar
colourStruct.b = 8.cuchar
if cvxEntRgbSet(colourStruct, 1, id) != 0:
cvxMsgDisp(fmt"Setting RGB Colour failed at index {i}")
return 1
return 0
proc ColourCycleNimInit(format: cint, data: pointer): cint {.stdcall,exportc,dynlib.} =
discard cvxCmdFunc("ColourCycleNim", ColourCycleNim, VX_CODE_GENERAL)
return 0
proc ColourCycleNimExit(): cint {.stdcall,exportc,dynlib.} =
discard cvxCmdFuncUnload("ColourCycleNim")
return 0
Relevant extracts from VxApi.nim with tweaks included:
{.push importc, header: "VxApi.h", hint[XDeclaredButNotUsed]: off.}
# snip
svxColor* {.bycopy.} = object
r*: cuchar
g*: cuchar
b*: cuchar
# snip
proc cvxMsgDisp*(Text: cstring) {.cdecl.}
# snip
proc cvxDataGetEnts*(idData: cint; idField: cint; Count: var cint; idEnts: var ptr UncheckedArray[cint]): cint {.
cdecl.}
# snip
proc cvxEntRgbSet*(Color: svxColor; Count: cint; idEnts: UncheckedArray[cint]): cint {.cdecl.}
# snip
That makes it quite a lot clearer in my head, I think.
I'm now passing idEnts: ptr UncheckedArray[cint] to a function expecting var ptr UncheckedArray[cint]. The function is return a pointer to an array of integers into that, so idEnts becomes a pointer to an unchecked array (a pointer to the first element in an array of integers). It does, however mean that either var ptr is a tautology or ptr UncheckedArray is (in my understanding), but I guess that's just Nim syntax. I interpreted UncheckedArray as being the equivalent of a C int *, which would imply ptr UncheckedArray is the equivalent of int **, but that would make var unnecessary.
I think what I expected to do was have a variable defined as an (undefined length, and hence probable "Unchecked") array of integers and to pass that to a function expecting with var in its argument definition. The keyword ptr wouldn't have appeared anywhere. I then would have passed a similar array to the following function, which would differ in its definition by not having var in its argument list.
I think what I need to do is treat ptr UncheckedArray as one word in my head! I clearly can't define a variable as var UncheckedArray[cint] anyway as I get a confusing error message if I do that.
Anyway, regardless of all that, I now need to create a new array to pass it to cvxEntRgbSet, which I can presumably do with something like this:
var idArray: array[1, cint] = [id]
I've added ptr to the definition of cvxEntRgbSet so that ptr UncheckedArray is back (but no var as it's int * instead of int ** this time and hence isn't mutable):
proc cvxEntRgbSet*(Color: svxColor; Count: cint; idEnts: ptr UncheckedArray[cint]): cint {.cdecl.}
but I need to convert that array into a "pointer to an unchecked array" (ptr UncheckedArray[cint] or int *) to call the C function. How do I go about doing that?
Or am I wrong and I need to leave the cvxEntRgbSet parameter as idEnts: UncheckedArray[cint] and revisit my (lack-of) understanding of how all these types relate?!
which would imply ptr UncheckedArray is the equivalent of int **, but that would make var unnecessary.
For proc parameters a var is similar as a (star) pointer. C needed pointers to modify passed values, Pascal languages used var instead. I think I wrote about that topic in my book?
Have you considered hiring someone to make the wrapper? I think I would, as making wrappers is not that much fun. And we have a few experts for making wrappers, Shaslik from IRC is one of them.
In Nim var ptr something is similar to ptr ptr something, with var ptr something you can modify the pointer itself, not only the pointed entity.
So is "UncheckedArray" not equivalent to a pointer in any way? If it is then what you've said would imply var ptr UncheckedArray[int] would be int ***, not int **.
Have you considered hiring someone to make the wrapper?
This is just for hobby stuff for me, so I can't really justify doing that. I'd love to be able to script the CAD system with a higher-level language like Nim, but if it came to "brass tacks", I'd have to just stick with C++.
So is "UncheckedArray" not equivalent to a pointer in any way?
I dont think so, I think it is just an array but without bound check, I think I have never used UncheckedArray as its own, but mostly ptr UncheckedArray, which is then a pointer to an array, but with bound checking disabled. But as my memory is not that good, I generally check manual to be sure.
just for hobby stuff for me,
Exactly in this case hiring someone may make sense. When you learn and do all the ugly wrapping stuff yourself, you may be very old when you are done. See my example with gintro, spend 5 years for it, and now I start transferring Ruby code to Nim, which was my intent when I started with Nim in 2015 :-). I would not do gintro again, but hire a smart rust and gtk core dev, at least now that rust has so good gtk support. Indeed we still intend that, when someone will pay the devs. Mozilla could do, they have a lot of money.
oof 836?
ok, well no time for niceties then.
ptr array[3,T]/ptr UncheckedArray[T] yep turns out it gets emitted as T* to help with exactly this use case. and you're correct that var is a hidden pointer. but yes, that's super confusing, and not something to be relied on without double checking the emitted code.
for an automated solution ptr ptr int as c2nim has done will always work, it's just harder to work with in the user code, hence my attempt to add some reasonable Nim semantics on top of the ptr ptr madness.
When data is coming into Nim it makes it easier if you sprinkle some 'var' and 'UncheckedArray' around, but when it's going into C, it's easier to use pointers.
So I'd keep cvxEntRgbSet the way c2nim defined it:
proc cvxEntRgbSet*(Color: svxColor; Count: cint; idEnts:ptr cint): cint {.cdecl.}
for id in idEnts.toOpenArray(0, entCount-1):
var colourStruct: svxColor
colourStruct.r = 150.cuchar
colourStruct.g = 70.cuchar
colourStruct.b = 8.cuchar
#maybe the output is a seq:
#var outArray:seq[cint] = @[id]
#or an array:
var outArray = [id]
#or use the original
#var outArray = idEnts
#either way you pass it like this
if cvxEntRgbSet(colourStruct, 1, outArray[0].addr) != 0:
cvxMsgDisp(fmt"Setting RGB Colour failed at index {i}")
return 1
In C arrays have two forms, there's the int x[5] and then when you pass it, somehow it's a int* so you could think of it as that UncheckedArray[T] has the first semantics when translated to C while a ptr UncheckedArray[T] has the second.
I want to stress that this discrepancy is not present within Nim code, in Nim code the semantics are consistent; ptr T is always a pointer to T, and dereferencing a ptr array[3,T] gives you an array[3,T]
Thank you again @shirleyquick - that's incredibly useful and it feels like it's all making a lot more sense to me now.
I've now written a python script that takes advantage of the consistent naming and parses the (well-formed) nim code generated by c2nim and generates a wrapper for some of the functions. At the moment it only handles:
However, handling just those parameter types is enough to automatically generate wrappers for 301 out of the 836 functions, which should keep me going for a while. I'll probably have to add more handlers as things go along, but I can just deal with each case as it comes along.
Most importantly, it works (for now!). I'd welcome any comments on things I'm doing that don't look right or are not best-practice.
Some relevant extracts from the generated code:
{.push hint[XDeclaredButNotUsed]: off.}
import sequtils
import VxApi
# int version of svxColor* {.bycopy.}, which contains cuchar variables
type
rgbColour* {.bycopy.} = object
r*: int
g*: int
b*: int
# snip
# ==================================================
# Wrapper for cvxDataGetEnts
proc zwDataGetEnts*(idData: cint; idField: cint; idEnts: var seq[cint]): cint =
var zw_entCount: cint
var zw_idEntsPtr: ptr cint
var ret = cvxDataGetEnts(idData, idField, addr zw_entCount, addr zw_idEntsPtr)
var zw_idEnts = cast[ptr UncheckedArray[cint]](zw_idEntsPtr)
idEnts = zw_idEnts.toOpenArray(0, zw_entCount-1).toSeq()
return ret
# snip
# ==================================================
# Wrapper for cvxEntRgbSet
proc zwEntRgbSet*(Colour: rgbColour; idEnts: openArray[cint]): cint =
var colourStruct: svxColor
colourStruct.r = Colour.r.cuchar
colourStruct.g = Colour.g.cuchar
colourStruct.b = Colour.b.cuchar
var ret = cvxEntRgbSet(colourStruct, idEnts.len.cint, idEnts[0].unsafeAddr)
return ret
#snip
The resulting source file for the colour changing script (now with randomisation included):
import strformat
import sequtils
import random
import ZW3DNim
import ColourArray
proc ColourCycle(idData: cint): cint {.stdcall,exportc,dynlib.} =
var idEnts: seq[cint] = @[]
if zwDataGetEnts(idData, 1, idEnts) != 0:
# This gets called - there's clearly something wrong with the call
# above, although I'm not sure what!
zwMsgDisp("Failed to get entities")
return 1
if idEnts.len == 0:
zwMsgDisp("Got 0 entities")
return 1
zwMsgDisp(fmt"Got {idEnts.len} entities")
var randomise = zwDataGetOpt(idData, 2) == 1
if randomise:
# Seed the random number generator
randomize()
var cidx: int
for idx, id in idEnts:
if randomise:
cidx = rand(colour_array.len)
else:
cidx = idx mod colour_array.len
var colour = colour_array[cidx]
var idArray = @[id]
if zwEntRgbSet(colour, idArray) != 0:
zwMsgDisp(fmt"Setting RGB Colour failed")
return 1
return 0
proc ColourCycleInit(format: cint, data: pointer): cint {.stdcall,exportc,dynlib.} =
discard zwCmdFunc("ColourCycle", ColourCycle, VX_CODE_GENERAL)
return 0
proc ColourCycleExit(): cint {.stdcall,exportc,dynlib.} =
discard zwCmdFuncUnload("ColourCycle")
return 0
type rgbColour* {.bycopy.} = object
r*: int
g*: int
b*: int
type Colour = object
r*,g*,b*,a*: uint8
I think Nim mapping cuchar to char instead of uint8 is a mistake when 'uint8' maps to 'unsigned char'. i'll see how much changing that would break.
seems crazy to me. 3 words to describe 3bytes? what's the benefit?
Yes, that's probably true - I was being lazy and trying to save myself from typing 150.cuchar - with that set up I could just type 150. I hadn't spent any time experimenting to see if it would work with other data sizes as it didn't matter. I'm guessing that I could use 150 to apply to to a uint8 and it would work - the issue I was seeing was presumably trying to apply 150 to a signed character (even though it shouldn't have been a signed character).
I've spent 25 years of my life programming C (and assembly) on tiny little microcontrollers where every byte (and sometimes bit) of memory counts; when I find myself writing code that runs on a PC that's also capable of running a 3D CAD system (the purpose of this code is for automating steps in 3D CAD model generation), I tend to get a bit lazy with the memory use!
I think Nim mapping cuchar to char instead of uint8 is a mistake when 'uint8' maps to 'unsigned char'. i'll see how much changing that would break.
It did seem strange to me given there are char, unsigned char and signed char as three different 8-bit types in the C standard - cuchar doesn't seem right for char if the code is going to work on different compilers. In the embedded code that I usually write, we have uint8_t, int8_t and char (sometimes typedefd to c8_t) to distinguish between the three C variable types. That's all MISRA safety-critical stuff, so perhaps the details are more important than they were to the Nim developers picking a mapping for cuchar.
Yes, that's probably true - I was being lazy and trying to save myself from typing 150.cuchar - with that set up I could just type 150.
I get it, and as you say, probably premature optimization, and given the issue with cuchar it's a bit tricky.
I think the best thing, since you're using cvxColor a lot in Nim code, is to define it with Nim types instead:
type cvxColor{.bycopy.} = object
r,g,b:uint8
var c:cvxColor(r: 150, g: 99, b: 98)
c.r = 254
#c.r = 257 #Error got int literal(257) but expected uint8