could not import: FreeImage_Initialise
I think that this is happening because the symbols in the DLL look like this:
_FreeImage_HasPixels@4 _FreeImage_HasRGBMasks@4 _FreeImage_Initialise@4 _FreeImage_InsertPage@12
If I change the 'importc' to use the name '_FreeImage_Initialize@4', that import succeeds... but I can't help but think that's not what I'm supposed to do.
Does anybody know why the symbols in the FreeImage DLL have these leading underscores and trailing numbers: is the wrapper using the wrong calling convention? If so, what calling convention is this?
[Update:] According to Microsoft's website, this is stdcall calling convention. The nim manual lists stdcall as one of the available options, so that suggests that nim supports this. But the wrapper already has a stdcall pragma at the top, and putting 'stdcall' into the individual wrapper functions doesn't seem to have any effect either. What am I doing wrong?
That name mangling happens for C functions that are declared as stdcall. So most likely that is why the names are that way. If you add a stdcall pragma to go along with the importc, then the name mangling should get taken care of for you so that you can use the plain name in your Nim code.
I haven't used the FreeImage library though, so it's possible that there's something else going on there. In fact, I would expect that if you had the calling convention wrong to begin with, then you would hit errors fairly quickly when you started using it, so I wonder if the author of the wrapper was using different DLLs than you are working with and the calling convention was changed for some reason?
I believe this name decoration is correct for stdcall. Search for "Microsoft MSDN stdcall", the official text you'll find is:
"Name Decoration Convention: An underscore (_) is prefixed to the name. The name is followed by the at sign (@) followed by the number of bytes (in decimal) in the argument list. Therefore, the function declared as int func( int a, double b ) is decorated as follows: _func@12"
I've read the code for the nim compiler, there's definitely no code in there to do this sort of name decoration. Insofar as I can tell, nim simply does not support DLLs compiled with stdcall. It supports static libraries compiled with stdcall by letting the C compiler do the name decoration.
This should be documented in the nim manual: stdcall is only supported for static linking.
Update: Windows 'GetProcAddress' doesn't do name decoration either. It is possible, when building the library, to use the 'def' file to override the default function name (and hence, get rid of the underscore and at-sign). I am starting to think that's what many library authors do to make it easier to fetch the symbols in the library.
Insofar as I can tell, nim simply does not support DLLs compiled with stdcall.
No, that's not correct. We've done quite a bit of work with Nim creating .DLLs / calling .DLLs that use stdcall calling convention. Nim works quite well for that.
It's not Nim's job to do the name decoration though. If you poke around through different mailing lists and forums, you'll see that it is a source of headaches for quite some time, and for all languages that do some C level interfacing (not just Nim).
What we do for dealing with this (when we are working with a .DLL did not use a .DEF file for their imports) is take advantage of Nim's compile time support for doing calculations and generate a const with the function name that is desired and use that const in the {.importc.} pragma, instead of having the function name as a string in the pragma. The const is calculated at compile time with some logic that decides if the name decoration is needed or not and sets the value appropriately.
For example, one part of the calculation is whether we are compiling for 64 bit or not, because name decoration is not used at all (even by MSVC) for 64 bit DLLs.
Another example calculation is related to the version of a .DLL we are importing that we did not have the source for. The newer versions of the .DLL had started using .DEF files for defining their exports, and we needed to be able to work with the newer ones and the older ones. It would be nice to not have to deal with strange scenarios like that, but that is what happens in the real world sometimes :) , so Nim's flexibility actually is an advantage in dealing with that.
Is there an example of how to do this somewhere? I'd like to update the freeimage wrapper to contain the necessary code.
This is my first take (I know this isn't quite legit nim syntax, I haven't gotten it memorized yet):
const decorated = "_FreeImage_Allocate@" & $(sizeof(cint) + sizeof(cint) + sizeof(cint) +
sizeof(cuint) + sizeof(cuint) + sizeof(cuint))
proc FreeImage_Allocate*(width: cint; height: cint; bpp: cint;
red_mask: cuint; green_mask: cuint; blue_mask: cuint): ptr FIBITMAP {.
importc: decorated, dynlib: FreeImageLib.}
That's okay, but a little wordy. I could certainly use macros to wrap that in some kind of code generator... is that what I should do?
when decorating_names:
const
HasPixels_Import = "_FreeImage_HasPixels@4"
else:
const
HasPixels_Import = "FreeImage_HasPixels"It's a bit of extra work to maintain, but we only have a few places where we need this, so it wasn't too bad.
I like the idea of coming up with something to do the calculation though based on the signature. You can do more sophisticated logic than what I have above at compile time to come up with the import name. Take a look at this example from NimBorg, which dynamically determines which Python .DLL to import.
https://github.com/chrisheller/NimBorg/blob/master/nimborg/py/python_settings.nim
That gets used in the import pragma like {.dynlib: getPythonDllName().}
If there was a macro or compile-time proc that could calculate the appropriation decoration for an import based on the proc signature, then you let it handle the details for you. You'd still need to have the proc signature correct for that to work, but might save some effort if there was a lot of it going on.
proc nimGetProcAddr(lib: TLibHandle, name: cstring): TProcAddr =
result = getProcAddress(cast[THINSTANCE](lib), name)
if result != nil: return
for i in countup(0, 50):
var decorated = "_" & $name & "@" & $(i * 4)
result = getProcAddress(cast[THINSTANCE](lib), cstring(decorated))
if result != nil: return
procAddrError(name)
Yes, I know that's a terrible hack. But it does have the strong point that it keeps the ugliness concentrated in a very small section of code... the entire rest of the system can then deal in undecorated names. And it doesn't affect any library that worked without the decoration.
[Update:] I've made some corrections to the code above, and it now works very well.
I am the author if the freeimage wrapper, and it "works on my machine":). I typically build the freeimage dlls from source using the same compiler as I'm using for nim. Tonight I'll try and look at the function names are in my copy of the dll and see what's up. I know free image's build system has changed a bit somewhat recently.
Can you report your system configuration (freeimage version, os, and compiler)?
[Update: I think you shouldn't try to merge this pull request. Although I did manage to create a github repository which is a fork of the nim repository, I don't think I did it the normal way.... I think github is confused. I would not touch this pull request.]
Okeydokey, Araq: I finally learned enough git to do this.
Here's that pull request:
https://github.com/jyelon/nim-lang/pull/1
I should caution: this is my very first use of git, and it's also my very first nim contribution, so even though it's quite simple, you should actually look at the code and hopefully run some unit tests to make sure that DLL loading is still fully operational.
Hey @jyelon. I fear that your PR does not work this way. I am not sure how you got the Nim-Lang Repo into your Github account but I think it is not a "fork" of the original repo.
You need to "fork" (press fork in github) in the original Nim-Lang Nim repo: https://github.com/nim-lang/Nim
After that you "git clone" your fork onto your computer: git clone <yourforkuri>. Then you checkout devel on your computer: git checkout devel. Then you create a branch for your changes: git checkout -b yourbranchname. After you made those changes you add/commit them (guess that is known by you now). Then you need to push your changes to your repository fork on github: git push -u origin yourbranchname. After that go to your github site. You will see that it offers to create a pull request! There you select as base-fork the "nim-lang/Nim" and the "devel" branch and as head-fork "your/Nim" and "yourbranchname".
if you ever want to continue working or creating new PRs you need to keep your forked version updated.
For this you need to add the "upstream" location (once): git remote add upstream https://github.com/nim-lang/Nim. After this you checkout devel (or make sure it is checked out) and "pull devel from upstream" to your "computer": git checkout devel and git pull upstream devel. After this you "push" the "original" upstream data to your github fork of it: git push origin devel. Then you are up to date locally and in your forked repository and can continue working on new stuff :)
There are other ways to do this but this way you have also already the updated version on your computer.
Zowie. :)
With each iteration, I think I've been getting closer and closer to that procedure. But it's nice to have it laid out. Saves me a lot of trouble. :)
OK, I think I'm a lot closer:
https://github.com/nim-lang/Nim/pull/3536
But one of the integration tests is failing:
https://ci.appveyor.com/project/Araq/nim/build/1.0.204
Looking through the output, I see some lines that contain the word "Fail", but I'm vaguely suspicious because these failures don't look like they have anything to do with my code. Of course, I could be wrong.
I guess my first question would be: is there a way to see the output from a non-failing version of the AppVeyor CI test? I'd like to see what "correct" output is supposed to look like.
My second question would be: where can I find the source code for the particular test case that failed?
[Update: I created another branch, with nothing but an edited comment, and the same test failed. So clearly this is just a broken test. Is there somewhere I can go to see the current set of broken tests?]
I really appreciate all the help. Thank you, this would have been much harder without your assistance.
So OK, then, Araq: here's the final Pull Request with the workaround for the stdcall name decoration issue:
Araq just merged this change.
For the authors of Nim DLL wrappers, the upshot is this: you no longer need to think about stdcall name decoration. The Nim runtime will take care of it for you.