For those who are not aware of what it is: http://jbremer.org/x86-api-hooking-demystified/#ah-method-5 In short, a function hook is a jmp 0xF00 instruction written at the start of another function like MessageBoxA. 0xF00 would be an address of a function implemented in Nim. What happens is every time anything calls MessageBoxA - a custom function implemented in Nim would be called instead.
Does Nim support function hooking? My concern is GC here. Any thread may call MessageBoxA and this invoke code implemented in Nim. GC will not be aware of these threads ofc. Is that bad? Are there any knobs that would help in this situation?
Hi rokups, should be possible - explained here: https://nim-lang.org/docs/backends.html#backend-code-calling-nim-nim-invocation-example-from-c
I personally never tried it but it would be nice if you could share your results here...
So my understanding is that GC will not scan foreign threads (expected) and will treat anything allocated on foreign thread as "unreferenced" therefore eligible for garbage-collection. If we need something to persist for a longer time that duration of the call we have to manually ref/unref. Use of system.setupForeignThreadGc() is probably not a good idea because lifetime of external thread is not controlled by us and we cant call system.tearDownForeignThreadGc() when necessary.
Did i get this right?
A better mental model would be several GCs, one per thread. So GC on one thread would touch only the memory allocated on that thread. That said I'm not sure if using GC without initialization is valid at all, but whence it is initialized it will free everything not reachable from the stack of its thread (or thread-locals, or forced GC_ref ed roots). And yes, you'll have to somehow hook onto thread termination to cleanup, likely using some platform specific api for that. I haven't tested but maybe TLS + CPP + Destructors could do the job.
If performance is not critical you can define a thread-agnostic nim function, that doesn't use GC, and will communicate to your nim thread. Just make sure to wrap it into {.push stackTrace: off.}
Function hooking is a can of worms. There are many, many constraints to what you can do in your hook, which depend on the context of the caller and what they expect to happen.
I've never seen a real, working, useful, hook that wasn't written in plain C (even if it was compiled in a C++ source). I suspect Nim --os:standalone would work for hooks, but you won't have GC, and you'll have very little library support -- which is also the case when doing this in C, in general.
The new runtime does change things: with --newruntime -d:useMalloc C's malloc implementation is used and that can work differently/better with DLLs. The new runtime doesn't use conservative stack tracing so that's also one thing less to worry about.
Having said that: 0.20 actually ships with "hot code reloading" (HCR), I'm reworking its documentation. HCR does work with the existing GCs, you don't have to wait until the new runtime is ready, enjoy!