Is there a good way to pass ownership of an object to another thread without channels? Most of the existing thread tutorials seem to be pre-ARC, so it's unclear how much is still valid.
Using ARC's shared heap, I've got a setup where I can pass the pointer to an object via a thread safe queue. It's error prone and requires making a new variable with a pointer to the object then passing the pointer of that variable (e.g. ptr ref object). Then calling GC_ref on the object, since otherwise the sending threads current scope will end and deallocated the object.
The queue is a FreeRTOS xQueue that passes a copy of data given from a given C-style pointer. I don't think channels will work as they're not ported to FreeRTOS (yet). Also I'd like to keep using the system's queue. Originally I was copying the object itself since it's a ref object. It'll work great, right!? That worked until it didn't and corrupted the heap (probably because ARC allocates extra room for the object's reference count before the ref object so simply copying the bytes of addr(refObject) does bad things to the heap eventually).
Here's a snippet of the sender:
...
var res: JsonNode = qh.router.route( rcall )
var pres: ptr JsonNode = addr(res)
GC_ref(res)
discard xQueueSend(qh.outQueue, addr(pres), TickType_t(1_000))
And the receiver:
var res: ptr JsonNode
while xQueueReceive(qh.outQueue, addr(res), 0) == 0:
logd(TAG, "rpc socket waiting for result")
continue
var rmsg: string = msgpack2json.fromJsonNode(res[])
I was hoping there's a nicer/cleaner way to do this. Maybe a way to repurpose move for this purpose. If not, I'll probably need to write a proc/template around FreeRTOS's xQueue mechanism to do the pattern above (it's not too bad really). I also don't know how to properly use the sink annotations to let the compiler know the ownership has passed which would be nice.
I think this should help: https://github.com/nim-lang/RFCs/issues/244
A version of this feature has already shipped in 1.4 and can be used via import std/isolation.
I think this should help: https://github.com/nim-lang/RFCs/issues/244
Alas, isolate looks like it's still pretty early stages. It turns out Channel`s *do* actually work on FreeRTOS (via pthread emulation)! Unfortunately they fail even when passing only strings. So `isolate wouldn't help anyways, though it's a great way forward.
There are some ideas here on inter-process communication (although I didn't use the module it's helpful as there's little info on using pipes in Nim, they are different in Windows and Unix) https://github.com/cheatfate/asynctools You can pass data byte by byte with a pipe: import posix, sequtils, strutils, sugar, os
var fd: array[2, cint] if pipe(fd) == -1: raiseOSError(osLastError()) var mystr: cstring = "hello pipe" discard write(fd[1], mystr, mystr.len) discard close(fd[1]) var buffer: array[80, char] var data = newSeqchar
data &= buffer[0 .. count - 1]
discard close(fd[0]) var output = newStringOfCap(data.len)
echo output
That's probably simplistic and less efficient than what you are trying to do though.
There are some ideas here on inter-process communication (although I didn't use the module it's helpful as there's little info on using pipes in Nim, they are different in Windows and Unix) https://github.com/cheatfate/asynctools You can pass data byte by byte with a pipe:
That looks handy for a lot of cases, but in my case pipes don't work. Passing via ptr works well if somewhat klunky. Using regular channels would be the way forward too. Both of those methods would benefit from Isolate[T] as well. (P.S. the memory corruption issues I have passing pointers are due to esp-idf/FreeRTOS libraries).
Here's a bit of magic sauce I found in stdlib channels' implementation:
It appears to me that wasMoved is a handy little proc to disable the destructor, and should make sure that your ref is not freed. Once the pointer is on the other side of the channel you should be able to just move it into place.
Here's my adaptation of that send/recv code of yours (100% untested):
var res: JsonNode = qh.router.route( rcall )
# Persumably this copies the content of the `res` variable, which contains the pointer to the managed JsonNode instance.
discard xQueueSend(qh.outQueue, addr(res), TickType_t(1_000))
wasMoved(res) # Zero out the variable to prevent freeing when scope ends
var res: JsonNode
# I'm assuming that this would just copy the pointer straight in to `res`. If that's the case then we have successfully "moved" the ref between two threads.
while xQueueReceive(qh.outQueue, addr(res), 0) == 0:
logd(TAG, "rpc socket waiting for result")
continue
var rmsg: string = msgpack2json.fromJsonNode(res[])
Isolate[T] would be useful to ensure safety but until we get a recover() proc to get T back I guess it won't work for now.
due to esp-idf/FreeRTOS libraries
Which ones? How did you find out?
It appears to me that wasMoved is a handy little proc to disable the destructor, and should make sure that your ref is not freed. Once the pointer is on the other side of the channel you should be able to just move it into place.
That's some magic I was hoping to find! I'm not convinced my trick of GC_ref works properly.. I still get the aforementioned issue which appears to be a double free sometimes. The isolate proc can still help confirm the data to pass is a single tree. It fails on the msgpack4nim.toJsonNode which is odd as the JSON parseJson works fine with isolate (maybe my issue with the double free's below).
Which ones? How did you find out?
@shirleyquirk I'm not 100%, but I switched to a newer release of ESP-IDF and the problem went from barely running to running a few thousand calls before crashing. Also, I ran into a similar malloc race bug when dealing with the esp-idf's UART <-> socket compatibility layer that was a known issue on their GitHub. Given my port of Nim's net libraries to FreeRTOS relies on the compat layer for sockets and select, I now suspect that bit of code has issues with race conditions.
Ideally, I'd like to have the time to write a pure Nim interface to the raw FreeRTOS select style mechanisms! It'd be interesting to write. Then Nim would sort of become it's own OS layer on FreeRTOS and be less bound to esp's. Well once ARC and multi-threading stabalizes a bit more..
Unfortunately, there also looks to be a double free in the Nim ARC when I turn on arcTracing after moving my serialization to the second core. So... it might be the worse case, and be two overlapping bugs! :-S
Today, I'm working on writing some test cases emulating my RPC method with normal OS stuff so I can run it on my dev machine.
If anyone has ideas on how to best write some test cases, I'm all ears! Especially for checking the heap integrity. Valgrind maybe?
Alright, after spending many hours trying to figure out why the heap was being corrupted when passing data, I finally tracked it down. It wasn't data passing or malloc, but code I copied from an RPC library which was using a field on the JsonNode's that the compiler "couldn't prove was accessible". It was library code and the same warning shows up in the standard library for JsonNodes so I ignored it. Still, despite the warning the compiler generates code which silently corrupts the heap... Not sure if that should be a compiler bug or not. Maybe it's new behavior with ARC?
The tricky part was that the problem wouldn't show up usually until the JsonNode's were freed. Between that and leaks introduced while trying data passing methods, it took me a while to find the real source of the bug.
However, I did test channels, pointer passing, ref node copying, and custom C-string. They all worked, but some required tweaking with moved to work properly. That's great news!
Still, despite the warning the compiler generates code which silently corrupts the heap..
Sounds like something worthy of a report to me.