@dom96, you are the author of async libraries, don't you? Maybe you can help me with it. Here is my experiments with it: https://github.com/vegansk/Nim/tree/async_buffers. I don't want to create the PR right now. Here is the problem. The tests works on windows, when I implement send via sendBuffer: https://github.com/vegansk/Nim/blob/async_buffers/lib/pure/asyncdispatch.nim#L865. But it fails on Linux in tests/async/tasyncawait.nim if I implement send like this:
proc send*(socket: AsyncFD, data: string,
flags = {SocketFlag.SafeDisconn}): Future[void] =
## Sends ``data`` to ``socket``. The returned future will complete once all
## data has been sent.
var c = data
socket.sendBuffer(addr c[0], data.len, flags)
and when I copy sendBuffer's body, to send (https://github.com/vegansk/Nim/blob/async_buffers/lib/pure/asyncdispatch.nim#L1464) it's just works.
Last I heard there was a plan to make asyncdispatch provide low-level unsafe methods that accept raw pointers, while higher-level modules (like asyncnet) would provide methods that work with GC-safe stuff.
In your case it seems the problem is that you pass pointer to GC data to an async proc without letting the GC know that you did that. So as soon as the current stack is left, the data is garbage collected. You should either call GC_ref at the beginning and GC_unref at the end of send or use alloc/copyMem/dealloc.
Although in the second sample data should be captured in the callback's environment so that should not be a problem anymore.
What is the reason to use the string as the buffer to send data and Future[string] to receive data asynchronously?
Because string is the most common and most intuitive type of data to send over a socket. What do you propose should be the default?
Raw pointers are unsafe and you yourself seem to have run into trouble with them. You need to ensure all your allocated data is not deallocated by the GC using the GC_ref procedure. If you don't then your application will crash.
@dom96
I'd like to suggest to provide both low level pointer and high level string.
recv(socket, pointer, size) send(socket, pointer, size) specially --- just like c socket: common, flexible, efficient.
I think mixing pointer (efficient) and GC (convenient) will be competitive for Nim.
Thanks to all, I got it.
I agreed with @_tulayang, we need both low and high level APIs.
For a Winsock application, once the WSASend function is called, the system owns these buffers and the application may not access them. This array must remain valid for the duration of the send operation.
UPDATE: First question resolved. The magic is in the poll implementation.
vega,
Could you show a safe non-GC version of readInt proc (btw, it shouldn't exist in a networking lib because of undefined size and endianness of int)? As I see it, by introducing raw pointer async send/recv procs, stdlib would just move burden to the user. To safely implement readInt you would have to alloc memory on heap (because stack will be destroyed in a moment - it is async), then, when recv is finished, copyMem from there into result, then dealloc. It feels that the amount of overhead would be somewhat similar to seq[byte]/string version.
Serialization procedures normally work with strings. Beyond sending/receiving strings one usually needs sending/receiving basic types in binary form (integers, floats, bools, etc.). So I think it makes more sense to implement the latter in stdlib and either hide unsafe procs from the user or rename them to unsafeSend, unsafeRecv so that it is crystal clear you shouldn't use them unless you understand what you are doing.
If Salvatore Sanfilippo or somebody want to write another redis by Nim, then what facilities does the libs offer?
Does Nim just focuse on the application layer softwares? Just another Java without VM?
@endragor, here it is. And it's not for network lib, it's from my new library asyncstreams. https://github.com/vegansk/asyncstreams/blob/2dd997ea1ce5bc0b0d74783d1053da95d7f07cd0/src/asyncstreams.nim#L138
@Krux02, I'm not saying about strings. I'm sayin that I need the operations with buf: pointer, size: int. See previous example.
@ka, thanks :-)
vega,
Alright, it seems I overstated about the burden. As long as async macro it used, users are fine, because addr result points to heap (to iterator's closure environment). But great caution should be taken if async is not being used.
Even if it's not networking lib, it should clearly define endianness and size of the data that's being written/read. Otherwise, it is limited to being usable only within the same machine. Take a look at other popular mainstream standard libraries - like Java, Node.js, C#. Java guarantees big endian order, C# guarantees little endian, Node.js provides both LE and BE functions, and all of them clearly define size of data that's being read/written. I think either C# or Node.js approach should be taken, because most of the chips today are little endian (perhaps there are BE microcontrollers? and previous generation consoles are BE). But there is still great diversion in 32-bit vs 64-bit, so readInt and writeInt would be quite dangerous to use.