Hello all,
I have been playing with the chat-server example from the asyncnet module but I hit this strange data loss: if the 7.5s time-out kicks in, the subsequent clients' data is lost (seems to start receiving after a few send attempts within current 7 sec frame).
Pretty sure the OS (on Linux) buffers internally small data, so there should be no internal data discarding. If using buffered sockets then there is no data received at all?
Any ideas what may happen here? Cheers.
import asyncnet, asyncdispatch, strutils, sequtils, oids
var clients {.threadvar.}: seq[AsyncSocket]
proc processClient(client: AsyncSocket, id: Oid) {.async.} =
while true:
echo "awaiting data from client: ", $id
let dataEvt = client.recv(256)
let readEvt = withTimeout(dataEvt, 7500)
if await(readEvt) == false:
echo "nothing yet"
continue
let line = dataEvt.read
if line.len == 0:
echo "socket gone"
break
if line.startsWith("quit"):
client.close
echo "has quit: ", $id
for i,c in pairs(clients):
if c == client:
clients.delete(i)
break
break
for c in clients:
await c.send(line)
proc serve() {.async.} =
clients = @[]
var server = newAsyncSocket(buffered = false)
server.setSockOpt(OptReuseAddr, true)
server.bindAddr(Port(12345))
server.listen()
while true:
let client = await server.accept() # client will inherit .isBuffered
clients.add client
let id = genOid()
asyncCheck processClient(client, id)
asyncCheck serve()
runForever()
Yes, the problem is that withTimeout doesn't cancel the dataEvt future. So it continues running even though the timeout occurred, eventually reads the data and loses it since the withTimeout has long disappeared.
Yep, this is horrible but sadly async in Nim doesn't yet support cancelling futures :/. We might have to put a big warning on withTimeout to watch out for this until we get that support.
In order to give you a workaround I need to ask: why are you doing this? Is it because you want to add code which disconnects the client if it doesn't send data in a certain amount of time? (if so, you should just create a processClientPings async proc and call it in the background with asyncCheck as well, closing a socket that is awaiting on a read in another async proc is safe).
Thanks Dom, good to know these sort of implementation details.
withTimeout doesn't cancel the dataEvt future
Similarly, does it mean that an or composite future will suffer from the same issue?
why are you doing this?
No particular reason, just testing to see how resilient the whole thing is. The processClient type routines may be well doing something else (short maintenance task) till new data becomes available.
closing a socket that is awaiting on a read in another async proc is safe
OK, what about deleting the client object from the global array? I would have thought that in this particular case other instances of processClient may carry-on looping through clients list via
for c in clients:
await c.send(line)
after one client disconnected and thus it may actually access invalid data. Would clients.delete(i) requires some sort of locking?
Note that Chronos supports future cancellation since v2.2.7, 3 weeks ago.
I'm not sure how different the internals are from asyncdispatch at this point so it might be hard to naively port the PR.
OK, what about deleting the client object from the global array? I would have thought that in this particular case other instances of processClient may carry-on looping through clients list via
As long as your loop doesn't have await then you're fine... but you shouldn't be looping through a list that you're deleting items from anyway, that's a common limitation (nothing to do with async).
Note that Chronos supports future cancellation since v2.2.7, 3 weeks ago.
I'm not sure how different the internals are from asyncdispatch at this point so it might be hard to naively port the PR.
It should be possible to port this, I might work on an implementation this weekend if it turns out relatively trivial.
I don't recommend using Chronos, you'll be split off from the rest of the async ecosystem.
@lucian, It is much better to use Chronos which has much less bugs and undefined behaviors inside.
So when you finish reading book you can start with more reliable asynchronous framework.
And yeah we still missing some toys like HTTP, FTP, SMTP servers and clients, but we are working on making it all the time. Also you will get much more support and help.
And i don't recommend to use asyncdispatch because you will fall into lot of bugs, leaks and problems.
Please provide some links to these bugs, leaks and problems.
Asyncdispatch is the foundation of chronos. Sure, maybe you've fixed some edge cases, but you've also broken the API and indeed are not compatible with most async Nim packages. Asyncdispatch has also continued to evolve and I'm sure there are things that have been fixed there that haven't been fixed in Chronos.
Instead of spending your efforts telling users to use chronos and hurting our community, why don't you instead upstream some of these fixes you've made? Asyncdispatch isn't going anywhere, it will remain in Nim 1.0 and beyond.
It is complicated at this time to use the asyncdispatch as it has several issues that have not been solved over time.
In what way is it complicated? Can you share some examples?
I'm was not first who has started to recommend "not to use chronos".
There many languages which do not support async/await too. But C# and Python async/await implementation support cancellation, so i think it is better to stick with mainstream languages then start to learn from languages which are younger then asyncdispatch itself.
Also @yglukhov can confirm that this work was done under Status patronage.
If you stop propagation of your recommendations not to use one or many Status libraries, i will stop to recommend not to use asyncdispatch.
I've put up an RFC to discuss the Async/Chrons split and how the Nim community should deal with that:
A similar thing happened in the OCaml community with the async programming libraries and there is fragmentation in the community between Lwt and Jane Street's Core.Async library. While diversity of implementations is cited by some as a sign of health, it is my opinion that in a "small" language community it is decidedly unhealthy to have a split at this level, exactly as @zevv writes in his RFC.
I agree with all of @zevv's points except the statement that this shouldn't get discussed on the forum. For better or worse, I prefer the forum. While I can tell that some feathers were ruffled in this thread, the discourse has been civilized. Everyone here is a Nim fan so we all want the best Nim possible!
I hope both libraries are being tested with --newruntime, and I'm looking forward to @Araq's longer reply.
This forum is a proper place for async discussions. Perhaps you should have created a new thread in your initial reply promoting Chronos?
IMO the topic line is general enough that the discussion here is fine. It's a bit imperious to ask people to stop talking about a related topic on a thread because you'd prefer a different venue. Topic threads often drift, and this one did not drift far.
Few cents from my side
Seeing all that makes me kinda sad. Nim is pretty neat language and the ecosystem seems to be moving into the proper direction BUT situation like that especially just in sight of 1.0 is at least super concerning.
My experience with asyncdispatch is mostly positive BESIDES times where I've hit some corner cases / weird behaviour / undefined behaviour. Having something that is having a lot (let's be honest there...) corner cases, and kinda unexpected design decisions (.connect with UDP) is not super optimal I think, especially that the aim is 1.0. Not having cancellation is really hard block for me, I can't imagine having something running on production without being able to cancel the futures. It will basically screw up most of the error handling.
I'm not saying that asyncdispatch is wrong, it's okish, but it requires A LOT of extra testing and stabilization.
The community split between asyncdispatch and chronos (a hard fork of asyncdispatch) sounds very bad for me, especially that our community is rather small. In the same time, the existence of chronos proves that async is possible outside of stdlib which is great I think.
Seeing and having that situation is definitely not helping nim with adoption... is there any sight how that can be solved/mitigated?
Well Chronos could be adapted for Nim v2, but our priority is to get v1 out and everything that we currently have in the stdlib is about to stay for v1. In hindsight plenty of modules that are there should have been Nimble packages (parsesql in the stdlib?) but there is only so much you can do in a single day and moving code to different github repos doesn't improve the code quality. And a web of dependencies does come with its own problems.
For new application development, feel free to use either framework, but new libraries should support asyncdispatch. Its quality isn't as good as I would like but bugs can and will be fixed and the API can be improved incrementally.
Also this whole "split" discussion is based on the assumption that 3rd library code cannot support both at the same time, but that's not true, see https://github.com/tormund/news the client code is
proc sendMsg() {.async.} =
var ws = await newWebSocket("ws://localhost:9001/ws")
await ws.send("hi")
while ws.readyState == Open:
let packet = await ws.receivePacket()
echo "received ", packet
waitFor sendMsg()
and it's identical for both versions. The diffent imports are unfortunate but it's not hard to imagine a better solution here. Thanks to macros, templates, when declared, when compiles and symbol re-exports accross modules Nim is the most flexible tool that I know that allows for these bridges. Nim is not an average programming language. Two implementations can be supported at the same time even if they differ in their APIs.
Nim is the most flexible tool that I know that allows for these bridges. Nim is not your average programming language, issues that caused trouble for other languages might not be applicable. Two implementations can be supported at the same time even if they differ in their APIs. - Araq
That should go on the home page lol. A great quote that shows the beauty and power of the language.