Mirror of forum.nim-lang.org

7042 :: Async unexpected behavior

• pposca (orginal) [2020-11-06T14:06:23+01:00] view original

  I am trying to fetch some urls asynchronously, and I am getting a weird behavior I do not expected.

  proc fetchAllAsync*(urls: seq[string]): Future[seq[string]] {.async.} =
    var c = newAsyncHttpClient()
    for url in urls:
      debugEcho &"Before {url}"
      result.add(await c.getContent(url))
      debugEcho &"After {url}"
  
  var feeds = @[
    "https://pliutau.com/index.xml",
    "https://www.avaxhome.biz/feed/",
    "http://feeds2.feedburner.com/Ebookss",
    "http://feeds.feedburner.com/IT-eBooks",
    "http://feeds2.feedburner.com/librosvirtual",
    "http://feeds.feedburner.com/wowebook",
    "https://dev.to/feed/tag/nim"
  ]
  
  let xmlFeeds  = waitFor fetchAllAsync(feeds)
  debugEcho $xmlFeeds.len
  Run

  The result I am getting is:

  Before https://pliutau.com/index.xml
  After https://pliutau.com/index.xml
  Before https://www.avaxhome.biz/feed/
  After https://www.avaxhome.biz/feed/
  Before http://feeds2.feedburner.com/Ebookss
  After http://feeds2.feedburner.com/Ebookss
  Before http://feeds.feedburner.com/IT-eBooks
  After http://feeds.feedburner.com/IT-eBooks
  Before http://feeds2.feedburner.com/librosvirtual
  After http://feeds2.feedburner.com/librosvirtual
  Before http://feeds.feedburner.com/wowebook
  After http://feeds.feedburner.com/wowebook
  Before https://dev.to/feed/tag/nim
  After https://dev.to/feed/tag/nim
  7

  And I expected something more like:

  Before https://pliutau.com/index.xml
  Before https://www.avaxhome.biz/feed/
  Before http://feeds2.feedburner.com/Ebookss
  After https://pliutau.com/index.xml
  Before http://feeds.feedburner.com/IT-eBooks
  After https://www.avaxhome.biz/feed/
  After http://feeds2.feedburner.com/Ebookss
  Before http://feeds2.feedburner.com/librosvirtual
  Before http://feeds.feedburner.com/wowebook
  Before https://dev.to/feed/tag/nim
  After http://feeds.feedburner.com/IT-eBooks
  After https://dev.to/feed/tag/nim
  After http://feeds.feedburner.com/wowebook
  After http://feeds2.feedburner.com/librosvirtual
  7

  What am I missing?

---

• geotre (orginal) [2020-11-06T18:33:45+01:00] view original

  You are blocking on each request with await

  You probably want to do it like this:

  proc fetchAllAsync*(urls: seq[string]): Future[seq[string]] {.async.} =
    var
      clients: seq[AsyncHttpClient]
      futures: seq[Future[string]]
    
    for i, url in urls:
      clients.add newAsyncHttpClient()
      debugEcho &"Before {url}"
      futures.add(clients[i].getContent(url))
      # debugEcho &"After {url}"
    
    result = await all futures
    debugEcho "all urls done"
  
  var feeds = @[
    "https://pliutau.com/index.xml",
    "https://www.avaxhome.biz/feed/",
    "http://feeds2.feedburner.com/Ebookss",
    "http://feeds.feedburner.com/IT-eBooks",
    "http://feeds2.feedburner.com/librosvirtual",
    "http://feeds.feedburner.com/wowebook",
    "https://dev.to/feed/tag/nim"
  ]
  
  let xmlFeeds = waitFor fetchAllAsync(feeds)
  debugEcho $xmlFeeds.len
  Run

---

• pposca (orginal) [2020-11-06T19:38:13+01:00] view original

  Thank you. That is exactly what I was looking for. One question, do I need one different newAsyncHttpClient for earch request?

---

• moerm (orginal) [2020-11-06T20:14:01+01:00] view original

  Your core problem is that you only have 1 client, no matter whether async or not. One solution would be to instead of having one var outside the loop have a let this = newAsyncHttpClient() inside the loop and to then add/append the Future result to the result list.

---

• bpr (orginal) [2020-11-06T21:32:00+01:00] view original

  Interesting. I haven't really used async in Nim very much, but when I compile your code (fixing the debugEco lines) I get exactly what @pposca got in his original post, namely that all the urls are emitted in their original order with no interleaving.

---

• dom96 (orginal) [2020-11-06T23:19:24+01:00] view original

  Keep in mind that these AsyncHttpClient's need to be closed(!).

---

• geotre (orginal) [2020-11-06T23:57:25+01:00] view original

  The requests are started one after the other, but then run in parallel. If you add back in the commented out debug echo you will see it trigger in order because it now prints before the request has returned. That is why I removed it

---