Mirror of forum.nim-lang.org

5329 :: Newbie question: downloading multiple files using async httpclient

• deepakg (orginal) [2019-10-11T11:37:40+02:00] view original

  Hello everyone!

  I am trying to make a bunch of http calls using the async http client. Code:

  import httpClient, asyncdispatch
  
  proc downloadAll(urls: seq[string]) {.async.} =
    var client = newAsyncHttpClient()
    for url in urls:
      var future = client.get(url)
      yield future
      if future.failed:
        echo "Problem!"
      else:
        echo "OK"
  
  let urls = @[
    "http://127.0.0.1:5000/json",
    "http://127.0.0.1:5001/json",
    "http://127.0.0.1:5002/json",
  ]
  waitFor downloadAll(urls)
  Run

  My expectation was that the yield would cause the event loop to schedule the get request and move on to the next iteration. However this code works sequentially. I know this is happening sequentially because localhost runs an app that sleeps for 5 seconds before serving the content and it takes 15s for this code to finish - I'd expect something close to 5s

  I'd like to better understand why this happens. Does yield not do what I think it does inside a loop?

  p.s. FWIW, I have another version based on previous threads on this forum that works that uses two procs to get the job done but I was wondering if there was a simpler way to go about it.

  Thanks! Deepak

---

• filcuc (orginal) [2019-10-11T13:21:51+02:00] view original

  I dont like async/await and I'm no expert but I would have thought that correct way was to send all the requests, collect the futures in a sea and then await all the futures...

---

• deepakg (orginal) [2019-10-11T13:38:17+02:00] view original

  So I changed it a little:

  proc downloadAll(urls: seq[string]) {.async.} =
    var client = newAsyncHttpClient()
    var futures = newSeq[Future[string]]()
    for url in urls:
      var future = client.getContent(url)
      futures.add(future)
    
    var results = await all(futures)
    echo("All Done")
  
  let urls = @[
    "http://localhost:80/status/200",
    "http://localhost:80/status/500",
  ]
  Run

  But now I get a runtime crash because the second request would lead to an exception. Is there a clean way to catch an error from an individual future?

  If I change both urls to status/200, i.e.

  let urls = @[
    "http://localhost:80/status/200",
    "http://localhost:80/status/200",
  ]
  Run

  I get a different problem:

  Exception message: Connection was closed before full request has been made Exception type: [ProtocolError]

  So looks like I also need a unique async httpclient instance per request. Is that right?

---

• mashingan (orginal) [2019-10-11T14:26:43+02:00] view original

  the problem that the client cannot be shared to other download

  import httpClient, asyncdispatch
  
  proc download(url: string): Future[string] {.async.} =
      await newHttpClient().getContent(url)
  }
  
  proc downloadAll(urls: seq[string]) {.async.} =
    var asyncresult = newseq[Future[string]](urls.len)
    for i, url in urls:
      asyncresult[i] = download url
    await all(asyncresult)
  
  let urls = @[
    "http://127.0.0.1:5000/json",
    "http://127.0.0.1:5001/json",
    "http://127.0.0.1:5002/json",
  ]
  waitFor downloadAll(urls)
  Run

---

• deepakg (orginal) [2019-10-11T14:41:55+02:00] view original

  Thanks!

  How/Where would you handle errors that arise during await all(asyncresult). For example an http 500 from the server which becomes an exception. Would proc download be responsible for handling it?

---

• mashingan (orginal) [2019-10-11T14:44:11+02:00] view original

  Yes, it's whether you to wait with asyncCheck or go with future.failed is up to application. But the idea is same.

---