Dears,
is there any way to optimise this little piece of async code to make faster response and shorter code maybe? URLs and search string being provided only for the researching purpose.
import std/[httpclient, asyncdispatch, strutils]
const urls = @[
"https://www.google.com",
"https://www.yahoo.com",
]
proc getContent(url: string): Future[void] {.async.} =
let client = newAsyncHttpClient()
let content = await client.getContent(url)
echo url & " " & $content.contains("tag")
var futures : seq[Future[void]]
for url in urls:
futures.add(getContent(url))
for future in futures:
waitFor future
I'm mostly curious about for loops.
Thanks
That doesn’t work, each client can only support one concurrent request, which ngl is kinda dumb for an async client and also isn’t mentioned anywhere and it gives you a weird error if you try and do multiple iirc.
If you want shorter code you should check out the all proc from asyncfutures.
For speed, there’s not much huh can do. Most of the time spent seems like it would be in the network request which you can’t speed up, or in the contains check which you can’t speed up.
PS make sure you close your clients
As far as number of lines go, you can combine the creation of http client, request, and checking for content in one line. Also, although you made an async proc, you use waitFor in a for loop, for each future, which will make the program behave just like doing non async http requests, it will make each http request sequentially, only start a new one after the future/proc is complete. To avoid this you can replace the for loop with "waitFor all futures" https://nim-lang.org/docs/asyncfutures.html#all%2Cvarargs%5BFuture%5BT%5D%5D
So the code would look like:
import std/[httpclient, asyncdispatch, strutils]
const urls = @[
"https://www.google.com",
"https://www.yahoo.com"]
proc getContent(url: string): Future[void] {.async.} =
echo url & " " & $(await newAsyncHttpClient().getContent(url)).contains("tag")
var futures : seq[Future[void]]
for url in urls: futures.add(getContent(url))
waitFor all futureshttps://github.com/nim-lang/Nim/blob/devel/lib/pure/asyncdispatch.nim#L2019
poll() queues all futures in same thread, there is no difference
Well, you (maybe) can speed up network io when you request compressed html (which the std client cannot do). I've done a small wrapper that can do this, but it's also very easy to just build this into your own clients.