Hi, How would you go about limiting the amount of concurrent async futures, for example, to download a website I don't want to open 100s of connections at the same time.
What's the best design for that? I guess, similar to a thread pool.
I can track the active connections, seeing how many have finished, then have a while active >= limit: sleep 10
I wonder if there is a more structured approach, or perhaps a template already exists for that.
Look into std/asyncdispatch and std/asynchttpserver.
Notable, the proc shouldAcceptRequest should be close to what you want
For downloading websites I would start with a list of 100 AsyncHttpClient`s. You'll also need to track when the http client is busy, but you can do that easily with a future and have the ability to `await until the http client is no longer busy (at which point you can request the next URL). This kind of thing should be available as a library, so have a look if anything exists already, if not it would be awesome if you could create it :)
Happy to help further if my explanation isn't clear.
I once made some pretty naive code to limit the amount of open TCP connections using async. The code was very similar to the code below:
import std/[asyncdispatch, random, strformat]
const limit = 10
var counter = 0
proc testProc(n: int) {.async.} =
let rwait = rand(1000) + 500
echo fmt"Running {n} async call..."
await sleepAsync(rwait)
echo fmt"{n} async call executed."
dec(counter)
proc main() =
randomize()
let performNCalls = rand(30) + 10
var x = 1
while true:
if x <= performNCalls and counter < limit:
inc(counter)
asyncCheck testProc(x)
inc(x)
elif not hasPendingOperations():
break
if hasPendingOperations():
poll(50)
main()
I don't know if this is the best approach... try it and see if it solves your case.