I notice that requestAux in httpclient.nim does a await client.parseBodyFut at the start of any request. This is not precisely correct behavior, though I'm being piccayunish here. HTTP/1.1 servers are required to continue accepting requests, before the response has been delivered, queuing them up in what they call an HTTP pipeline <https://en.wikipedia.org/wiki/HTTP_pipelining> That can seriously speed things up in high bandwidth, low latency conditions, especially when loading a lot of small files.
What it should do instead is (if the connection is HTTP/1.1) save the future where it says await client.socket.send(body) or await client.socket.send(headersString) if there's no body. Substituting that future in for client.parseBodyFut should enable HTTP pipelining in httpclient.nim, without any other modifications AFAICT.
Of course, that leaves it to the user to ensure that they do send several requests in parallel, but it wouldn't be hard to write a requestURLs(urls: string[]) function that grouped the URLs by hostname, then for a given group, a single client could fire off several requests without awaiting them, then await them collectively. I think that would produce HTTP pipelining behavior, provided that clients are not waiting on parseBodyFut but instead waiting on the completion of sending the previous request.
Honestly not a huge deal. HTTP pipelining has its own problems <https://en.wikipedia.org/wiki/Head-of-line_blocking> and if you're not writing a web browser loading up 300 thumbnails in parallel, most nim scripts are going to only ever request files serially. I just wanted to mention it, since I noticed httpclient.nim doesn't seem to support pipelining. I think it's kind of cool how simply switching the future from "after the previous request's response has been received" to "after the previous request has been sent" pipelining would just sort of... work. Nim's futures really are a powerful abstraction.
I don't even know why I try to understand things sometimes. Toying with adding pipelining support, I added some echo("OUT ", headersString) and echo("IN", line) to httpclient.nim to see what was actually being sent when. Then I ran this script:
import asyncdispatch, asyncfutures, httpclient
from strformat import `&`
proc runclients() {.async.} =
var futures: seq[Future[AsyncResponse]]
let client = newAsyncHttpClient()
for i in 0..5:
let tag = &"{i}"
echo("requesting to server ", tag)
let f = (
client.request(&"http://127.0.0.1/tag/{tag}", "GET"))
futures.add(f)
try:
let resps = await futures.all()
for resp in resps:
echo("response from server ", resp.headers)
echo("getting bodies now")
for resp in resps:
discard await resp.body
echo("done")
except ProtocolError:
echo("protocol error whee")
waitFor runclients()
And it... hung until the socket timed out then died with a protocol error.
But! Looking at the IN and OUT stuff, I saw:
OUT GET /tag/0 HTTP/1.1
Host: 127.0.0.1
Connection: Keep-Alive
user-agent: Nim httpclient/0.20.99
OUT GET /tag/1 HTTP/1.1
Host: 127.0.0.1
Connection: Keep-Alive
user-agent: Nim httpclient/0.20.99
OUT GET /tag/2 HTTP/1.1
Host: 127.0.0.1
Connection: Keep-Alive
user-agent: Nim httpclient/0.20.99
IN HTTP/1.1 404 Not Found
IN Server: nginx/1.17.0
IN Date: Sun, 09 Jun 2019 05:32:56 GMT
etc
And that's uh... pipelining. So apparently when you wait for the entire response body, Nim just uses its psychic powers to determine that what you really want to do is send more requests while you're waiting. httpclient.nim is doing HTTP pipelining just fine.
As for why my script then hangs... it probably has to do with the fact that there hasn't been some joker yet, firing off a bunch of requests on one AsyncHttpClient before waiting for any of them to complete. All the data is successfully sent and received over the async socket, but httpclient.nim stalls out turning the response data into various AsyncHttpResponse objects. Probably naively overwriting a future or something, where a queue should be used instead.