It has been a while since I last posted some new Nim stuff so I thought this would be a good one to share. https://github.com/guzba/curly
Curly is a new HTTP client built on top of libcurl. What makes Curly interesting is that it enables running multiple HTTP requests in parallel while controlling how and when you want to block.
Some highlights are:
import curly
let curl = newCurly() # Best to start with a single long-lived instance
let response = curl.post("https://...", headers, body) # blocks until complete
var batch: RequestBatch
batch.post("https://...", headers, body)
batch.get("https://...")
for (response, error) in curl.makeRequests(batch): # blocks until all are complete
if error == "":
echo response.code
else:
# Something prevented a response from being received, maybe a connection
# interruption, DNS failure, timeout etc. Error here contains more info.
echo error
curl.startRequest("GET", "https://...") # doesn't block
# do whatever
var batch: RequestBatch
batch.get(url1)
batch.get(url2)
batch.get(url3)
batch.get(url4)
curl.startRequests(batch) # doesn't block
# do whatever
let (response, error) = curl.waitForResponse() # blocks until a request is complete
if error == "":
echo response.code
else:
echo error
# Or use `let answer = curl.pollForResponse()` and `if answer.isSome:`By choosing what blocks and doesn't block, you can manage your program's control flow however makes sense for you.
My Mummy HTTP server mostly makes blocking requests to a handful of endpoints through one Curly instance that is used from many threads. This results in great connection re-use to keep latency as low as possible.
I do however have one specific HTTP API call I need to make a lot that does not need to block the Mummy request handler. For this, I created a second Curly instance just for these requests, and use startRequests instead. I then have a thread that blocks reading responses and handles any cleanup necessary.
I have an example you can run to see the time difference between sequential and parallel HTTP requests here.
Running requests in parallel is obviously going to be much faster than sequential.
Thanks for taking a look!
This is amazing work as always!
Not sure if you've documented this elsewhere, but what is the difference between curly and https://github.com/treeform/puppy? Is curly more intended for use on servers (running Linux), or does it work on Windows as well with the libcurl dll? Is it possible to include a similar batching API in Puppy as well?
Puppy's origin story is "I want to make easy cross-platform HTTP requests without -d:ssl and without extra stuff on Windows". Not really server-focused but it certainly works there.
Curly's origin story is "my server needs a good way to do lots of low-latency HTTPS RPC calls". This was a harder requirement.
Imo these are pretty different origins which makes a difference at least to where they stand today. Also when I worked a bit on Puppy I knew a lot less about all of these ways of solving the HTTPS problem so there's that.
I think the APIs are moving closer together but there is one fundamental difference--Curly is a libcurl-centric thing which means it'll always need a DLL on Windows. Puppy uses OS APIs on Windows to avoid this.
These differences kind of don't matter but kind of do. I've not thought enough about it but a revisit of Puppy is due with the knowledge gained over the past couple years. There is no reason it shouldn't have similar batching support etc.
Long vague answer but yeah Puppy has its niche imo and can improve too.
Sorry, I don't have a github account. But I was able to add a proxy this way
curly.nim
# Follow up to 10 redirects...
discard easyHandle.easy_setopt(OPT_MAXREDIRS, 10)
# New code for proxy (proxy from https://spys.en)
let proxy = "socks5://98.181.137.83:4145" # pass proxy string from main program as batch.get argument?
discard easyHandle.easy_setopt(OPT_PROXY, proxy.cstring)
# many proxy failed without this
discard easyHandle.easy_setopt(OPT_SSL_VERIFYHOST, 0)
discard easyHandle.easy_setopt(OPT_SSL_VERIFYPEER, 0)main program
import curly, std/times
let curl = newCurly()
var batch: RequestBatch
for i in 0 ..< 3:
batch.get("https://api.ipify.org")
for (response, error) in curl.makeRequests(batch, timeout = 30):
if error == "":
echo response.code, ' ', response.url, ' ', response.body
else:
echo errorSomething like this, when for each request you can set an independent proxy would be very useful for scraping.
for i in 0 ..< 3:
let proxy = "..."
batch.get("https://api.ipify.org", proxy=proxy)