by the time you ask for a body stream, the entire file is sitting in a string inside StringStream
it might be impossible at the moment
You'd have to quite drastically fork and modify httpclient.nim
Since there are various alternative ways how content length is encoded, depending on transfer-encoding being chucked, missing content-length headers, etc it will be moderately involved to change and adapt parseBody, parseChunks and recvFull to stop receiving after N bytes.
I don't think it'd be that drastic of a fork: https://gist.github.com/jrfondren/ff770a2ab3518f560ca534ec1f084ebd
httpclient.nim is <800 lines of code, without whitespace and comments. The assumption that the entire response is always wanted is baked into recvFull and its callers.
Not having fully understood the code, quick question about the code. I'm interpreting it like this so far:
"contentProgress" is a field that represents a number of so-far received characters that gets incremented as the reply is being received. The events that update contentProgress trigger a callback stored in "onRecv", which returns a boolean that states the answer to the question "Have I finished receiving the response?". If n > contentProgress, that means you're still waiting for more data and thus the answer is false. Other way around means you have received as much as you need, any further data you don't care about, the answer is true.
The await in there is somehow receiving the output of "onRecv" and once "onRecv"evaluates to true, the await stops waiting and the code continues, granting you now have access to the body of the request.
You only want the first n characters of that response-body (since you might not have more), so you explicitly substring whatever you received to be only the n characters you want.
Am I interpreting the code correctly?
I also got this solution from someone in the community: https://gist.github.com/MischaU8/1eead273cbd0abb57ca8bfba3e42a263/revisions
Not sure which one is "better"
"contentProgress" is ...
this is all correct.
The await in there is somehow receiving the output of "onRecv" and once "onRecv"evaluates to true, the await stops waiting and the code continues, granting you now have access to the body of the request.
This is incorrect. The diff doesn't enough context, so you'd need to read this with httpclient.nim around it. httpclient.recvFull is an unexported function responsible for repeatedly recv'ing from a socket until it gets a certain number of bytes. With this patch recvFull also checks onRecv and returns early if that function returns true.
You only want the first n characters of that response-body (since you might not have more), so you explicitly substring whatever you received to be only the n characters you want.
I think you probably understand the, but you phrased it opposite way. With my patch, httpclient itself doesn't limit the size of the content it returns. It just stops collecting content after the onRecv returns true. So if you want 3 bytes you can get 1000 bytes if that's what the very first recv call returned. The substring strips off the excess.
Not sure which one is better.
My solution's overly general (do you want to do anything other than check progress is onRecv?) and relatedly is not as nice for the caller (the desire to limit the content received has to be expressed twice, in onRecv and in the substr).
The other solution, I don't think works in all cases. Like this one: https://gist.github.com/MischaU8/1eead273cbd0abb57ca8bfba3e42a263#file-althttpclient-nim-L818
There are so many callers to recvFull, I think putting the limiting logic in that function will be cleaner. Anyway I think it's a better solution, to implement only the requested feature and to limit the amount requested and only return that amount.
Hey this is something I was creating the other week. It's a http client more like the fetch library. It is both async and sync by using the multi sync macro. Also have patched sockets to accept unix socket addresses.
the body() returns either as a proc or iterator. The iterator is essentially is the chunks sent from the server. You can also use streams to if you don't want to work with the iterator.
It needs some work and I had a dog ate my homework situation and lost 2 days of dev because "git push ." was returning "repo already sync'd and up to date" but it wasn't and my drive was wiped
I'm trying to keep the api simple. Theres a todo on the repo if you want to contribute :)
let client: HttpClient
let resp = client.fetch(HttpGet, "https://httpbin.org/get")
case resp.httpCode:
of Http200:
# data iterator
for data in resp.body():
echo datawith chronos:
import chronos/apps/http/httpclient
proc retrievePage*(uri: string, bytes: int): Future[seq[byte]] {.async.} =
# Create a new HTTP session
let session = HttpSessionRef.new()
var
req: HttpClientRequestRef
resp: HttpClientResponseRef
reader: AsyncStreamReader
try:
req = HttpClientRequestRef.get(session, uri).valueOr:
return @[]
resp = await req.send()
reader = resp.getBodyReader()
var res: seq[byte]
await reader.readMessage(proc(data: openArray[byte]): (int, bool) =
let consume = min(data.len, bytes - res.len)
res.add(data.toOpenArray(0, consume - 1))
(consume, res.len == bytes)
)
res
finally: # Close the session
if reader != nil: await noCancel reader.closeWait()
if resp != nil: await noCancel resp.closeWait()
if req != nil: await noCancel req.closeWait()
await noCancel(session.closeWait())
echo bytesToString(waitFor retrievePage(
"https://raw.githubusercontent.com/status-im/nim-chronos/master/README.md", 100))Thank you @arnetheduck, but now I am too far in to switch to chronos based stuff. It's nice to know they did a better design to the httpclient than nim's std. I ended up using the solution posted 2 years ago: https://gist.github.com/MischaU8/1eead273cbd0abb57ca8bfba3e42a263/7a6231050428a7e889bb0008256ab37cb9e2f5f0
but for some reason when I try to catch an HttpRequestError raised by it, it doesn't catch it, even though exception.name is "HttpRequestError" (works if I use IOError or CatchableError). Any ideas ? Maybe someone knowledgeable will want to write a new one based on newest httpclient.nim ? Any chances this could be merged ? Will nim forever have this subpar httpclient in the stdlib?
@void09 did you rewrite? it in the end?
the recv_line can be solved I think quick