Mirror of forum.nim-lang.org

6103 :: Http(s) Stream

• TheDragonsApprentice (orginal) [2020-03-22T13:10:06+01:00] view original

  Hi,

  I'm trying to stream data from a (https) website, using the httpclient module. The data is in ndjson form - basically, JSON objects separated by \n, and new objects are added periodically. Using the normal getContent method doesn't work, because the data is a continuous stream so it never finishes reading. From noodling around in the source code I see that there Response has a bodyStream attribute, so I tried calling some stream methods on that to get the data one line at a time - readLine, peekLine, readData, readDataStr - but none of them worked.

  If it is of any use, the API- I'm calling is the Lichess API, and the url is https://lichess.org/api/stream/event . An API key is needed to use this however.

  Is there a way to do this using Nim? Thank you :)

  This is the code:

  1 import httpClient, os, streams
    2 let token = "Bearer " & paramStr(1)
    3
    4 var client = newHttpClient()
    5 client.headers = newHttpHeaders({"Authorization":token})
    6 var n = ""
    7 let res = client.get("https://lichess.org/api/stream/event").bodyStream
    8 while res.readLine(n):
    9   echo "next"
   10   echo n
   11
   12 res.close()
  Run

  
  > nim c -r -d:ssl main.nim {secret token}
  Run

---

• TheDragonsApprentice (orginal) [2020-06-19T11:39:55+02:00] view original

  So the answer was I had to use the recvLine proc of the net package.

  Something like this:

  import net
  let s = newSocket()
  wrapSocket(newContext(), s)
  s.connect("lichess.org", Port(443))
  
  var req = "GET /api/stream/event HTTP/1.1\r\nAuthorization: {token}\r\n"
  
  s.send(req)
  while true:
    echo s.recvLine
  Run

---

• tsojtsoj (orginal) [2023-02-19T04:03:59+01:00] view original

  Just in case someone stumbles over this thread: I had to make some slight changes for this to work again:

  import std/[
      net,
      strformat,
      strutils
  ]
  
  const
      host = "lichess.org"
      token = "wn89m89uer8999e9"
  
  let s = newSocket()
  wrapSocket(newContext(), s)
  s.connect(host, Port(443))
  
  let req = &"GET /api/stream/event HTTP/1.1\r\nHost: {host}\r\nAuthorization: Bearer {token}\r\nAccept: x-ndjson\r\n\r\n"
  echo req
  
  s.send(req)
  while true:
      let line = s.recvLine
      if line.strip.len > 0 and line.strip[0] == '{':
          echo line
  Run

---