Hey all,
I'm building a simple CLI that scrapes some content off a private page. For this, the CLI must first login to the website. How do I keep this connection open across multiple restarts of the CLI. Is there any way of utilizing the cookies that are set by the website on successful login (they have an expiry period of 1 month)? Could you please share some example code to achieve the same?
Thank you
http is a state less protocol so`connection open` is kind of a wrong term. There is something like http connection reuse though, but afaik the authentication process is the same.
So save the cookie data somewhere, maybe in a file. Then on a new http request, read it in and set it to the http header for your next request.
A simple example code for doing by hand:
# index.php (sorry for php here but i was lazy)
# <?php
# $value = 'something from somewhere';
# setcookie("TestCookie", $value);
# ?>
import httpClient
var client = newHttpClient()
# our "login", index.php just sets a cookie
let resp = client.get("http://127.0.0.1:80/ttt/index.php")
# server sends a cookie useing "Set-Cookie" in its http headers
let cookie = resp.headers["Set-Cookie"]
echo "the cookie: ", cookie
#
# here store the cookie in a file/or load it
#
# now every request we do should contain the cookie
# we set this with "Cookie"
client.headers["Cookie"] = cookie
client.get("http://127.0.0.1:80/ttt/index.php")
https://nim-lang.org/1.0.6/cookies.html
and last but not least, example code that i use in my small asyncHttpTools library for handle cookies:
import cookies
export cookies
import strtabs
export strtabs
import cgi # for decodeData
import uri # for encodeQuery
import asynchttpserver
type
Cookie* = StringTableRef
proc parseCookies*(request: Request): Cookie =
## Parses a cookie transmitted by the browser to the server
if request.headers.hasKey("Cookie"):
return parseCookies(request.headers["Cookie"])
else:
return newStringTable()
proc addCookie*(headers: var HttpHeaders, data: StringTableRef) =
## adds cookies to the HttpHeaders.
# format(expires.utc, "ddd',' dd MMM yyyy HH:mm:ss 'GMT'")
headers["Set-Cookie"] = @[] # @[line
var lines: seq[string] = @[]
for key, val in data:
lines.add key & "=" & val.encodeUrl & ";"
headers["Set-Cookie"] = lines
proc decodeData*(data: string): StringTableRef =
## Decodes form data into a StringTable
## TODO dublicated keys are not supported
result = newStringTable()
for key, val in decodeData(data):
result[$key] = $val
proc encodeData*(tab: StringTableRef, usePlus = false; omitEq = true): string =
## Encodes a StringTable into a form data string
var dat: seq[(string, string)] = @[]
for key, val in tab:
dat.add( (key, val) )
return encodeQuery(dat)