Mirror of forum.nim-lang.org

10183 :: Save return value using threads

• StLa (orginal) [2023-05-08T16:13:19+02:00] view original

  I have pasted some of my code below. I am using threads.

  How can I save the values

  getSellerCountry(html)
  Run

  and

  sellerId
  Run

  in a global, thread-safe variable (table), so that I can later when the threads have finished, write the results into a database? Must I use channels?

  proc getSellerCountry(html: string): string =
      let xml = parseHTML(html)
      let element = xml.querySelectorAll("div.a-row.a-spacing-none.indent-left > span")
      return element[element.high].innerText.strip
  
  proc getContent(sellerId: string, url: string, ip: string, user: string, i: int, userAgent: string) {.thread.} =
      let headers = setHeader("https://www.google.fr", userAgent)
      var myProxy = newProxy(ip, user)
      var client = newHttpClient(proxy=myProxy, headers=headers)
      let r = client.request(url).body
      let html = uncompress(r)
      
      try:
          echo getSellerCountry(html), sellerId
      except Exception as e:
          echo e.msg
          echo "Error ", sellerId
  
  for sellerId in sellerIds:
      let ua = fk.UserAgent()
      url = "https://www.amazon.fr/sp?ie=UTF8&seller=" & sellerId & "&ref_=dp_merchant_link"
      spawn getContent(sellerId, url, "http://" & ip[i], userPass[i], i, ua.random.to(string))
      i += 1
  sync()
  Run

---

• Zoom (orginal) [2023-05-08T16:48:16+02:00] view original

  https://nim-lang.org/docs/threads.html has an example of using a lock for accessing a global shared resource. Remarkably, the locks module comes with no documentation and no examples whatsoever.

---

• StLa (orginal) [2023-05-08T19:32:36+02:00] view original

  @Zoom Thanks. I now am using "parallel". Now it works. The return values of a function in a thread can now be saved. Just an example how to use it:

  {.experimental: "parallel".}
  
  import httpclient, threadpool
  
  proc fetch(url: string): string =
      return getContent(url)
  
  let urls = ["https://example.com", "https://example.org", "https://example.net"]
  var res = newSeq[string]()
  
  var tasks = newSeq[FlowVar[string]](urls.len)
  
  parallel:
      for i in 0 ..< urls.len:
          tasks[i] = spawn fetch(urls[i])
  
  for task in tasks:
      res.add(^task)
  
  echo res
  Run

---