Mirror of forum.nim-lang.org

10238 :: Need suggestions for time scheduling async tasks

• void09 (orginal) [2023-06-01T18:37:54+02:00] view original

  I am writing a client/crawler/scraper library for a certain website, hopefully my first nimble package. The site has some rate limiting which I haven't figure out exactly yet, but will try to do. Once I do, I want to have a request scheduler so that all requests (async) done to the website are automatically delayed by the right amount of time once I reach the rate limit threshold value (eg, 1000 requests per hour, 30 requests per minute etc), so that it doesn't trip the rate limiting. I am thinking a template to wrap the procedures call bodies in and send it to the schedueler would be a proper interface. Or maybe I should just compute the required sleep time value (if needed) at each call, and make a proc that wraps the call to the proc, adding a sleepAsync call before it ? I feel this is a rather common use case, and we should have a library for this. I could only find this after 5 minutes of searching: https://nimble.directory/pkg/taskqueue - but I can't say I understand how it works/how to use it after reading the readme.

---

• PMunch (orginal) [2023-06-01T19:42:13+02:00] view original

  Maybe the at package could be helpful?

---

• cmc (orginal) [2023-06-01T20:59:54+02:00] view original

  @PMunch thanks for recommending at!! @viud09 it's a lightweight persistent task scheduler.

  I have an unpublished comment system that uses at combined with limdb for that exact purpose- in fact, having a rate limiter is the original use case why limdb AND at were written.

  I would recommend to explicitly call the delay proc in each endpoint. No magic to forget about, more clarity, and if you need an endpoint to work differently, it's straight forward.

---

• PMunch (orginal) [2023-06-02T10:58:46+02:00] view original

  Had a look at your repos after you mentioned that the german tax thing was your first project to receive a contribution (although that's not a repo tsk tsk). I seemed to remember you had some interesting projects so I thought it sounded strange. And that's when I found at after having first seen it during "This Month in Nim" (you should post your tax story to that as well to share with non-forum readers).

---

• void09 (orginal) [2023-06-02T12:56:00+02:00] view original

  Thank you for the suggestion, unfortunately the way it works is not clear to me, and how I should implement it for my use case. I realized the actual requests are done by a single procedure in my code, and I can calculate a sleep delay at the start of it, given the times previous requests were made and the ratelimit constraints. It will do. Although a more generic queue-like (outside of asyncdispatch) method would have been preferable.

---

• arnetheduck (orginal) [2023-06-02T14:21:01+02:00] view original

  The classic approach to rate limiting is via the use of a token bucket - here's an implementation for chronos (asyncdispatch alternative): https://github.com/status-im/nim-chronos/blob/master/chronos/ratelimit.nim#L126

---