Mirror of forum.nim-lang.org

4164 :: Question regarding parallel_counts.nim from Nim in Action

• wizzardx (orginal) [2018-08-29T13:56:45+02:00] view original

  As per this file referred to in Chapter 6:

  https://github.com/dom96/nim-in-action-code/blob/master/Chapter6/WikipediaStats/parallel_counts.nim

  Isn't this kind of a race between how fast the disk can read the file, and how fast the CPU threads can process it?

  eg, with a huge file, slow CPUs, fast IO, low RAM, the system can run out of ram because of too many separate threads being in the middle of the parseChunk() function, where each thread is hanging onto at least 1_000_000 bytes of ram?

  Or does something special happen (eg, "spawn" function starts blocking) when the # of spawned functions reaches MaxThreadPoolSize (256).

---

• Araq (orginal) [2018-08-29T15:51:02+02:00] view original

  Well if the number of threads exceeds the MaxThreadPoolSize threshold, spawn blocks until a thread becomes ready. Which seems to avoid the problem that you're describing but these things are subtle. :-)

---

• wizzardx (orginal) [2018-08-30T09:51:31+02:00] view original

  Thanks, good to know!

  Can we document spawn as having blocking behavior in that case?

---