Mirror of forum.nim-lang.org

202 :: Some Question About Threading.

• NewGuy (orginal) [2013-08-14T05:10:52+02:00] view original

  I'm just a bit lost on how I should go about doing threading in Nimrod.

  I'm writing an engine similar to Intel's SMOKE framework, and it makes heavy use of concurrency. I need:

  • Passing of closures between threads (in the forms of jobs). I remember reading at one point that closures cannot be passed between threads because of the whole "non-sharing heap".
  • Channels - which I can't find how to handle. Do I make it a global variable within my module, and then talk to it from all the threads? Or do I pass it betwixt all my threads and have them store it as a local and it's implementation does the sharing automagically?
  • Locks - again, I can't quite figure out how I should handle this. Same way as the channels?

  P.S. After I figure out how to do all this myself, I'll try my hand at writing a proper tutorial so future nimroders will have an easier time understanding.

---

• Araq (orginal) [2013-08-14T12:31:02+02:00] view original

  1. Passing of closures is unlikely to be supported in the future. It's a very common bug in other languages to implicitly share closures producing data races.
  2. Yes make the channel a global variable. Note that you can pass 'ptr' instead of 'ref' and then no insane deep copying is performed but then you need to ensure that the owning thread keeps a pointer to it (GC_ref and GC_unref help with that). Blocking channels like Go would help with this but I am not too fond of channels in general.
  3. You can share GC'ed memory without message passing in Nimrod already via global variables. You have to ensure that the data structure is created by the main thread and not modified by other threads though. Basically this means that large lookup tables computed at program startup are supported with 0 cost.
  4. You can use the system.|| iterator to get OpenMP parallel for loop support. Note that this currently only works with --stackTraces:off and you better do not allocate any GC'ed memory in the loop body.
  5. There is system.allocShared to get shared non-GC'ed memory. Manual memory management for a shared data structure is IMHO not as bad as it sounds because both manual memory management and memory sharing between threads require the programmer to have a decent understanding about ownership, so a GC doesn't buy you that much for concurrency. (Yes, I know about Rust's revolutionary static ownership tracking.)

  We're aware the current situation is not entirely satisfying, so a better concurrency design is in the works. However, I hope what we already have suffices for your needs.

---

• NewGuy (orginal) [2013-08-14T19:45:57+02:00] view original

  1) Well this is a major part of the engine - sending jobs between threads. Do you have any recommendations here? Am I allowed to send proc pointers between threads? 2+3) Sounds good, I assume locks work like this as well, then. Might I ask why you aren't fond of channels?

  5. Yeah, I'll probably end up resorting to this.
  6. I'd be a lot more impressed with rust if it tried to ever be a 1.0 release. From what I've seen, it can't make up it's mind on how to do anything. Even on something as important as whether or not it will use GC. Sorry, it's a tangent from our conversation, but you mentioned rust.

  Well I'll work with what I've got - worst comes to worst this isn't an important project, just a learning one, so I can try something else instead.

---

• Araq (orginal) [2013-08-14T20:04:46+02:00] view original

  Check out the actors.nim implementation then. That implements what you have in mind. Sending proc vars is just fine, it's the environment part of a closure that causes problems. I am not pleased with actors.nim however and now consider it mostly a stress test for the generics implementation though. ;-)

---