libdill manipulates the C call stack in various ways; I suspect you can use it "as is" through FFI when using the C backend(s), but there's likely not going to be any way to use it from JS or NimVM.
However, there's a rather promising pure-nim library https://github.com/disruptek/cps which gives similar facilities (and a lot more) while still staying pure-nim and usable on any backend. If I understand correctly, it can also naturally enable the Zig "this works both sync and async" style.
What do you guys think about this specifically, and in general about integrating finished high quality focused libraries as part of the language or its std lib?
It doesn't run on Windows, and C code tends to produce things like
https://github.com/sustrik/libdill/commit/8ffd14a29d90c55d23face575139974128cfccfc
No thanks, this "finished high quality focused library" isn't for me. I'm also not a fan of using C instead of C++ where you then emulate virtual functions, RAII, refcounting and exception handling by hand, everywhere. It's simply unprofessional.
Weave is really tuned for compute-bound workload and not IO-bound workload because it optimizes for throughput (expediting all the work at once) instead of latency/fairness (make sure early tasks are executed earlier).
A blog post waiting to be published on that (https://github.com/weavers-guild/weave-io/blob/master/blog/multithreading_flavors.md)
Some glossary before I continue:
In terms of implementation, libdill is using fibers. This means that they reserve OS page and swap the stack pointer, this is problematic at multiple levels:
To solve the cactus stack problem, libdill chooses caching with 1 to 64 256kB stacks (https://github.com/sustrik/libdill/blob/de7a917/stack.c#L41-L44), this excludes it for embedded like Arduino.
Also @treeform reported that greenlet https://github.com/treeform/greenlet which are similar to Lua coroutines but comes only with the core (and no IO abstraction like kqueue, epoll, fd, tcp, udp, tls, ...) didn't work with ref objects. I suspect any fiber-based asynchrony will need significant guard for the GC.
To be clear, using fibers significantly increases maintenance, reduce portability, and significantly increase security risks compared to an approached based on compiler code transformation. In particular this means specific code for each combination of OS:
and platforms:
This is why when I implemented Weave I choose not to use fibers even though my approach has slightly more overhead than a fiber-based approach at https://github.com/chaoran/fibril. Note that the overhead is only measurable in toy tasks and for real workload Weave has less overhead than any production-ready multithreading framework including GCC, LLVM and Intel OpenMP or Intel TBB (or Rust, Haskell, Julia, C++ HPX, Crystal, ...).
Lastly, C++ designers have been reviewing fibers for years before choosing to go with stackless coroutines as fibers don't live up to their promises and cause more problem than their worth (seen in Java, Go and Rust): http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1364r0.pdf
I don't think we have the resources to support fibers in Nim. Also Nim happens to have excellent compiler developers so compiler transformation is a natural fit rather than fiber-based concurrency.
Lastly, we are exploring a new approach to concurrency using continuations see:
That approach would be composable unlike fibers, because it would split the code transformation (CPS) from the scheduling.
libdill tightly couples the code transformation to the scheduler hurting composability. It's also unsuited to things like iterators or parsers due to allocating a 4kB page per fiber or using complex split-stack scheme.