Mirror of forum.nim-lang.org

3454 :: Sockets in Nim: I must choose between "select" and "accept"?

• monster (orginal) [2018-01-03T16:23:40+01:00] view original

  My in-process "messaging" library now has all the basic features I wanted, so it's time to add (TCP) cluster communication.

  I would have liked to use something like zmq or nanomsg, but it seems my use-case is not covered by those. I've decribed it lower down; please correct me if I'm wrong.

  So, I checked into the nativesockets module, but this one does not seem to support "accept", which I need for a "server socket" (a socket accepting client connections). Than, I also looked at the net module, which has "listen" and "accept", but no "select", which I need for scalability reason (I don't want to use the one-thread-per-client model).

  As this state of affairs doesn't make sense to me (in particular, nativesockets supporting "listen", but not "accept"), I must assume that I'm missing something... Maybe you can't use "select" together with SSL, and that is why it's not available in the net module?

  So, can someone tell me how to either use "select" in "net" or "accept" in "nativesockets"?

  ---

  Here is my use-case:

  1. Each cluster node will have a fixed-size pool of worker threads; presumably pinned to CPU cores.
  2. "Entities" are assigned to a specific thread of a specific node, using the thread-local heap.
  3. Entities "exchange messages", both across threads, and across the cluster.
  4. Each message contains the node-ID, thread-ID and entity-ID of the sender and reciever.
  5. Worker threads do not ever do any blocking actions; in particular, no IO.
  6. IO must therefore be handled by separate threads from the worker pool.
  7. But the threads doing the IO cannot "process" the messages; that's the worker thread's job.
  8. Therefore, it must be possible to read new incoming messages, without immediately answering previous messages.
  9. Each cluster node has one "server socket", accepting connections from other nodes.
  10. When a node boots up, it (somehow?) queries the running nodes, and connects to each of them.
  11. So, we have a fully-connected node graph. I'm aiming for 10 to 100 nodes.
  12. Once connected, the "direction" of the connection becomes meaningless, since communication is "symmetric".
  13. I'm assuming a relatively "stable" cluster (hardware cluster nodes, not per-hour VMs), so new "connections" should be rare.
  14. Code should run (at least) on Linux, and each cluster node is assumed to be "all of the same hardware/OS".

  Given all of that, I'm looking for a model with two networking threads:

  1. Does "selectRead"s, transfer incoming messages to the appropriate worker threads, and occasionally checks for new connections ("accept").
  2. Fetches (queued) outgoing messages, and sends them ("selectWrite").

  I'm open to any solution that allows me to do this (efficiently) in Nim. Ideal solution would also support encrypted communication, since the cluster is going to be "in the cloud".

  #8 means I cannot use "pairs" from nanomsg or zmq, for example.

  FYI, I've written the networking code, and load-balancer, for a 20 node cluster that has been productive for over 10 years, so I have experience in sockets (and all the nasty networking issues that come with them) but, that was all in Java. Since I know how much of a pain it is to cover all corner cases, I was hoping to use a "higher level" solution this time.

---

• dom96 (orginal) [2018-01-03T20:03:55+01:00] view original

  So, can someone tell me how to either use "select" in "net" or "accept" in "nativesockets"?

  TL;DR: Use the selectors module.

  The select procs scattered throughout the stdlib are largely deprecated and replaced by the selectors module. Depending on your use case though you probably want to use async await with asyncnet which builds on top of selectors on POSIX and IOCP on Windows (via asyncdispatch).

  Keep in mind that nativesockets is just a low-level sockets wrapper. The net module is built on top of it (but it's meant to support synchronous workloads).

---

• 2vg (orginal) [2018-01-04T02:05:40+01:00] view original

  > "accept" in "nativesockets"?

  you can use "accept".

  this is example code.

  check accept

  if you support only linux, can use accept4.

  (can make socket non-blocking at the same time as accept.)

  var server = newNativeSocket()
  
  # ~
  # bind, listen
  # ~
  
  var
    client: SocketHandle
    sockAddress: Sockaddr_in
    addrLen = SockLen(sizeof(sockAddress))
    
    # for accept4
    SOCK_NONBLOCK {.importc, header: "<sys/socket.h>".}: cint
    
    # for accept4
    SOCK_CLOEXEC {.importc, header: "<sys/socket.h>".}: cint
  
  proc accept4(a1: cint, a2: ptr SockAddr, a3: ptr Socklen, flags: cint): cint
    {.importc, header: "<sys/socket.h>".}
  
  client = accept(server, cast[ptr SockAddr](addr(sockAddress)), addr(addrLen))
  
  client = accept4(server, cast[ptr SockAddr](addr(sockAddress)), addr(addrLen), SOCK_NONBLOCK or SOCK_CLOEXEC)
  
  Run

---