Mirror of forum.nim-lang.org

3058 :: How to enforce usage from module scope / top level only?

• bluenote (orginal) [2017-07-19T21:05:05+02:00] view original

  Is there a way to enforce that a macro is only used from the top level scope? I'm still experimenting with distributed computing, which requires some macros to be used only at top level scope (to make sure certain things are registered during module initialization). I couldn't find a pragma for that and the first things that came to my mind instantiationInfo and callsite are probably not applicable. A more hacky idea was to exploit that some statements like import/export/converter only work at top level scope. But I couldn't find a way to wrap them in a when not compiles(...) because they aren't expressions. Any suggestions how to solve this?

---

• Krux02 (orginal) [2017-07-19T22:18:51+02:00] view original

  Well, I would document it properly. Nim is so flexible I would not know about anything you can emit that works only top level. Maybe you can emit a function declaration that is C calling convention. Not sure if it works though.

---

• bluenote (orginal) [2017-07-22T14:06:10+02:00] view original

  Sure if it is not possible that's what I'll have to do. I was just hoping to move the time of error from runtime to compile time.

---

• Araq (orginal) [2017-07-22T15:58:21+02:00] view original

  Can you please explain a bit more? Why is the construct only valid as a toplevel statement? I'm willing to add yet another compiler builtin to support it if the use case really justifies it.

---

• bluenote (orginal) [2017-07-22T21:08:37+02:00] view original

  The problem arises in the context of distributed computing. I'm actually following a suggestion you came up with in this thread. To make distributed computing work at all in native languages, the client (aka driver) and the workers need to run the same binary. My first working draft of the low-level API looks like this (code on GitHub):

  # testapp.nim
  import math
  
  import runner
  import driver
  import remote
  
  # registerSerializer and remote are macros that expand to
  # additional runtime registration code.
  registerSerializer(float)
  registerSerializer(int)
  registerSerializer(seq[int])
  
  proc remoteHelloWorld(i: int, data: seq[int]): float {.remote.} =
    echo "I'm running on a worker"
    echo i
    echo data
    result = sum(data).float
  
  proc remoteApp(ctx: Context) =
    # Arguments need to be pushed to the the key/value store of the workers
    ctx.pushData("i", 42)
    ctx.pushData("data", @[1, 2, 3])
    
    # Remote function call stores the result under a new key
    ctx.remoteCall(remoteHelloWorld, @["i", "data"], "result")
    
    # Now we can fetch the data from the worker
    let result = ctx.pullData("result", float)
    echo result
  
  launch(remoteApp)
  Run

  launch provides a command line interface to determine the role, i.e., you would run distributed instances via testapp master, testapp worker, testapp driver. The important thing is that all involved parties have a same understanding of registered remote procs and serializers so that they can communicate internally based on proc/serializer IDs. As long as registerSerializer and {.remote.} are only used during module initialization this should work, because master/worker/driver will execute the same code. As soon as the client code would put them into a local scope, the workers would not know about the IDs, leading to a runtime error. An alternative to using module-scope-runtime-registration would be to solve the registration entirely at compile time. As far as I can see this would require to run some kind of compile-time-finalization, because e.g. registerSerializer could be used from multiple modules and I don't know how to "freeze" the set of all registered types to build a lookup table.

  The whole use case is definitely exotic and pushing any native + statically typed language to its limits, but I'm surprised how well it works so far in Nim.

---