Mirror of forum.nim-lang.org

2252 :: Passing a seq to a C callback's void* argument, chaining iterators [solved]

• komerdoor (orginal) [2016-05-10T15:33:59+02:00] view original

  Hello everyone,

  I am currently deciding if I want to use Nim as the default wrapper for my game engine written in C instead of inventing my own language. I already tried many other candidate languages including Lua, Rust, Python etc. but soon bumped into show-stopping barriers with all of them (no low-level access, lack of operator overloading, too verbose when wanting side-effects etc.).

  After 2 days of using Nim I bumped into some issues when writing a code generator to create Nim bindings for my game engine (yes I know about c2nim but it misses a lot of stuff I need). As follows:

  • How can I pass a Nim sequence to a callback void* argument while keeping a reference when it grows? I know that realloc will copy the entire memory if it cannot grow inside the same space when memory is fragmented and then returns the new address. How can I handle this correctly when using Nim sequences and void* in C callbacks? (see Issue 1 and 2 below)
  • I want to lazily filter an iterator and assumed filter(...) would do this for me, but instead it only works on sequences (which is immutable but not lazy). Instead I wrote my own filter iterator to be able to filter the output of another iterator, but it does not work. (See Issue 3 and Question 1 below)

  ...
  import future
  import libclang
  ...
  
  # Question 1: Why is something like this not in standard library?
  iterator ifilter[T](xs: (iterator: T), f: (x: T) -> bool): T {.closure.} =
    for x in xs():
      if f(x) == true:
        yield x
  
  # Issue 1: Sequence is not the same as was passed and empty again
  # Issue 2: Reference to sequence lost as soon as it grows (reallocated), causing SIGSEGV
  proc visit_child(cursor: CXCursor, parent: CXCursor, client_data: CXClientData): CXChildVisitResult {.cdecl.} =
    var cursors = cast[ptr seq[CXCursor]](client_data)[] # CXClientData is just a typedef for void*
    cursors.add(cursor); # It crashes here after 4 elements were added
    echo cursors.len # Some debug side-effect to test if something gets added
    return CXChildVisitResult.Continue
  
  iterator get_child_cursors(cursor: CXCursor): CXCursor =
    var cursors: seq[CXCursor] = @[]
    discard visit_children(cursor, visit_child, CXClientData(addr cursors))
    for cursor in cursors:
      yield cursor
  
  proc main: int =
    
    ...
    
    # Issue 3: Get "Attempting to call undeclared routine 'get_child_cursors'", but compiles fine without the ifilter part:
    for child_cursor in get_child_cursors(cursor).ifilter((x: CXCursor) -> bool => x.kind == CXCursorKind.AnnotateAttr):
      ...
    
    ...
  Run

  I've had some situations where generated C code could not be compiled by clang and/or gcc. Should I report them as issues?

  Update: Changed title to better explain content

  Thank you,

  Komerdoor

---

• LeuGim (orginal) [2016-05-10T19:22:49+02:00] view original

  In your example you call get_child_cursors as procedure, result of which is to be passed to ifilter as the first argument. And there's no such procedure, so the message. In other words: this call is the same as ifilter(get_child_cursors(cursor), ...), so get_child_cursors is trying to be called as usual procedure prior to iteration context, and only ifilter is used as iterator.

---

• komerdoor (orginal) [2016-05-10T22:59:29+02:00] view original

  In your example you call get_child_cursors as procedure, result of which is to be passed to ifilter as the first argument. And there's no such procedure, so the message. In other words: this call is the same as ifilter(get_child_cursors(cursor), ...), so get_child_cursors is trying to be called as usual procedure prior to iteration context, and only ifilter is used as iterator.

  Thank you. I modified the iterators to be procedures that return an iterator and made a module similar to Python's itertools. See:

  https://gist.github.com/komerdoor/70d9c25820952624cf797890a18f9cd5

  Any idea about the other issue?

---

• komerdoor (orginal) [2016-05-11T04:07:40+02:00] view original

  While reading the sourcecode of Nim itself I found out how to pass a seq to a C callback's void* so it can be used inside the callback, as follows:

  # The C callback
  proc visit_child(node: CXCursor, parent: CXCursor, client_data: CXClientData): CXChildVisitResult {.cdecl.} =
    var nodes_ptr = cast[ptr seq[CXCursor]](client_data) # Get and store the address of the "nodes" sequence from the user data
    nodes_ptr[].add(node) # Resolve the pointer to the value using [] and add a value to the sequence (nodes_ptr[] in Nim = *nodes_ptr in C)
    return CXChildVisitResult.Continue
  
  proc main: int =
    var nodes: seq[CXCursor] = @[]
    ...
    # Calling a C function that calls a C callback for every subnode in node, passing the address of the "nodes" sequence as user data
    discard visit_children(node, visit_child, CXClientData(addr nodes))
    ...
  Run

  Please let me know if there is a better / safer way to do this.

---