nimforum mirror - How compiler decides where to put the value?

xomachine (orginal) [2018-06-14T22:48:30+02:00] view original

Let's consider the example

import macros
import tables
proc temp() =
    echo "test"
proc aa(): Table[string, seq[pointer]] {.compileTime.} =
    result = initTable[string,seq[pointer]](2)
    let test = cast[pointer](temp)
    result["hi"] = @[cast[pointer](0),
                     #test, # <= uncomment this to move all the addresses to the heap
                     ]

const a = aa()
echo a.repr

it produces following output


[data = 0x426e20[[Field0 = 32261366769984,
Field1 = 0x426df0"hi",
Field2 = 0x426e00[nil]], <...>

when the highlighted line is uncommented the addresses of seq's and strings are being changed in the following way


[data = 0x7f6981d9a048[[Field0 = 32261366769984,
Field1 = 0x7f6981d9b058"hi",
Field2 = 0x7f6981d9c048[nil, 0x40130f]], <...>

similar result can be achieved by compiling the code from the first case via the cpp command

according to the memory map, in the first case the addresses were in some static segment of executable file, while in the second one - all strings and seq's were moved to the another memory segment (may be heap?)

The memory map looks like this:

 00400000-00431000 r-xp 00000000 fd:01 30876 00630000-00631000 r--p 00030000 fd:01 30876 00631000-00632000 rw-p 00031000 fd:01 30876 00632000-00648000 rw-p 00000000 00:00 0 010d6000-010f7000 rw-p 00000000 00:00 0 7f6981314000-7f69814d4000 r-xp 00000000 fd:01 771878 7f69814d4000-7f69816d4000 ---p 001c0000 fd:01 771878 7f69816d4000-7f69816d8000 r--p 001c0000 fd:01 771878 7f69816d8000-7f69816da000 rw-p 001c4000 fd:01 771878 7f69816da000-7f69816de000 rw-p 00000000 00:00 0 7f69816de000-7f69816e1000 r-xp 00000000 fd:01 771891 7f69816e1000-7f69818e0000 ---p 00003000 fd:01 771891 7f69818e0000-7f69818e1000 r--p 00002000 fd:01 771891 7f69818e1000-7f69818e2000 rw-p 00003000 fd:01 771891 7f69818e2000-7f69819ea000 r-xp 00000000 fd:01 771910 7f69819ea000-7f6981be9000 ---p 00108000 fd:01 771910 7f6981be9000-7f6981bea000 r--p 00107000 fd:01 771910 7f6981bea000-7f6981beb000 rw-p 00108000 fd:01 771910 7f6981beb000-7f6981c11000 r-xp 00000000 fd:01 771858 7f6981d88000-7f6981e0c000 rw-p 00000000 00:00 0 7f6981e0f000-7f6981e10000 rw-p 00000000 00:00 0 7f6981e10000-7f6981e11000 r--p 00025000 fd:01 771858 7f6981e11000-7f6981e12000 rw-p 00026000 fd:01 771858 7f6981e12000-7f6981e13000 rw-p 00000000 00:00 0 7fffd0a42000-7fffd0a63000 rw-p 00000000 00:00 0 7fffd0b06000-7fffd0b09000 r--p 00000000 00:00 0 7fffd0b09000-7fffd0b0b000 r-xp 00000000 00:00 0 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0

I also found that the Table[string,seq[string]] does not cause the value movement. So the question is: why compiler moves this values to other segment of memory?

And one more question: What happens with global variables (like strings or seq's) in the library after leaving the DllMain's scope with --gc:regions? Are they staying persistent or being destroyed with the region of the DllMain's scope? (regions are being allocated in the stack, right?)

It is a bit messy, but I have no idea that happens with memory in Nim and why I encounter random crashes in my library (and often outside it) with various GC's.

twetzel59 (orginal) [2018-06-15T19:32:47+02:00] view original

Well, I'm not the best one to answer this, but these things stick out immediately. Just some thinking points:

Your proc is compileTime. It wouldn't surprise me if string gets special treatment in the VM. I don't know how this works, though. However, in the "old runtime", string is a traced reference type (despite it's value semantics).

Two things actually changed, the type (from pointer to string), as mentioned, but also the number of elements in the seq. Could it be so simple as that a seq with only one element is not heap allocated in the VM?

I'm wondering if pointer, equivalent to void * in C, precludes storing the seq in the stack... I can't answer why, I haven't worked on the compiler. I'll have a look through the VM source and see if it's obvious.

It seems to me that the reason you are asking is because we are unsure of the definitive memory model of more complicated Nim applications.

As a question for everyone: is there definitive documentation for Nim's memory layout?

@xomachine, What is your reason to compare the VM repr to compiled code in the general case? You said that your programs crash with other garbage collectors. Is that the issue that we could help with?

twetzel59 (orginal) [2018-06-15T19:34:05+02:00] view original

Just realized a possible error in my second thought there. Table allocates a sequence of buckets, so I don't think number is the issue.

Araq (orginal) [2018-06-15T19:51:33+02:00] view original

ref, closure procs, string and seq are on the GC'ed heap, ptr is an untraced pointer, other things are put on the stack. If your code crashes, use fewer cast's.

xomachine (orginal) [2018-06-15T20:42:06+02:00] view original

It is pretty hard to track down the error when the Nim code have been put inside a WineLib and is being called from third party Windows application running under WINE.

@twetzel59 I have many lines of code is being generated during compile time. In particular the table of seq s of function pointers are being generated (to emulate C++ virtual method tables for various classes). Now they are all placed into the heap and I just generate the code that creates new Table and put all the values to it inside the DllMain function. While the table is the global variable I am in doubt about its persistence and GC behaviour related to it. While it is a constant I would rather see it in some immutable segment of memory. I would like to know how exactly GC works. I also wonder if the runtime can cause a stack corruption below the call of the Nim function. I would like to try -d:useNimRtl but it is impossible while a bug is not fixed.

@Araq there are not so much cast s in my code, only the necessary amount to perform some hackery with virtual tables. In the example above the proc was not a closure I believe. And in both of cases its address does not belong to the heap. So addresses of string and seq in the first case neither belong to the heap. Well the actual question is why the addition of the proc ( cast ed of by construction of the Table[string, seq[proc()]] ) triggers the compiler to move string s and seq s to the heap. And why in C++ codegen they are already in the heap while in C - they are not.

Mirror of forum.nim-lang.org

3924 :: How compiler decides where to put the value?