proc wait(message: string) =
write(stdout, message)
discard readLine(stdin)
proc allocate(n: int) =
wait("Press enter to allocate memory...")
var x = newSeq[uint8](n)
wait("Press enter to free memory...")
allocate(50_000_000)
allocate(50_000_000)
wait("Press enter to exit...")Shouldn't sequence memory be garbage collected?
I'm open to suggestions...
I just tried that. The third allocation dropped memory usage down to 164 KB (under 64-bit Linux) just as if --deadCodeElim:on had been used.
Then a forth allocation raised memory usage to 50 MB.
Maybe the GC works only after a certain number of cycles?
First, I don't see a difference between --deadCodeElim:on and --deadCodeElim:off on my system.
Second, the GC kicks in the first time after >4 MB of memory are allocated or the zero count table has 500 entries. What you're experiencing is probably old references being kept alive because the stack is being scanned conservatively.
Indeed there is no difference between --deadCodeElim:on and --deadCodeElim:off when not using -d:release. In this case memory usage switches between 50 MB and 100 MB with each allocation / deallocation.
When using -d:release you experience the behavior I've mentioned only with --deadCodeElim:off. Otherwise the sequence is never allocated at all, since it is never really being used.
I checked on another PC today (again 64-bit Linux + GCC 4.9.2) and found the exact same behavior.
Then I tried on a third PC running 32-bit Windows XP + GCC 4.8.1 with -d:release. In this case the first memory allocation got 50 MB and from the second memory allocation on, memory usage was constantly 100 MB.
So maybe these findings relate to the version of GCC being used.
I also checked how malloc() and calloc() of GCC 4.9.2 perform for multiple allocations and deallocations of memory:
How are you checking memory usage? GC_getstatistics() or getOccupiedMem() ? And when do you check it?
The behavior of malloc() and calloc() should be irrelevant, since Nim uses mmap() and munmap() on POSIX systems to obtain memory from the system (and virtualAlloc()/virtualFree() on Windows).
Note that because of conservative stack scanning, you can get all kinds of different behavior about when exactly a chunk of memory is being freed, so it's totally possible to have all or none of the big seqs being freed. The only thing I question is your claim that with --deadCodeElim:On memory gets never allocated (because the allocation call simply does not get optimized away, so that's not possible). What you're more likely seeing is that a small allocation elsewhere (possibly as part of the readLine call) triggers a GC and frees the allocated memory. Note that for large allocations, the Nim GC can return the memory to the OS via munmap() or virtualFree() when they're freed.
...Dead code elimination only removes unreachable functions...
Not necessarily. If Nim is passing a switch to GCC/VCC, then dead code detection is not just unreachable functions, but also code that is determined as irrelevant, and the var x = newSequint8 statement could certainly meet that criteria, since it doesn't contribute to a function return or output.
GravityWell: Not necessarily. If Nim is passing a switch to GCC/VCC, then dead code detection is not just unreachable functions, but also code that is determined as irrelevant, and the var x = newSeq[uint8](n) statement could certainly meet that criteria, since it doesn't contribute to a function return or output.
Let me be precise. First, when I said that "dead code elimination only removes unreachable functions", I was describing the effect of the the compiler switch --deadCodeElim:on (technically, it also removes unneeded variables, but that's not relevant here).
Second, gcc/vcc/clang cannot optimize this away, either. Memory allocation is not a pure function; in fact, most of what it does is cause side effects to the allocator's internal structures (and also a system call to mmap()). That the result isn't returned to the original caller is absolutely irrelevant. No C compiler is allowed to optimize this away without correctly inferring that these side effects can be omitted in their entirely because the next GC cycle will remove it (and even that is questionable, because you're also removing a system call that changes the memory map of the process). This is pretty much impossible for even the best state of the art compilers; they would have to conjecture and prove not only that these side effects are eventually reverted, but also that intermittent allocations (which change the GC's data structures in non-trivial ways) essentially do not affect that property. It would then have to transform the program in such a way as to satisfy (e.g.) ยง5.1.2.3 of the C99 standard, which is essentially impossible (given that there are any intermittent allocations that do not 100% commute with the original allocation).
Ah, mmap()! I see now... VM memory is allocated but no memory usage should be visible in the system until the relevant pages are accessed.
Of course both getOccupiedMem() and GC_getStatistics() show this memory as being allocated.
Then, when --deadCodeElim:off is used, some quirk causes system memory to be visibly allocated at times.
Thank you very much, Jehan, for clarifying things to me.