nimforum mirror - Have I written a silly program or is there a "controlled" memory leak?

thegraynode (orginal) [2024-03-06T08:57:22+01:00] view original

Hi,

I have written a small program called nginwho for my company that parses nginx logs and stores them in a database. Here is the link to the repo:

https://github.com/pouriyajamshidi/nginwho

The program is compiled using -d:release --mm:orc --threads:off flags and Nim version 2.0.2:

> nim --version
Nim Compiler Version 2.0.2 [Linux: amd64]
Compiled at 2023-12-15
Copyright (c) 2006-2023 by Andreas Rumpf

git hash: c4c44d10df8a14204a75c34e499def200589cb7c
active boot switches: -d:release

While monitoring the process on one of our servers (and then others), we noticed it is consuming 100MB memory which seemed not right. So, I wrote a Bash script to continuously monitor the memory usage of this process and noticed this recurring pattern:


2024-03-05 18:13:13 -> 1 MB
2024-03-05 18:13:18 -> 2 MB
2024-03-05 18:13:28 -> 4 MB
2024-03-05 18:14:59 -> 5 MB
2024-03-05 18:16:49 -> 6 MB
2024-03-05 18:18:29 -> 7 MB
2024-03-05 18:19:49 -> 8 MB
2024-03-05 18:21:29 -> 9 MB
2024-03-05 21:17:17 -> 10 MB
2024-03-05 21:19:07 -> 11 MB
2024-03-05 21:20:48 -> 12 MB
2024-03-05 21:22:08 -> 13 MB
2024-03-05 21:24:18 -> 14 MB
...
2024-03-05 21:01:10 -> 104 MB
2024-03-05 21:02:50 -> 105 MB
2024-03-05 21:04:30 -> 106 MB
2024-03-05 21:05:50 -> 107 MB
2024-03-05 21:07:10 -> 1 MB
2024-03-05 21:07:15 -> 2 MB
2024-03-05 21:07:26 -> 4 MB

The consumption starts at 1MB and roughly every one minute and forty-fifty seconds it consumes another 1MB till it reaches exactly 107MB. Then the consumption goes back to 1MB and the story repeats.

Our nginx logs are rotated whenever they hit the 200KB size. Not sure what is happening here.

Could anyone please guide us? Is it something with the program itself?

I am trying to advocate Nim's usage within my company and I would like to not have unforeseen edge cases that ultimately push back this beautiful language.

sls1005 (orginal) [2024-03-06T10:52:20+01:00] view original

It always go to 107MB and back to 1MB? Never exceeds 107MB? When it goes back, always to 1MB and not 2MB? This happens automatically?

If so, I think there's no memory leak. If there is a memory leak, wouldn't it be something like 1MB -> 2MB -> ... 107MB -> 2MB -> 3MB -> ... -> 108MB -> 3MB -> 4MB -> ... -> 109MB -> ...?

Araq (orginal) [2024-03-06T11:47:05+01:00] view original

Please provide a file with test data, it's interesting.

PMunch (orginal) [2024-03-06T12:01:20+01:00] view original

Very hard to tell without actually seeing the program. Have you tried running it with -d:useMalloc? By default Nim has an internal memory pool. So when it frees things it doesn't necessarily free it back to the OS right away.

ingo (orginal) [2024-03-06T12:45:45+01:00] view original

database

OT for the question, as you use the SQLite library you don't need to open and close the 'connection' to it every time. You can open it once an leave it open until the program finishes, the close it. setControlCHook()

thegraynode (orginal) [2024-03-06T13:03:55+01:00] view original

It always goes to 107MB and back to 1MB. No one restarts the program. There is no manual intervention at all.

Each loop takes a few milliseconds so this memory usage does not make sense.

I have added the memory report log to the repository under the name of memreport.log

thegraynode (orginal) [2024-03-06T13:04:35+01:00] view original

Hi Araq,

I have added the nginx log, the executable and memory report log to the same repository.

thegraynode (orginal) [2024-03-06T13:06:12+01:00] view original

I have not tried with -d:useMalloc since I did not know about it.

However, I have added the memory log report file, one of our nginx logs and the executable to the same repository.

sls1005 (orginal) [2024-03-06T14:15:13+01:00] view original

According to my knowledge, "memory leak" means the memory is allocated and is not freed. Leaked memory does not go back until the program is terminated. If there's a memory leak, the amount of used memory will increase by each iteration (See the example in my last post). It will not go down to the exactly same amount as the previous iteration. And of it does, it's not "memory leak," but just the program using too much memory.

zevv (orginal) [2024-03-06T16:59:18+01:00] view original

Missing db.close() in the "Rows are already written to DB" return

brainproxy (orginal) [2024-03-06T17:09:16+01:00] view original

Try with --threads:on instead, even though you’re not using threads. If I recall correctly, it can impact ORC behavior in some ways.

thegraynode (orginal) [2024-03-07T18:40:37+01:00] view original

This did the trick! Thanks.

zevv (orginal) [2024-03-08T18:04:01+01:00] view original

valgrind is your friend.

giaco (orginal) [2024-03-09T15:40:39+01:00] view original

also compiler instrumentation is your friend

https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/

https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html

use it with --debugger:native

thegraynode (orginal) [2024-03-09T16:29:15+01:00] view original

Thank you everyone for your input, leading to solve the issue.

Although, it still remains a mystery for me that why the memory utilization never exceeded 107MB and somehow we always "reset" back to 1MB.

Mirror of forum.nim-lang.org

11163 :: Have I written a silly program or is there a "controlled" memory leak?