Mirror of forum.nim-lang.org

9633 :: How to manually free/clear memory used by ref

• ThomasTJdev (orginal) [2022-11-19T07:53:19+01:00] view original

  Problem

  I'm using the superb nimPDF library for generating PDF-files in various shapes and forms, but for some of my very large files the memory usage is becoming a problem. I generate the PDF-files within a loop, so I can keep headings, preloaded images and so on ready - but after each loop, the previous used memory is not freed.

  Question

  My main goal is to have way to free or clear the memory between the loops. Can I manually free the var doc in a safe way?

  Example pseudo code

  import nimPDF/nimPDF
  
  var opts = newPDFOptions()
  opts.addImagesPath("pic")
  var doc = newPDF(opts)
  
  for i in 0..10:
    doc.addPage(getSizeFromName("A4"), PGO_PORTRAIT)
    doc.drawText(15, 15, "Hello World!")
    
    var image = doc.loadImage("large-pic.jpg")
    var x = 40.0
    var y = 40.0
    doc.drawImage(x, y, image)
    
    if not doc.writePDF("hello.pdf"):
      echo "cannot open: hello.pdf"
    
    echo "------------------"
    GC_fullCollect()
    echo GC_getStatistics()
    
    # Can I free the memory here? In the next line I reassign the doc, but the memory used above is still present.
    doc = newPDF(opts)
  Run

  Stats

  When running in 4 loops with some pretty heavy PDF files these are the stats. I can just continue to run these loops, and the memory usage increases steadily.

  
  # Compile options
  nim version == 1.6.0
  nimPDF version == 0.4.3
  switch("gc", "orc")
  
  ------
  
  # Loop1 - 425mb filesize
  [GC] occupied memory: 648737024
  [GC] total memory: 810221568
  
  # Loop2 - 423mb filesize
  [GC] occupied memory: 1941934384
  [GC] total memory: 2394058752
  
  # Loop3 - 435mb filesize
  [GC] occupied memory: 2865117408
  [GC] total memory: 3682131968
  
  # Loop4 - 1.2gb filesize
  [GC] occupied memory: 2506910752
  [GC] total memory: 5641519104
  
  ------
  
  
  +------+--+---------------+----------------+--+-------------------+----------------+--+-------------+
  | Loop |  | Occupied (mb) | Diff from prev |  | Total memory (mb) | Diff from prev |  | Filesize MB |
  +------+--+---------------+----------------+--+-------------------+----------------+--+-------------+
  |    1 |  |           649 |              0 |  |               810 |              0 |  |         425 |
  |    2 |  |          1942 |           1293 |  |              2394 |           1584 |  |         423 |
  |    3 |  |          2865 |            923 |  |              3682 |           1288 |  |         435 |
  |    4 |  |          2507 |           -358 |  |              5642 |           1959 |  |        1200 |
  +------+--+---------------+----------------+--+-------------------+----------------+--+-------------+
  
  
  Run

---

• Araq (orginal) [2022-11-19T08:45:13+01:00] view original

  Please try --mm:orc; maybe combined with -d:useMalloc.

---

• ThomasTJdev (orginal) [2022-11-22T07:51:33+01:00] view original

  Hi @Araq,

  Thanks for the suggestion. My current solution already uses orc, but using -d:useMalloc was actually helping on smaller files and loops, but when the files are +500mb it is status quo and it still does not free the memory. I have tested with newest stable nim (1.6.8) and various combination of compile flags.

  Is there another path I can investigate? Otherwise then I might just go with spawning new processes or threads.

---

• Araq (orginal) [2022-11-22T10:11:11+01:00] view original

  "still does not free the memory" sounds like you expect the allocator to return memory back to the OS. Few allocators do this well IME. You can try to link against mimalloc or similar (all combined with -d:useMalloc) to see if it makes any difference. However, not returning memory back to the OS is usually not as big of a problem as it would seem thanks to the memory management being virtual.

---

• ThomasTJdev (orginal) [2022-11-22T11:35:48+01:00] view original

  Yes, I expected it to return the memory back to the OS. It might just be a specific problem for us due to the microservice consumes to much memory for too long time. I'll try out the linking options, and otherwise I just go with a process spawn-way. If I find a solution I'll report it back here. Thanks for the help!

---