nimforum mirror - Segmentation fault in least_squares

rusty (orginal) [2025-09-30T17:08:22+02:00] view original

Hi,

I'm running into a segmentation fault in least_squares_solver from arraymancer. I've created a small test program that demonstrates it, but when I try running it on Playground all I get is "Unable to open output log". Here is the code:


import arraymancer, arraymancer/linear_algebra/least_squares, arraymancer/tensor/init_cpu, math,
  times, random, osproc
discard execCmd( "uname -a; echo; nim --version")
type F32 = float32
36290471.randomize
let nr = 1000
let nf = 170
var a = randomNormalTensor[ F32]( [nr, nf], 3.0, 2.0)
var b = randomNormalTensor[ F32]( nr, 3.0, 2.0)
echo "Before least_squares_solver ", getTime().utc
var x = least_squares_solver( a, b)
echo "After least_squares_solver ", getTime().utc

And here is the log from running it on my own computer:


Hint: used config file '/home/dave/lang/nim/nim-2.2.4/config/nim.cfg' [Conf]
Hint: used config file '/home/dave/lang/nim/nim-2.2.4/config/config.nims' [Conf]
.........................................................................
/home/dave/.nimble/pkgs2/nimblas-0.3.1-e1ecdea4bb8176f12d66efd4aa0c7b3bea970027/nimblas/private/common.nim(52, 7) Hint: Using BLAS library matching pattern: lib(blas|cblas|openblas).so(|.3|.2|.1|.0) [User]
...............................................................................................................................................................................................
/home/dave/.nimble/pkgs2/nimlapack-0.3.1-fcb25795c6fb43f9251b7f34c3ad84c39e645afd/nimlapack.nim(20, 7) Hint: Using LAPACK library matching pattern: liblapack.so(|.3|.2|.1|.0) [User]
.....................................................................
/home/dave/lang/nim/tst.nim(3951, 5) Hint: 'x' is declared but not used [XDeclaredButNotUsed]
CC: nim-2.2.4/lib/system.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/laser/dynamic_stack_arrays.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/laser/tensor/datatypes.nim
CC: nim-2.2.4/lib/posix/posix.nim
CC: nim-2.2.4/lib/pure/os.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/laser/tensor/initialization.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/init_cpu.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/private/p_shapeshifting.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/init_copy_cpu.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/private/p_accessors_macros_read.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/shapeshifting.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/operators_blas_l1.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/tensor/operators_broadcasted.nim
CC: ../../.nimble/pkgs2/nimlapack-0.3.1-fcb25795c6fb43f9251b7f34c3ad84c39e645afd/nimlapack.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/linear_algebra/helpers/least_squares_lapack.nim
CC: ../../.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/linear_algebra/least_squares.nim
CC: nim-2.2.4/lib/pure/osproc.nim
CC: tst.nim
Hint:  [Link]
Hint: mm: orc; threads: on; opt: none (DEBUG BUILD, `-d:release` generates faster code)
119998 lines; 2.620s; 270.098MiB peakmem; proj: /home/dave/lang/nim/tst.nim; out: /home/dave/lang/nim/tst [SuccessX]
Linux caprica2 5.19.0-35-generic #36~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 17 15:17:25 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Nim Compiler Version 2.2.4 [Linux: amd64]
Compiled at 2025-04-22
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: f7145dd26efeeeb6eeae6fff649db244d81b212d
active boot switches: -d:release
Before least_squares_solver 2025-09-30T14:21:07Z
Traceback (most recent call last)
/home/dave/lang/nim/tst.nim(3951) tst
/home/dave/.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/linear_algebra/least_squares.nim(31) least_squares_solver
/home/dave/.nimble/pkgs2/stb_image-2.5-abf5fd03e72ee4c316c50a0538b973e355dcb175/stb_image/read.nim(408) gelsd
/home/dave/lang/nim/nim-2.2.4/lib/system/alloc.nim(1165) dealloc
/home/dave/lang/nim/nim-2.2.4/lib/system/alloc.nim(1063) rawDealloc
/home/dave/lang/nim/nim-2.2.4/lib/system/alloc.nim(815) addToSharedFreeListBigChunks
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Segmentation fault (core dumped)

On my own computer, the failure happens either in least_squares_solver or in the code that examines its output. Exactly where it occurs seems to depend on the random number seed. The larger program in which least_squares_solver is called has run many times without segmentation faults, which only showed up when I added least_squares_solver.

Any ideas on what is going wrong or how to fix it? Or why I can't seem to run my program on Playground, which I have used successfully in the past?

Thanks.

Araq (orginal) [2025-09-30T17:33:48+02:00] view original

Does it work with -d:useMalloc?

rusty (orginal) [2025-09-30T17:50:22+02:00] view original

With -d:useMalloc I get this:


Before least_squares_solver 2025-09-30T15:43:31Z
munmap_chunk(): invalid pointer
Traceback (most recent call last)
/home/dave/lang/nim/tst.nim(3951) tst
/home/dave/.nimble/pkgs2/arraymancer-0.7.33-c37fb5d98fce8661ade439d89d0a6a3c9d65c8fc/arraymancer/linear_algebra/least_squares.nim(31) least_squares_solver
/home/dave/.nimble/pkgs2/stb_image-2.5-abf5fd03e72ee4c316c50a0538b973e355dcb175/stb_image/read.nim(408) gelsd
SIGABRT: Abnormal termination.
Aborted (core dumped)

elcritch (orginal) [2025-09-30T18:43:03+02:00] view original

Try running it with Valgrind, that should give more info.

rusty (orginal) [2025-10-01T06:05:03+02:00] view original

Thanks @elcritch for your suggestion. Running it with "valgrind --tool=memcheck" definitely caught some invalid memory writes, perhaps in lapack? I'm not sure how to proceed. Here is valgrind's output:


import arraymancer, arraymancer/linear_algebra/least_squares, arraymancer/tensor/init_cpu, math,
  times, random, osproc
discard execCmd( "uname -a; echo; nim --version")
type F32 = float32
36290471.randomize
let nr = 1000
let nf = 170
var a = randomNormalTensor[ F32]( [nr, nf], 3.0, 2.0)
var b = randomNormalTensor[ F32]( nr, 3.0, 2.0)
echo "Before least_squares_solver ", getTime().utc
var x = least_squares_solver( a, b)
echo "After least_squares_solver ", getTime().utc

Hint: used config file '/home/dave/lang/nim/nim-2.2.4/config/nim.cfg' [Conf]
Hint: used config file '/home/dave/lang/nim/nim-2.2.4/config/config.nims' [Conf]
........................................................................
/home/dave/.nimble/pkgs2/nimblas-0.3.1-e1ecdea4bb8176f12d66efd4aa0c7b3bea970027/nimblas/private/common.nim(52, 7) Hint: Using BLAS library matching pattern: lib(blas|cblas|openblas).so(|.3|.2|.1|.0) [User]
...............................................................................................................................................................................................
/home/dave/.nimble/pkgs2/nimlapack-0.3.1-fcb25795c6fb43f9251b7f34c3ad84c39e645afd/nimlapack.nim(20, 7) Hint: Using LAPACK library matching pattern: liblapack.so(|.3|.2|.1|.0) [User]
.....................................................................
/home/dave/lang/nim/tst.nim(11, 5) Hint: 'x' is declared but not used [XDeclaredButNotUsed]
Hint:  [Link]
Hint: mm: orc; threads: on; opt: none (DEBUG BUILD, `-d:release` generates faster code)
114597 lines; 3.088s; 270.113MiB peakmem; proj: /home/dave/lang/nim/tst.nim; out: /home/dave/lang/nim/tst [SuccessX]
==6734== Memcheck, a memory error detector
==6734== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6734== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==6734== Command: ./tst blah gorp zoom
==6734==
Linux hpz2ext 5.15.0-139-generic #149~20.04.1-Ubuntu SMP Wed Apr 16 08:29:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Nim Compiler Version 2.2.4 [Linux: amd64]
Compiled at 2025-04-22
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: f7145dd26efeeeb6eeae6fff649db244d81b212d
active boot switches: -d:release
Before least_squares_solver 2025-10-01T03:49:27Z
==6734== Invalid write of size 4
==6734==    at 0x53762E4: slamrg_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A3DE3: slasd7_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A385C: slasd6_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A59A9: slasda_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53755CD: slalsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x5314A83: sgelsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x19204E: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==  Address 0x4d48314 is 4 bytes after a block of size 11,568 alloc'd
==6734==    at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==6734==    by 0x134AB2: allocImpl__system_u1747 (in /home/dave/lang/nim/tst)
==6734==    by 0x134AD8: allocSharedImpl (in /home/dave/lang/nim/tst)
==6734==    by 0x137BE8: alignedAlloc__system_u1912 (in /home/dave/lang/nim/tst)
==6734==    by 0x137EFC: newSeqPayloadUninit (in /home/dave/lang/nim/tst)
==6734==    by 0x13E957: newSeqUninit__tst_u17579 (in /home/dave/lang/nim/tst)
==6734==    by 0x191C34: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==
==6734== Invalid write of size 4
==6734==    at 0x5376310: slamrg_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A3DE3: slasd7_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A385C: slasd6_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A59A9: slasda_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53755CD: slalsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x5314A83: sgelsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x19204E: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==  Address 0x4d48318 is 8 bytes after a block of size 11,568 alloc'd
==6734==    at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==6734==    by 0x134AB2: allocImpl__system_u1747 (in /home/dave/lang/nim/tst)
==6734==    by 0x134AD8: allocSharedImpl (in /home/dave/lang/nim/tst)
==6734==    by 0x137BE8: alignedAlloc__system_u1912 (in /home/dave/lang/nim/tst)
==6734==    by 0x137EFC: newSeqPayloadUninit (in /home/dave/lang/nim/tst)
==6734==    by 0x13E957: newSeqUninit__tst_u17579 (in /home/dave/lang/nim/tst)
==6734==    by 0x191C34: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==
==6734== Invalid read of size 4
==6734==    at 0x53A3E10: slasd7_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A385C: slasd6_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A59A9: slasda_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53755CD: slalsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x5314A83: sgelsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x19204E: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==  Address 0x4d48314 is 4 bytes after a block of size 11,568 alloc'd
==6734==    at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==6734==    by 0x134AB2: allocImpl__system_u1747 (in /home/dave/lang/nim/tst)
==6734==    by 0x134AD8: allocSharedImpl (in /home/dave/lang/nim/tst)
==6734==    by 0x137BE8: alignedAlloc__system_u1912 (in /home/dave/lang/nim/tst)
==6734==    by 0x137EFC: newSeqPayloadUninit (in /home/dave/lang/nim/tst)
==6734==    by 0x13E957: newSeqUninit__tst_u17579 (in /home/dave/lang/nim/tst)
==6734==    by 0x191C34: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==
==6734== Invalid read of size 4
==6734==    at 0x53A4575: slasd7_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A385C: slasd6_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53A59A9: slasda_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x53755CD: slalsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x5314A83: sgelsd_ (in /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0)
==6734==    by 0x19204E: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==  Address 0x4d48314 is 4 bytes after a block of size 11,568 alloc'd
==6734==    at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==6734==    by 0x134AB2: allocImpl__system_u1747 (in /home/dave/lang/nim/tst)
==6734==    by 0x134AD8: allocSharedImpl (in /home/dave/lang/nim/tst)
==6734==    by 0x137BE8: alignedAlloc__system_u1912 (in /home/dave/lang/nim/tst)
==6734==    by 0x137EFC: newSeqPayloadUninit (in /home/dave/lang/nim/tst)
==6734==    by 0x13E957: newSeqUninit__tst_u17579 (in /home/dave/lang/nim/tst)
==6734==    by 0x191C34: gelsd__tst_u164 (in /home/dave/lang/nim/tst)
==6734==    by 0x193D38: least_squares_solver__tst_u144 (in /home/dave/lang/nim/tst)
==6734==    by 0x197CAE: NimMainModule (in /home/dave/lang/nim/tst)
==6734==    by 0x197943: NimMainInner (in /home/dave/lang/nim/tst)
==6734==    by 0x197958: NimMain (in /home/dave/lang/nim/tst)
==6734==    by 0x197996: main (in /home/dave/lang/nim/tst)
==6734==
After least_squares_solver 2025-10-01T03:49:32Z
==6734==
==6734== HEAP SUMMARY:
==6734==     in use at exit: 24,794 bytes in 23 blocks
==6734==   total heap usage: 1,753 allocs, 1,730 frees, 1,608,642 bytes allocated
==6734==
==6734== LEAK SUMMARY:
==6734==    definitely lost: 0 bytes in 0 blocks
==6734==    indirectly lost: 0 bytes in 0 blocks
==6734==      possibly lost: 0 bytes in 0 blocks
==6734==    still reachable: 24,794 bytes in 23 blocks
==6734==         suppressed: 0 bytes in 0 blocks
==6734== Rerun with --leak-check=full to see details of leaked memory
==6734==
==6734== For lists of detected and suppressed errors, rerun with: -s
==6734== ERROR SUMMARY: 315 errors from 4 contexts (suppressed: 0 from 0)

elcritch (orginal) [2025-10-02T02:11:03+02:00] view original

Definitely looks like something related to LAPACK. Threading can cause similar issues. I'd recommend try running it with --mm:atomicArc as well.

Honestly the quickest for this sort of problem is to set up Claude Code or ChatGPT Codex in a CLI. Tell it to run the program with valgrind and to find the issue. It'll be able to search your deps and find the issue likely quicker than most people could.

Note I'd recommend using Atlas to install in a deps/ folder or Nimble develop mode. Then Claude Code or Codex GPT can search the dependencies.

rusty (orginal) [2025-10-02T10:22:03+02:00] view original

A few more things I observed: The memory access errors occur on multiple x86_64 machines running either Ubuntu Linux 20.04 or 22.04, and with lapack 3.9.0 or 3.10.0. They appear dependent on the values of my "nf" variable (number of columns), but not on "nr" (number of rows). With nf = 42, the access errors "disappear", even with millions of rows.

I was hoping to run my test code on Playground to determine whether it's a general problem with arraymancer/lapack or just a quirk of my particular systems or nim installations. I can get echo "Hello, world" running on Playground, but whenever I try something that imports arraymancer, the machine grinds around for a while but finally gives up with "Unable to open output log". This happens whether I'm logged into forum or not. Does anyone know why Playground is failing this way? Should I post a separate topic about this?

I've also downloaded and built lapack 3.12.1 to see if by any chance that fixes the problem. But by default make only builds a static library (liblapack.a), not a liblapack.so, and from what I've read it's a bit tricky to modify the various build files so they produce a dynamic library. Is it possible to get nim to use my liblapack.a (without rebuilding nim itself)?

Araq (orginal) [2025-10-02T11:09:14+02:00] view original

Isn't there a setting not to use libblas but Arraymancer's own implementation?

rusty (orginal) [2025-10-02T12:52:42+02:00] view original

Actually, it's lapack that I think I need to substitute, not blas. In any case, the Arraymancer index has entries for something called auxiliary_lapack as well as auxiliary_blas, but neither of those pages seem to have anything to do with using a different version of either lapack or blas.

rusty (orginal) [2025-10-02T13:08:02+02:00] view original

Ah, I found this flag: -d:lapack=lapacklibname. Am testing it now.

rusty (orginal) [2025-10-05T00:55:38+02:00] view original

Thanks @cblake for the advice, and I appreciate the help I'm getting on this forum from you and the other contributors. As it turns out I am already familiar with most of the principles you mention. But I'm relatively new to nim, and the program I am working on is basically an exercise in learning the language and evaluating its strengths and weaknesses (so far I'm fairly favorably impressed). I've been programming computers since 1962 (!), and in the 1980s I started specializing in predictive analytics. I retired from my "day job" in 2009, and since then I have continued to program as a hobby, including competing in numerous machine learning contests, mostly at kaggle.com, using a variety of algorithms such as boosted tree ensembles (xgboost, lightgbm, etc.), and neural nets (keras, tensorflow).

I started working on my first nim project about a month ago. It's a simple checkers (draughts) playing program. The basic infrastructure is working: alpha-beta tree search with iterative deepening, simple evaluation function, positions hash table. Recently I decided to add some machine learning, so I started exploring what nim provides in this area. For the last few years I've been coding mostly in Python, and being familiar with numpy I saw arraymancer as a good place to start. I'm focusing on linear regression not because the data are particularly suited to it but because I wanted to get something simple working fairly quickly before advancing to more complicated methods like neural nets.

Unfortunately these invalid memory writes have dampened my enthusiasm for nim somewhat, even if the problem may not strictly be nim's fault but perhaps lies elsewhere, like in lapack. In my long programming career I've played detective many times, chasing down bugs both in my code and other people's, and I have developed an ingrained instinct to try to get to the bottom of these kinds of problems before moving on.

rusty (orginal) [2025-10-13T00:28:33+02:00] view original

A few more notes regarding this invalid memory write bug:

I installed Ubuntu 25.10, which was just released, on an x86_64 machine that I wasn't using, together with a fresh copy of nim from nim-2.2.4-linux_x64.tar.xz. The OS came with lapack 3.12.1, which is the latest version. The segfault persists with nf (number of columns) = 170, but runs okay with nf = 100. This is the same behavior I've observed on my other systems.

I tried using arraymancer's pca (principal components analysis) function, and that runs okay on my data (no segfaults), even with large numbers of rows and columns.

I've also made a brief (and so far futile) attempt to make sense of memory allocation and use in gelsd in lapack, which looks like it might be the culprit.

I'm no longer so keen on resolving this problem, since I've abandoned linear models and am now calling the boosted tree ensemble program lightgbm from nim by writing my data to files and invoking the standalone lightgbm executable via execCmd. However, I'd still be interested, time permitting, in trying to get the lapack developers to look into this issue by demonstrating that it occurs with a program in a language they might be more familiar with than nim.

rusty (orginal) [2025-10-14T02:38:16+02:00] view original

I think the bug is most likely not in lapack, but in nim's interface to gelsd: linear_algebra/least_squares.nim and linear_algebra/helpers/least_squares_lapack.nim

rusty (orginal) [2025-11-01T20:13:40+01:00] view original

I just installed nim-2.2.6, but unfortunately the segmentation fault in least_squares_solver persists, and valgrind reports the same invalid memory writes as in 2.2.4. Since I suspect that the bug is in arraymancer, should I report this problem on its github page?

rusty (orginal) [2025-11-03T02:58:02+01:00] view original

I posted this issue over at github: https://github.com/mratsim/Arraymancer/issues/683

rusty (orginal) [2025-11-10T18:30:50+01:00] view original

Looks like this was a real bug in least_squares_solver in arraymancer:

https://github.com/mratsim/Arraymancer/issues/683

https://github.com/mratsim/Arraymancer/pull/685

However, I'm not sure how to access the fixed version. I tried 'nimble install arraymancer', which seemed to succeed, but my test still segfaults. Shouldn't that work, or do I need to clone the github repository?

thegraynode (orginal) [2025-11-10T19:05:11+01:00] view original

Since they have not had a release since last year, Nimble is using the old version. I think you could give nimble a github repo + commit hash to ensure you are using the latest version, but unfortunately I do not remember the syntax.

Abathargh (orginal) [2025-11-10T19:12:19+01:00] view original

You can use the head from the git repo or a specific commit by specifying it in your nimble file:

https://nim-lang.github.io/nimble/use-packages.html#installing-a-specific-version

rusty (orginal) [2025-11-11T09:09:39+01:00] view original

Thanks @Abathargh. I ran nimble install "arraymancer@#head", and my test program no longer segfaults.

Mirror of forum.nim-lang.org

13434 :: Segmentation fault in least_squares_solver from arraymancer