The use of an array is very restricted since its length has to be known at compile time. On the other hand, the flexibility of a seq which can change its length during its lifetime and which can even be reallocated at lifetime, is not necessary in many cases.
Many applications read the dimensions of a problem at runtime, but then arrays have fixed sized after their definition. It's posiible in C++ like
int getdimension();
...
int N = getdimension();
double A[N];
What to do in Nim in that case?
Use let.
What you are trying to do in C++ is a VLA (variable-length array), and has many restrictions besides the possibility to blow up your stack if you try to allocate more than a couple hundreds of bytes.
@mratsim For me the C++ example is not a variable length array since it cannot change size of the declaration. Why should it blow up the stack? On a 64bit machine the stack segment can be really large and it can grow. I have definitely used objects of several megabytes on the stack.
And each reasonable programmer would check the size of N before using it for allocating arrays.
Sounds like you should wrap the C alloca interfaces and do your own container either like UncheckedArray[T] or like seq[T].
There is nothing fundamentally wrong with growing your stack big on many machines these days, but deployment can be tricky. Limits are often still set up with 1990s era hardware assumptions. And stack size limits are, well, very variable..from 64 KiB to 8 MiB is typical. The original idea was to crash programs with unbounded recursions rather than crash your system from OOM.
Like limits on open files, what sorts of leaking/unboundedness errors people make may/may not be what was commonly assumed when the OS was designed and/or the culture of default settings in force today was established. This can honestly just vary among people and prog.langs - some make auto-closing files pretty easy, for example. The limits are sort of a poor man's guard rail for C programmers. For example, it was common to see a soft limit of only 64 open files on many Unix boxes until quite recently, and I would be unsurprised if when this limit was introduced it was considered generous under some rationale like "recursive directory tree walks are never that deep".
Re overflow - I usually configure generous limits. To not rely upon sysadmin configs on Unix, you can probably just unlimit it with setrlimit at the start of your program with Nim's posix.setrlimit. "Probably" because the sysadmin can have low hard limits...So, you should be careful to handle errors there and error out with a "fix your hard limit this way..." kind of message. In the (probably) rare case that the user has no admin rights or cannot persuade the admin to bump them up, there could be a deployment problem.
It seems possible on Windows, but you may need to do some passC/passL type mojo or have a special build unless someone has made that part easier in Nim.
Anyway, to clarify @mratsim's warning - you don't have to avoid using lots of stack memory, but if you do plan on doing that a little extra care like the above ideas will be appreciated by your users.
Thanks, I'll have a look at wrapping alloca, but I am pessimistic:
The problem arose when I tried to return a seq within a parallel for loop. I fear (currently) a parallel for loop doesn't work with custom types. Here is a slightly modified version of an example given by ynfle which fails to compile
import strutils, math, threadpool
type
Coeff = object
c: array[1,float]
proc term(k: int): Coeff =
result = Coeff(c: [(if (k and 1) == 0 : 1.0 else: -1.0) * 4.0/(float(k+k) + 1.0)])
proc pi(n: int): float =
var ch = newSeq[FlowVar[Coeff]](n+1)
{.push experimental: "parallel".}
parallel:
for k in 0..ch.high:
ch[k] = spawn term(k)
# type mismatch: got 'Coeff' for 'spawn term(k), nimCreateFlowVar' but expected 'FlowVar[Vx.Coeff]'
for k in 0..ch.high:
result += (^ch[k]).c[0]
{.pop.}
echo formatFloat(pi(5000))
In general I strongly discourage using the parallel construct. I tried to look into what went inside before writing Weave to check if something was usable. I didn't really understand what was going on, in particular in terms of checks and rules for Flowvar lifting.
Anyway here is the fixed code
import strutils, math, threadpool
type
Coeff = object
c: array[1,float]
proc term(k: int): Coeff =
result = Coeff(c: [(if (k and 1) == 0 : 1.0 else: -1.0) * 4.0/(float(k+k) + 1.0)])
proc pi(n: int): float =
var ch = newSeq[Coeff](n+1)
{.push experimental: "parallel".}
parallel:
for k in 0..ch.high:
ch[k] = spawn term(k)
# type mismatch: got 'Coeff' for 'spawn term(k), nimCreateFlowVar' but expected 'FlowVar[Vx.Coeff]'
for k in 0..ch.high:
result += ch[k].c[0]
{.pop.}
echo formatFloat(pi(5000))
Also, the nimbleverse has vla. It's pushing 4 years old and looks under-tested, but it might help @HJarausch as a starting point.
If no one points out a more up-to-date thing, I would encourage @HJarausch to do their own though, maybe call it StackArray or ArrayA or something. (The last 'a' in alloca is for "auto" like C's auto because it is automatically released post function call return.) Maybe look into how to deploy well on Windows & others and document that with your new package and contribute a new little atom. :-)
If we end up with a new container between an array and a seq I think we should call it an arreq.
...🥶🤣
So, I'd like to ask the Delphic Oracle what the future of multi threading will be in Nim.
https://github.com/nim-lang/RFCs/issues/244 is one piece of the puzzle. https://github.com/nim-lang/RFCs/issues/347 is another.
There is at least one other common type of memory (not counting GPUs) - from memfiles (aka mmap, memory views, etc.). It is both more Wild West regulated than stack (in terms of limits - both number of mappings and total size), but also (typically) less Wild West in access rights since people can/do make overt decisions about read-write-execute permissions. (You can do that after the fact with things like mprotect, of course, but that is far more rare to see.)
mmaps/memory views share the run-time not compile-time length, and also the usually fixed after creation aspect of the vla/stackarray/arreqs. (There is mremap on Linux, but that is not portable.) Also, Zig has some "every creation proc takes an allocator object handle" kind of pattern.
I don't know how much (if any) of this will factor into @HJarausch's design, but it seemed worth mentioning in case he didn't know/it didn't occur to him to consider it.
vla is broken since it relies on the unchecked pragma, which was removed from the language.
Perhaps I missed the announcement of its removal, or of a deprecation period? It's generally considered a bad thing to just remove features that people may be relying on. Castles built on sand, and all that.