Hi, I am looking for the elegant way to convert compile time seq into const array
So far what I have is:
const myarray = block:
var tmp: array[myseq.len, MyType]
for i in 0..< tmp.len:
tmp[i] = myseq[i]
tmp
Is there a better solution?
Thank you
Not sure if it is a better solution, but still an alternative
static:
var a = @[1, 2, 3]
from macros import newTree, nnkBracket, newLit
from sequtils import mapIt
macro convert(sq: static[seq[int]]): untyped =
newTree(nnkBracket, sq.mapIt(newLit(it)))
const b = convert(a)
echo b.repr
Except var {.compileTime.} would work better than static: var since the latter won't work in the next version.
I think the author was looking for a non-copy solution, which I don't think is possible currently in the VM
I'm trying @cdome 's example:
static:
var myseq = @[1,2,3]
const myarray = block:
var tmp: array[myseq.len, MyType]
for i in 0..< tmp.len:
tmp[i] = myseq[i]
tmp
Should this work or is there another way of converting compile-time seq to const array?
Use var {.compileTime.} instead of static blocks.
var myseq {.compileTime.} = @[1,2,3]
const myarray = block:
var tmp: array[myseq.len, MyType]
for i in 0..< tmp.len:
tmp[i] = myseq[i]
tmp
I just noticed that using this kind of compile-time magic ballons the size of the executable by a factor of about 2.
When using a regular array, for example: const myarray: array[4096, uint8] = [0, 1, 2, ...], the size of the executable is about 2.7 MB.
But when using the compile-time magic to read and parse the array from a text file, the executable size jumps to about 4.9 MB.
Anybody got any ideas why? (Win10 x64, Nim-devel, MSVC++)
I narrowed down the spike in executable size to when the compile-time created const array is assigned to a var! I need to pass the var to a C function using addr, that's why I'm using it. Example:
var array_bytes {.compileTime.}: array[5, seq[uint8]]
static:
var
array_strings = [
staticRead("images/hg_pixmap_0.txt"),
staticRead("images/hg_pixmap_1.txt"),
staticRead("images/hg_pixmap_2.txt"),
staticRead("images/hg_pixmap_3.txt"),
staticRead("images/hg_pixmap_4.txt")
]
i = 0
for arr in array_strings:
array_bytes[i] = newSeq[uint8]()
for x in arr.split(","):
if x.isNilOrWhitespace() == false:
var value = parseInt(x.strip()).uint8
array_bytes[i].add(value)
i += 1
const compiletime_to_runtime_array = block:
var temp: array[array_bytes.len, array[array_bytes[0].len, uint8]]
for x in 0 ..< length:
var tmp: array[array_bytes[0].len, uint8]
for i in 0 ..< tmp.len:
tmp[i] = array_bytes[x][i]
temp[x] = tmp
temp
var
# THIS PART INCREASES EXECUTABLE SIZE
pixmap_arrays = [
compiletime_to_runtime_array[0],
compiletime_to_runtime_array[1],
compiletime_to_runtime_array[2],
compiletime_to_runtime_array[3]
]
Any ideas why?
Hey @LeuGim, even if it was copied, each file is only 15kB in size. That would be 75kB for the 5 files that are read, but not 2MB.
I have no idea what's going on here.
Well, you can investigate resulting binary and assembler. Try to count numbers in first 4 files (you don't store the 5th into the the variable in your example) (their textual contents is not stored, just the numbers) and multiply it by the size of processor instruction, needed to store it. For me (x86, 32bit exe) each such instruction takes 7 bytes in the binary, and for my test with 2K numbers in each file the difference in binary size is 57017 bytes, which is very close to expected 2K*4*7=57344, the small difference is probably because of padding. So, the size just matches for me.
To get the size of the instruction you can fill your test files with one and the same number and just look at the resulting binary in a hex editor: you'll see a small repeating fragment through large part of the file, you'll recognize it instantly.
But that it copies each number may probably be bad for very big data sets. For my example though it's not noticeable.
B.t.w. your example doesn't compile as is (even when data files are created), so may be your actual code differs from it. Say there's undeclared variable length, I assumed its temp.len.
Try this:
import strutils
const cap = 0x400*1
const c = block:
var
array_strings = [
staticRead("images/hg_pixmap_0.txt"),
staticRead("images/hg_pixmap_1.txt"),
staticRead("images/hg_pixmap_2.txt"),
staticRead("images/hg_pixmap_3.txt"),
staticRead("images/hg_pixmap_4.txt")
]
var array_bytes {.compileTime.}: array[5, seq[uint8]]
for i, arr in array_strings:
array_bytes[i] = newSeq[uint8]()
for x in arr.split(","):
if x.isNilOrWhitespace() == false:
var value = parseInt(x.strip()).uint8
array_bytes[i].add(value)
array_bytes
var a = [
c[0],
c[1],
c[2],
c[3]
]
import random
echo c[rand(4)][rand(cap)]
This doesn't copy, as I understand, and at least in the resulting binary the numbers are represented just by themselves. That is by 1 byte for each uint8.
@LeuGim Excellent, no size bloat!
Can you please explain what is happening between my example and yours? Thanks.
P.S.: I just copy&pasted my code and quickly edited it, so as you noticed with the length, I deleted too much :)
I didn't look at why this happens, just changing array to seq in constant construction results in creating in C a construct struct filled with values (numbers), and they get as is into the executable, and using array results in an assignment of each value seaparately into the run-time variable, which is movb number, address (assembler), i.e. instruction code and memory offset (+ 6 bytes for me) are put into the binary for each number.
The code in previous post gives an array of read-only seqs. Just replacing arrays with seqs in your code will give you mutable seqs:
const compiletime_to_runtime_array = block:
var temp: array[array_bytes.len, seq[uint8]]
const length = temp.len
for x in 0 ..< length:
var tmp = newSeq[uint8]()
for i in 0 ..< array_bytes[0].len:
tmp.add array_bytes[x][i]
temp[x] = tmp
temp
const compiletime_to_runtime_array = block:
var temp: array[array_bytes.len, seq[uint8]]
const length = temp.len
for x in 0 ..< length:
for i in 0 ..< array_bytes[0].len:
temp[x].add array_bytes[x][i]
temp