Hi to all,
As I'm quite fresh to Nim, having problem to open and read txt files bigger then 2Gb on raspiOS 32 bit.
I'm getting response like this:
Error: unhandled exception: cannot open: ./big_2.4gb_file.txt [IOError]
I had similar problem with Clozure Lisp, 32 bit version that can't open files bigger than 2 Gb,...
Is there some work around, as I normally open such big files on same OS with python?
BR, Robert
Your issue is that you're trying to read the entire file into memory. Try using buffered reads like io.readLine, io.readBuffer, io.readChars, and io.readChar.
var file = open("test.file")
#echo readAll file # Tries to read all 10 GB of this file (I only have 8GB of ram!)
#echo readLine file # Reads a line of data, which is probably around 80 characters
try:
while true:
stdout.write file.readChar # Just reads a single character
except EOFError:
echo "\nEnd of file!"
Oh, no, that's not the case, I'm reading file line by line, something like this:
var line:string
for line in lines f:
..and so on....
same problem with this little program:
from os import fileExists
proc main =
var f = "./big.txt"
if os.fileExists(f):
echo "File exists, "
quit()
else:
echo "there is some problem"
main()
in case of file bigger than 2GB I get always error, but if file is for example 1.2GB, it works... I'm reading plain txt file.... tnx for reply!
Sounds like something is a signed 32 bit Integer... but what?
This page talks about https://comp.unix.programmer.narkive.com/rhRJ0ctl/fopen-file-size-limitation
"Use fopen64 or #define _FILE_OFFSET_BITS 64"
Maybe this could give some insight
I tried to compile and execute on raspiOS64 beta build 2021-03-04-buster and it runs without problem.
python3 on raspi32 is able to open such file, so this is probably bug in nim32 ??