Hi, I have mydiff.py to compare two files using two sets. my python version is as follows but nim version takes five times longer. Can anyone explain the performance difference or give me a better nim code [I am comparing about two million lines of files: dba01:/oracledba/jlee/python_scripts>time mydiff.py check_diff_dsdan17.lst check_diff_dsosu27.lst > /dev/null
real 0m8.274s user 0m4.661s sys 0m0.779s dba01:/oracledba/jlee/python_scripts>time mydiffn (nim) check_diff_dsdan17.lst check_diff_dsosu27.lst > /dev/null
real 0m39.135s user 0m25.623s sys 0m1.711s
mydiff.py:
import sys
if len(sys.argv) != 3:
print "Usage: %s file1 file2" % sys.argv[0]
sys.exit(1)
try:
file1 = open(sys.argv[1])
file2 = open(sys.argv[2])
except IOError, e:
print "file not found: %s" % e
sys.exit(2)
old_lines = file1.read().split('\n')
new_lines = file2.read().split('\n')
file1.close()
file2.close()
old_lines_set = set(old_lines)
new_lines_set = set(new_lines)
old_added = old_lines_set - new_lines_set
old_removed = new_lines_set - old_lines_set
for line in old_lines:
if line in old_added:
print '-', line.strip()
elif line in old_removed:
print '+', line.strip()
for line in new_lines:
if line in old_added:
print '-', line.strip()
elif line in old_removed:
print '+', line.strip()
import os, sets, strutils
if paramCount() != 2:
echo "mydiff file1 file2"
quit()
let file1 = open(paramStr(1))
let file2 = open(paramStr(2))
let old_lines = file1.readAll().splitLines()
let new_lines = file2.readAll().splitLines()
file1.close()
file2.close()
let old_lines_set = toSet(old_lines)
let new_lines_set = toSet(new_lines)
let old_added = old_lines_set - new_lines_set
let old_removed = new_lines_set - old_lines_set
for line in old_lines:
if line in old_added:
echo "-", line.strip()
elif line in old_removed:
echo "+", line.strip()
for line in new_lines:
if line in old_added:
echo "-", line.strip()
elif line in old_removed:
echo "+", line.strip()
thanks Joseph
I tested for 7.7 mb files:
python2 ... 0,54s user 0,23s system 54% cpu 1,417 total
nim release ... 0,56s user 0,18s system 54% cpu 1,337 total
nim debug ... 3,96s user 0,30s system 99% cpu 4,262 total
By the way you can replace it:
let file1 = open(paramStr(1))
let file2 = open(paramStr(2))
let old_lines = file1.readAll().splitLines()
let new_lines = file2.readAll().splitLines()
file1.close()
file2.close()
let old_lines = readFile(paramStr(1)).splitLines()
let new_lines = readFile(paramStr(2)).splitLines()
Now I resolved this issue using gcc rather than tcc.
tcc version took 25 sec while gcc version 5 sec comparable to python version!
I thought tcc faster than gcc but nim using gcc makes faster on linux.
Of course gcc will optimize much better than tcc!
Also put your nim code in a proc for some more performance. Here's what parts are taking most of the time: https://i.imgur.com/hLaOHVe.png
Just to add to the conversation:
Each file is 1.7MB
System specs
OS: Ubuntu 16.04 xenial
Kernel: x86_64 Linux 4.4.0-83-generic
CPU: Intel Core i5-6600K @ 3.9GHz
RAM: 15980MiB
Python 2.7
$ time python test1.py test1.log test1-2.log
real 0m0.042s
user 0m0.040s
sys 0m0.000s
Nim 0.17.2
$ nim c -d:release test1.nim
$ time ./test1 test1.log test1-2.log
real 0m0.056s
user 0m0.044s
sys 0m0.008s
Nim 0.17.2 (DEBUG)
$ nim c -o:test1-dev test1.nim
$ time ./test1-dev test1.log test1-2.log
real 0m0.273s
user 0m0.264s
sys 0m0.008s
Times seem correct to me.
We have to remember that the majority of Python's file I/O stuff is implemented in C. The fact that Nim is ever so slightly slower than native compiled C is not surprising.
On my system, compiling with -d:release gives a 5x speed increase. This is also not surprising.
@def
What program is that?
kcachegrind:
nim -d:release --debugger:native c mydiff
valgrind --tool=callgrind ./mydiff file1 file2
kcachegrind callgrind.out.*
Also used here for example: https://hookrace.net/blog/writing-an-async-logger-in-nim/#optimization
As mentioned in another thread, Python memoizes the hash value for strings, which yields faster processing for Set operations.
You can easily do this yourself with Nim by creating an object with the pre-computed hash value:
type
HString = object
str: string
hash: Hash
and supplying a corresponding hash proc:
proc hash(str: HString): Hash =
str.hash
Using those, the Nim version becomes ~20% faster than the Python version on my system.
Full code example: https://gist.github.com/aboisvert/2daf9ed487214d810e0689d3e3a3714f
@def
Thank you.
@boia01: If a faster hash function is used, your code is one third faster than python.
For those who want to try: clone xxHash right next to the Nim source file and insert this below the import statement:
import hashes
const hashSeed: culonglong = 0x4FB3CC02B6AA901C'u64
{.passC: "-DXXH_PRIVATE_API -IxxHash".}
proc XXH64(key: pointer, size: csize, seed: culonglong): culonglong
{.importc, header: "xxhash.h".}
proc hash(s: string): Hash =
result = XXH64(cstring(s), s.len, hashSeed).int