Hi all!
I'm planning to use nim in a large project with most code written in C++. Our project used lua to do some computation but the increasing amount of data made them unbarably slow now.
I admit that I just discovered nim a few days ago and only wrote a few tiny programs. Most of them are benchmark programs that mimic our usage (complex data structure access and computation).
But that's enough to give me a feeling that nim is the best candidate so far: it's fast, easy to learn (the plugins are intended to be written by programmers with average skill, not experts), and easy to embed into our C++ codebase.
According to my (not very typical) benchmarks, nim is as fast as C, C++ and Rust, and significantly faster than V8 and LuaJIT (I only tested these two because I believe they can represent the most advanced JIT compilers for dynamic languages).
Our use case is a bit special: some of the computation is done in client side (currently lua called from C++), then send some intermediate result to the server via HTTP, then the server do the rest (currently lua called from Java).
The two parts (lua called from C++ and Java) have a lot of common so actually they are in a single codebase, developed independently of our project. Each lua module receives a JSON and produced another JSON which made it even slower (parsing/encoding JSON takes more than 1/2 time)
My current plan is to use cap'n proto for both 1. sharing data between the C++ and nim, 2. as a serialization format to call our HTTP service, but my first obstacle is... capnp.nim does not compile :(
C:\Users\aaa>nimble install capnp
Downloading https://github.com/zielmicha/capnp.nim using git
Verifying dependencies for [email protected]
Info: Dependency on collections@>= 0.1.2 already satisfied
Verifying dependencies for [email protected]
Installing [email protected]
Building capnp/capnp/capnpc.exe using c backend
C:\Users\aaa\.nimble\pkgs\collections-0.5.2\collections\iface.nim(146, 62) Error: undeclared identifier: '!'
Tip: 4 messages have been suppressed, use --verbose to show them.
Error: Build failed for package: capnp
... Execution failed with exit code 1
... Command: D:\dev\nim-1.4.2_x64\nim-1.4.2\bin\nim.exe c --colors:on --noNimblePath -d:release -d:NimblePkgVersion=0.0.3 --path:C:\Users\aaa\.nimble\pkgs\collections-0.5.2 --hints:off -o:C:\Users\aaa\AppData\Local\Temp\nimble_22320\githubcom_zielmichacapnpnim\capnp\capnpc.exe C:\Users\aaa\AppData\Local\Temp\nimble_22320\githubcom_zielmichacapnpnim\capnp\capnpc.nim
(I'm using nim 1.4.2 x64 on windows)
Also, it'd be great if we can hot-reload a nim plugin just like lua. I've heard of NimScript but I don't know whether it's as fast as native nim, because it's executed by NimVM. Also, I haven't found whether nim has a bytecode-like intermediate structure so that we can at least pre-compile a nimscript and store the "bytecode" in memory, so that it can be executed faster later.
Anyone using capnp.nim in production? Any comments on capnp.nim, capnp itself or my plan in general, are all appreciated. I hope I can integrate nim into our project with great success :)
I've never used capnp protocol. While it's intriguing, I never liked that it only appeared to have one implementation.
You might try messagepack (or cbor) as an alternative binary format thats json-likebut faster. I've used it extensively in a few domains as its faster and produces smaller than json. Though perhaps protobuf or thrift might fit your use case and have more stable wrappers.
Thanks for your suggestions! I wrote an email to capnp.nim's author and he replied that the project is no longer maintained. However, after evaluating flatbuffer I think it's even more appropriate. Simpler encoding scheme and much smaller generated source code. There is no official nim support, but I found this:
https://github.com/RecruitMain707/NimFlatbuffers
However, I can't find a way to ask the author. If any one happen to know him, could you politely ask him the status of the project? It looks like a complete set of (runtime) library functions but no codegen. But it looks like much easier to develop than cap'n proto so I might be able to continue even if I failed to contact him.
I already tried CBOR, but there will be still significant overhead of copying data around. There are many small routines (in different C++ -> plugin calls) that will random access the shared data, so I'd like a zero-copy shared buffer instead of a traditional serialization method.
Thanks for your comments about NimScript and WASM. I'm already using WASM in production on server-side (inside a rust service), but never tried client side embedded in C++. Fast WASM runtime with JIT (like wasmtime or wasmer) feels big to me (compared to nim generated C codes), but of course, that's still a viable option. Thanks!
Thank you!!! I'm excited to see your reply, and also your updated repository. I'm using a lot of strings and vectors so I guess I had to change the codegen myself. I don't have schema file yet because we haven't even started migrating to flatbuffers, but I guess I will start that after 1~2 months.
I'll try to test NimFlatbuffers with some handmade schemas and data that's similar to our actual use case ASAP and let you know what I feel. And It would be great if you can add some tests of the runtime library, showing how to build/get data without a codegen. When we're sure runtime library is feature complete and stable enough, adding codegen should not be a problem.
I'm getting some errrs:
D:\dev\NimFlatbuffers-master>nimble test
Executing task test in D:\dev\NimFlatbuffers-master\Nimflatbuffers.nimble
Hint: used config file 'D:\dev\nim-1.4.2_x64\nim-1.4.2\config\nim.cfg' [Conf]
Hint: used config file 'D:\dev\nim-1.4.2_x64\nim-1.4.2\config\config.nims' [Conf]
....
D:\dev\NimFlatbuffers-master\tests\test1.nim(1, 8) Error: cannot open file: ../flatn
D:\dev\NimFlatbuffers-master\tests\test1.nim(2, 1) Error: undeclared identifier: 'generateCode'
D:\dev\NimFlatbuffers-master\tests\test1.nim(2, 13) Error: attempting to call routine: 'generateCode'
found 'generateCode' of kind 'unknown'
D:\dev\NimFlatbuffers-master\tests\test1.nim(2, 13) Error: attempting to call routine: 'generateCode'
found 'generateCode' of kind 'unknown'
D:\dev\NimFlatbuffers-master\tests\test1.nim(2, 13) Error: expression 'generateCode' cannot be called
D:\dev\NimFlatbuffers-master\tests\test1.nim(3, 14) Error: cannot open file: output/rlbot_flat
stack trace: (most recent call last)
C:\Users\eryiju\AppData\Local\Temp\nimblecache-0\nimscriptapi_299490439.nim(187, 16)
D:\dev\NimFlatbuffers-master\Nimflatbuffers.nimble(11, 8) testTask
D:\dev\nim-1.4.2_x64\nim-1.4.2\lib\system\nimscript.nim(260, 7) exec
D:\dev\nim-1.4.2_x64\nim-1.4.2\lib\system\nimscript.nim(260, 7) Error: unhandled exception: FAILED: nim check tests/test1 [OSError]
Tip: 1 messages have been suppressed, use --verbose to show them.
Error: Exception raised during nimble script execution
What am I doing wrong here? Maybe something stupid because I'm still a novice :(
Thank you for your effort! I think I'm getting close. I created a file which does nothing except importing the module, but still cannot compile:
D:\dev\NimFlatbuffers-master>type import_only.nim
import Nimflatbuffers
D:\dev\NimFlatbuffers-master>nim c import_only.nim
Hint: used config file 'D:\dev\nim-1.4.2_x64\nim-1.4.2\config\nim.cfg' [Conf]
Hint: used config file 'D:\dev\nim-1.4.2_x64\nim-1.4.2\config\config.nims' [Conf]
................................
D:\dev\NimFlatbuffers-master\src\Nimflatbuffers\flatn\codegen\Codegen.nim(563, 6) Hint: 'newTableUnionSetter' is declared but not used [XDeclaredButNotUsed]
......
D:\dev\NimFlatbuffers-master\src\Nimflatbuffers\nimflatbuffers\src\builder.nim(1, 14) Warning: imported and not used: 'sugar' [UnusedImport]
.
D:\dev\NimFlatbuffers-master\src\Nimflatbuffers.nim(6, 8) Error: cannot export: builder
And I cannot even directly compile that src\Nimflatbuffers.nim file:
D:\dev\NimFlatbuffers-master\src>nim c Nimflatbuffers.nim
Hint: used config file 'D:\dev\nim-1.4.2_x64\nim-1.4.2\config\nim.cfg' [Conf]
Hint: used config file 'D:\dev\nim-1.4.2_x64\nim-1.4.2\config\config.nims' [Conf]
...............................
D:\dev\NimFlatbuffers-master\src\Nimflatbuffers\flatn\codegen\Codegen.nim(563, 6) Hint: 'newTableUnionSetter' is declared but not used [XDeclaredButNotUsed]
......
D:\dev\NimFlatbuffers-master\src\Nimflatbuffers\nimflatbuffers\src\builder.nim(1, 14) Warning: imported and not used: 'sugar' [UnusedImport]
.
D:\dev\NimFlatbuffers-master\src\Nimflatbuffers.nim(6, 8) Error: cannot export: builder
The (almost) same error going one level further (error message omitted), but the final level (compiling D:\\dev\\NimFlatbuffers-master\\src\\Nimflatbuffers\\nimflatbuffers\\Nimflatbuffers.nim) works!
Then I tried to change the problematic line (need to change two Nimflatbuffers.nim files) from
export builder, table, struct
export nimflatbuffers.builder, nimflatbuffers.table, nimflatbuffers.struct
Then every file, including tests\test1.nim compiled. However, executing test1.exe seems to give me nothing (I emptied output directory but no files are generated).