I'm not into writing tutorials right now, but I can answer any concrete questions. Jnim at this point allows "implementing" java classes in nim (still experimental feature though). So as long as you know how to write your app in java, it's just a matter of 1-to-1 porting.
One important point to consider is how pure-nim you wanna go with the toolchain. Surely we want to build apks without java, android sdk, gradle, etc. But then comes reality and you want to add e.g. firebase push notifications to your project, and it's straightforward for gradle, but for a hypothetical custom toolchain... realistically impossible, imo.
Then comes the runtime question, in particular: on which android thread is your nim "main" thread. Your mentioned use-case suggests nim main thread is equal to activity UI thread, but any high-fps opengl/vk app would likely prefer a secondary thread, as it is less runloop overhead, larger stack size, and allowed to "hang" without activity being killed (that might be a useful or harmful feature, depending on the usecase). And if pretty much all of your logic lives in secondary thread, it would be handy/logical to run nim "main" thread on it as well, like it is done with e.g. SDL2, ndk_native_glue (and thus glfm). And controlling this threading aspect from nim still remains unsolved afaik, partly because of nim threading nuances.
I'm not into writing tutorials right now, but I can answer any concrete questions.
I can appreciate that not everyone is into writing tutorials; thus I am proposing that I will write the tutorial so that others can refer to it from their documentation, including yourself, but I need some help getting it working so that I can document it...
Perhaps I need to start with a repo showing my problems so far in which I can produce an APK file, but can't install the file and you can quickly spot my problems and make suggestions on how I can improve the process.
Jnim at this point allows "implementing" java classes in nim (still experimental feature though).
It seem that many things in Nim are still "experimental" including the memory management model, but unless we provide some reasonably easy-to-use methods to use these features, they will never get beyond being "experimental".
One important point to consider is how pure-nim you wanna go with the toolchain. Surely we want to build apks without java, android sdk, gradle, etc....
The above would seem to be the @akavel/Dali approach, but other than problem with the restrictive license he chose, I see the same problems as you do with the required reverse engineering, etc. I also look at why we must insist on a pure Nim toolchain as to advantages, as we already have a usable NDK/JDK/Android Studio/Gradle toolchain and the main problem is that I don't want to be forced to write C/C++ code but just have a way to cause the output of the Nim compiler to be used instead. This is what I would like to document as to how to go about it.
Then comes the runtime question, in particular: on which android thread is your nim "main" thread. Your mentioned use-case suggests nim main thread is equal to activity UI thread, but any high-fps opengl/vk app would likely prefer a secondary thread ... controlling this threading aspect from nim still remains unsolved afaik, partly because of nim threading nuances.
Am I right that if Nim procedures are just called from Java/Kotlin through JNI that they will then just run on the activity thread from which they are called? If this is true, the threading control would be from the JDK end and would only be a problem if inter thread communication were needed at the Nim end, which isn't always required.
Also, I seem to see that Nim multi-threading on Android probably already works but the main problems are due to Nim's legacy GC'ed memory management. For simple app's, manual C-style memory management is likely enought, and for the future, --gc:arc or --newruntime will likely work (as per @araq's aims) and provide what is needed.
My main drive for trying to do this at this time is to be able to demonstrate the speed of a multi-threading algorithm on Android, which algorithm can actually be written to use manual memory management (although working --gc:arc, channels, and spawn'ing that worked with --gc:arc would make it more beautiful in terms of code and minimize extra "glue" code); however, in order to easily change test parameters, it would be good to have a user interface and an easy way to do it would seem to be as proposed.
I guess if the above is too difficult to get working, I could take the alternate approach of using a working example of SDL2 with a working interface and just patch in the algorithm I am trying to show working, but it would seem that SDL2 would be more complex am more something to be used for writing native games that what I am trying to do here.
Hi @GordonBGood!
As a quick side note regarding Dali, I'm actually still slowly chipping in at progressing with the project when I can, with my current goal being to try and write a very basic game-like graphical demo with it. My ultimate goal is also to be able to write GUI apps, but I decided a demo like this would be a nice intermediate milestone, while also giving me a bit more fun (which is the crucial currency in purely hobby projects), as well as being already useful to some people. However, even with that, I can't give any concrete timeline. With every step forward a new complication or bug tends to show up, that I must then find the reason of and resolve. And until I'm there, there's just nothing really to show. I will try to remember to ask you if you're still interested in writing some polished tutorial when I reach a point when I'll be starting to dig at it myself. That's an important and task indeed in my opinion too, and I found it also tends to require a lot of energy, so the more at it the merrier, I hope.
Now, having said that, I totally understand you being OK with using Android Studio. That's the reasonable choice. So, if you're interested in writing a tutorial for this approach, personally I would probably recommend to go with a path more or less like this:
- Try and start with bootstrapping a simplest possible Android NDK tutorial, and then making sure you can successfully run the resulting app on your Android device or simulator.
- Then, try and generate the same C code as in the tutorial, but from Nim. I suggest importing the jni_wrapper.nim module from yglukhov's jnim, it contains the same contents as "jni.h" in C, which is very simple and very useful for starters. Now, crucially, you need to write function with a name conforming to JNI protocol, so for function declared public native int world(); in file com/gordonbgood/HelloActivity.java, you would need something like a proc Java_com_gordonbgood_HelloActivity_world*(jenv: JNIEnvPtr; jthis: jobject): jint {.cdecl, exportc, dynlib.} = return 42 in your hello_jni.nim. Now, try to nim c this file with the Android NDK cross-compiler, see e.g. my nim.cfg from dali's hellomello demo. Again, if you succeed, verify that the .apk you build with Android Studio successfully runs on your Android device. If not... what can I say, try running adb logcat, then Ctrl-C it immediately after the crash, and backtrack to try finding some helpful-looking stacktraces that will hopefully hint you in right direction. Or maybe Android Studio has some helper tools for collecting and browsing logcat logs?
- If you managed to complete above step, you can try bringing in more of jnim. As yglukhov mentioned above, it's basically a hobby project too, so be prepared that you may stumble upon quite a lot of missing pieces, and maybe even bugs occasionally. The missing pieces you may be able to try and contribute back to the project, while the bugs you may either try to fix yourself, or report to yglukhov with as simple and helpful hints as possible, to make it easier for him to find time and energy to try resolving. Still, github.com/yglukhov/jnim together with github.com/yglukhov/android will save you from writing _a looooot of boilerplate. They're an absolute gem if you're trying to write Android NDK code in Nim.
- Optionally, you may then try to take a look at jexport macro in jnim; however, please note that this one is even more experimental than jnim itself. I'm actually trying to use it in Dali, in collaboration and with invaluable help and support from @yglukhov, so as to not duplicate effort/reinvent the wheel. However we're still regularly finding some areas in need of improvement there and slowly trying to bring it closer to general usability, with me working basically as a glorified crazy guinea pig, and @yglukhov patiently trying to help me in the especially narrow places on my way to the end of the labirynth (or at least something like that...)
- ???
- Profit!
But that's just my $0.02; maybe jnim gives some easier way, I'm not really sure as I'm personally not at all interested in using JDK/Android Studio, so didn't explore if there are any shortcuts possible on this path. But all in all, I believe the steps I outlined above are what you need to do some way or another to reach your goal, if I understood it correctly. I think they also serve as good "control points" along the way, to make sure you have a solid foundation before jumping one step higher, and thus less things to debug when something's not working right (and it will be...)
Good luck!
Hi @akavel:
I have quite a lot of time to spend on what I think are worthy projects, too, and find that easy-to-use tutorials tend to lead to advances along certain paths that otherwise tend to be ignored; I think one of those is the use of Nim for building Android apps.
I would probably recommend to go with a path more or less like this...
Yes, that is generally my approach and I have done the first few steps already. My problems are likely at the stage of introducing Nim to generate the C/C++ code in trying to do too much too soon in trying to compile for arm8a before just doing a basic arm build - the APK compiled but wouldn't install even though my phone is 64-bit Android with a arm8a SOC; I should have gone back a step to compile for just arm to be sure that works, and I'll get to that.
As to multi-threading, I have no slightest idea how it works with Nim and NDK on Android...
I actually know quite a bit about mutli-threading with Nim and also a fair amount about it with the JDK (which I assume should be about the same as for Android); also, I must assume that the basic Nim threading interfaces work with Android but I'll have to test to verify that once I get the interfaces between native Nim and Java/Kotlin working.
EDIT: One extra note: please bear in mind, that with the Android Studio path, you'll never be able to fully escape writing some Java/Kotlin code, esp. when using GUIs...
I'm not worried about writing the GUI glue code in Java/Kotlin; this isn't much different than what one has to do for a Web Assembly app running in a web page as the glue code needs to work through Javascript.
Also, the memory model between Java's garbage collector and Nim's garbage collector is somewhat tricky...
Yes, and I intend to avoid that for now by not using GC (--gc:none) and manual memory management for this particular project at least until --gc:arc gets more stable, as that would seem to be the path forward as per @araq's goals.
My general aims are to show that one can use the same Nim code base with very few changes to generate a web page by compiling directly to Javascript ("nim js"), compiling through emscripten to either asm.js or to web assembly, compiling to mobile apps as is the point of this thread, or obviously compiling to a desktop environment with a GUI of one's choice as would be commonly currently done.
...using Android Studio will let you include various JAR packages from the Internet in your project, while the Dali path won't, at least for a very long time.
Without knocking your project, that is why I've chosen this approach rather than your Dali approach.
Thanks for your suggestions and for your Dali project which got me started thinking of the possibilities of writing Android app's using Nim.
I wouldn't say everything is that experimental. I've been using nimx for android in production since 4 years ago, and I've got crashlytics integrated, so I kinda have some idea :). Java-nim memory model is pretty well defined in jnim, and it just works with nim default GC. The only thing experimental in jnim is jni_export which allows implementing Java classes. Including the main activity btw, so no need to stick to NativeActivity.
@GordonBGood offtopic, but i've got a wasmrt and jsbind projects which demonstrate binding to surrounding js environment without external js glue.
Now to the point. I don't know what's the exact problem you're hitting now, feel free to demonstrate it preferably with steps to reproduce. Also feel free to DM me on gitter or telegram for more streamlined communication.
@yglukhov: Now that I'm stuck at home and have lots of time to work on this, I've made it work as follows:
Most of the things that made it not work before were things not complete in other examples such as @akavel's "hellomello" or @treeform's GLFM project with people not really understanding the relationship between the Android GC and Nim's, with many recommending not using Nim's GC but using --gc:none or one of the new experimental ones; doing that isn't really satisfactory as one then can't easily use Nim's strings (or any other structure that uses heap data) and needs to do a lot of work generating cstrings themselves and manually memory managing them to avoid memory leaks.
In fact, the official Nim documentation contains most of what is needed, with @araq seeming have worked on this at one time, as follows:
- The need to call NimMain() before doing anything else is documented here. I think the very slickest way to do this for a dynamically loaded library is to use the JNI_OnLoad event so that is what I've done and will continue to do.
- The above link recommends using Nim as compile-only to generated the required C or C++ files implying that one then uses the Android Studio Gradle build scripts to do the rest, but that second step means we need to make "nimbase.h" available to Gradle to find, which is mentioned here.
There are things that I haven't seen mentioned anywhere as follows:
- Gradle is usually used to produce different "flavors" of the required APK/Bundle files containing APK's for the different supported ANDROID_ABI's (armeabi-v7a, arm64v8a and x86, and x86_64 currently) ans well as a "fat" APK containing all of them from which the Android installation system can select the native libraries required. That is easy for C/C++ where primitive types always have the same number of bits no matter the platform, but Nim has different int/uint bitedness depending on the platform and thus we need separate compilations for each CPU. I chose to use ndk-build Android.mk make files (I find them simplere to use) and adjusted them to account for the (four) different directories that must be used to cover this case.
- The above mentioned Android.mk file needs to be adjusted to refer to LOCAL_LIBS in order to add other Android capabilities such as logging.
- Theoretically, one could refer to the required "nimbase.h" file in place from the "Android.mk" file so one wouldn't need to make a copy into the code base, but I couldn't get that working nor did I see any forum posts where anyone has. If anyone can solve this, it would be appreciated!
In my projects, I have included a copy of @yglukhov's jnim "jni_wrapper.nim" file as a convenient way to use all the boilerplate, and have tried to include all due licence conformity.
So my first working project was to translate Android's NDK sample "hello-jni" from C to Nim, which is successful as per my repo <https://github.com/GordonBGood/NimHelloJNI>'. This example is similar to @akavel's "hellomello" project but doesn't depend on `NativeActivity, needs Android Studio (and its contained JDK) in order to be used, but is more complete due to the above reasons.
My second and more useful example was to translate Android's NDK sample "hello-jnicallback" from C to Nim, which is successful as per my repo <https://github.com/GordonBGood/NimHelloJNICallback>'; this includes all of the above but shows how to move data back and forth between native Nim and the Android Java/Kotlin code via called methods and includes a background thread as @glukhov mentioned using in one of the posts `above.
So I'm finally ready to write my Android benchmark app using what I have learned so far, and will post a link to the repo and comment on its development here when it is up and running, which I no longer have any doubts about being able to do. Although I have the Nim GC running, my actual multi-threaded code won't be using heap data to avoid the problems and overhead of multiple heap spaces, thus I will give up on the use of channels and threadpools which I don't regard as being stable enough for use until they are stable for use with the new -gc:arc; that would make the code more elegant but won't make any difference to the speed of the benchmark.
So, to get on with it...
So here is my version of a Maximally Factorized Sieve of Eratosthenes Benchmark as an Android app using an algorithm which is at least as fast as Kim Walich's primesieve on platforms on which they run in common up to the range of about 65 billion or so; it lacks the optimizations for more efficient sieving above this point in order to keep the code simpler and easier to understand. The required optimizations in order to extend the range to be the same are explained in the README file of the repo.
This work was done to show that one doesn't need to write code in C/C++ in order to be fast as per a Quora question on prime sieving, but the Quorum Moderators didn't accept my answer, likely because it uses Nim rather than C++ and shows that it doesn't take a €500 PC to perform the task of sieving to two billion in under one second but that it can be done on the least expensive of currently available smartphones and not using C++.
You can try this app by downloading the zip file from here, extracting the APK file, and side loading the APK file to an Android smartphone. The results should look something like the following:
@GordonBGood, great job! It's a pity the answer was deleted on Quora.
I've updated nimx naketools a few weeks ago to use ndk toolchain directly. So that compilation goes as follows: 1. Run nim to produce a static lib. 2. Repeat 1 for every arch. 3. Run gradle to link everything together. This solves quite a bunch of problems, like:
- nimbase.h dependency is gone
- {.compile.} pragma works
- {.passC.} pragma works
- Long paths problem on windows is gone
@yglukhov: glad you liked it. In a couple of weeks when I'm unblocked from editing, I'll remove or change a few links and try my answer again, as it seemed to be good:. It showed that the task didn't need C++ to do the job on a Intel i3-8100 or i3-9100 as the right algorithm (mine ;-) ) can do it in JavaScript, and if we use the same algorithm with a few extra tweaks made possible, an efficient native code producing language such as Nim can fulfill the task on the least expensive smartphone available.
I considered trying the approach you use in nimx (also couldn't one use a dynamic library as easily as a static one) for the reasons you give, but didn't do it because it seemed one would have to hard code the root location of the NDK in the make file. Or am I wrong? I might go back and try that.
also couldn't one use a dynamic library as easily as a static one?
By building static lib we offload the final linking to gradle. This gives an advantage to use non-nim dependencies more freely. E.g. you can pull in third-party native android libs in whatever gradle-compatible form they are distributed (source, or static lib, or dynlib) and it will just work. Otherwise if doing the .so from nim you would have to tell nim what to do about these dependencies, and source-distributed dependencies just will not work at all.
one would have to hard code the root location of the NDK in the make file
Firstly, there's no more makefile to care about :). But yes, you have to pass ndk clang path to nim, and have to locate the ndk somehow. Nimx naketools infer the ndk automatically from ANDROID_HOME env var, which is usually set whenever android sdk is used.
@glukhov:
Thanks for the advice on compiling as a static library; looks like I should give it a try for these repos.
Also, thanks for the pointer on using ANDROID_HOME to find the location of the Android NDK. However, it isn't automatically set on my Windows machines with Android Studio installed; does it always have to be set manually?