Hi! I would like to share with you a hobby project I've been working on recently. The idea & motivation behind it is, to try and enable writing native (.apk) Android apps with Nim, without having to install and use Android Studio. What I want to share with you is the first major success on this path — a showcase JNI-based .apk, with Dalvik bytecode assembled purely with Nim, and the ARM code written in Nim and cross-compiled with Android NDK.
The project at this point of life is composed of the following elements:
- hellomello — a sample "hello world" project, with step-by-step instructions how to build it;
- dali — Dalvik Assembler/Linker — a small Nim package, implementing an assembler for Dalvik bytecode, emitting .dex files; (this repo currently also accidentally contains part of the hellomello sources, that should really be moved into the hellomello repo; sorry that I haven't cleaned this it up yet)
- marco — Manifest and Resource Compiler for .apk — a small Nim tool for compiling AndroidManifest.xml files from regular XML to a binary format required for .apk;
- apksigner — a small tool for cryptographically signing .apk files — NOTE: this one is unfortunately currently written in Go; it was faster for me to prototype it this way, as I had trouble finding appropriate crypto library in Nim. I'd be super grateful if someone could help me with this part (I need the following primitives: parsing a PKCS#8 file with priv. key; parsing an x509 PEM file with cert; signing a message using a PKCS#7 signature/format) — with your help, we could make this project 100% Nim-based.
- Android NDK — the cross-compilation toolchain (standalone variant, as is the default now) — At this stage of the project, I managed to get rid of Android Studio/SDK, but we still need the NDK. Notably, you can actually already live without Android NDK, as long as you plan to write your whole app purely in Dalvik assembly (example — a hello-world). But if you prefer to write as much as possible in Nim, you'll currently need to cross-compile it as a JNI library, and the NDK is what provides cross-compilers.
The resulting .apk file works on my phone (a Samsung Galaxy A3 2017), displaying a simple "Hello world from Nim" message, as can be seen on the following (I'm afraid not very glamorous) screenshot:
Build Steps
To reproduce the hellomello .apk, follow the steps listed in the README of the hellomello project. (Please tell me if something doesn't work for you. I tried to test/verify the instructions after listing them, but I could miss or break something.)
Various Notes
Completion status. I must admit I find it surprisingly hard for me to adequately describe the status of the project at this moment. On one hand, it's kinda "alpha/prototype quality" — in that it's not a "complete, polished, high-level solution". Notably, if it crashes on you (which may happen!), please don't be scared! I would basically treat it as a signal meaning: "(some part of) a functionality you need is not yet implemented; you're welcome to add it if you like!" On the other hand, I believe where it works, it just works. I mean, after you write some new app, you have to check if it works on an Android device, and fix any errors emitted by the Android verifier into the adb logcat console. But Dali is an assembler, which means it's low-level, and that's kinda how one works with assemblers. However, once you make your app install and open without Android verifier errors, I would expect it should just work thereafter. So, should I call it a "prototype"? an "MVP"? "Technology preview"? Not sure.
Minimal Android version. So, that's a bit tricky. I basically don't know. I tested the resulting .apk on a device I own, which has Android 8. There are a few places where the specs required me to put some "minimal version", and I just used whatever value worked for me (from what I recall currently: in the .dex header and in manifest files). I have no idea if the .apk will work on earlier Androids, or not. Please have a look at the next paragraph as to why I myself don't plan to test with earlier versions.
Support. It's important to note, that as of now, this is purely a hobby project for me. I'm very much open to sponsoring/hire to work on it full time (and actually, now would be the very best moment to contact me if you're interested, as I'm very actively looking for a new job — please email me); though unless that happens, I'm very limited in time & resources I can allocate to this project. As such, for the time being, I'm afraid I cannot afford to provide any support for anyone interested in this project, unless hired/sponsored (see below). You're more than welcome to report any issues, questions, and send pull requests, and you can be sure I will read them, and heartfully appreciate them. But even for PRs, I cannot promise I will give them enough due attention; I may, or may not. I will try to at least respond with a "thank you" and a reminder of no promise of support. I'm sorry, I just cannot stretch my available time any more. The chances of me responding are probably highest for questions, as those are easiest to respond to. I'm especially happy to answer any questions in this thread on this forum over the next several days, though even here I might not be able to do that quickly (may take me a few days). Please note you're welcome to fork this project, and modify it to your needs. My target goal is actually writing some personal apps for Android, while avoiding Java/JVM, so if you make it possible with this project faster than I, please just let me know!
Future, plans, ideas. To tell the truth, I'm currently not 100% decided where I want to take the project further. That's because there are really many possibilities, and I'm not sure which one to choose! :) One of the first ideas I had, would be to add .dex/.apk as a backend for the Nim compiler. Notably, I believe it could be surprisingly reasonable and simple, given how high-level the Dalvik bytecode is. Specifically, I believe it would be most similar to the JS backend, even more than to the C backend, not even mentioning any kind of typical machine code. That's because in Dalvik bytecode, you don't have to worry about stuff like garbage collection, call stack management, or (mostly) register allocation (roughly, you can just allocate each local variable to one of the 65535 available function-local registers). I found the official Dalvik bytecode reference very simple, readable and approachable. In case of doubt, the smali wiki clears up some remaining uncertainties. In collaboration with the awesome Richard Jansson (I recommend you take a look at his innovative text input methods project!), we started some initial experiment/exploration in this direction. I believe it is a promising path, and should be relatively simple; but I also estimate it's just too much for me to be able to pursue purely in my spare time. (If you're interested, as I already said, I'm open to hire/sponsoring — see below; alternatively, please feel free to fork this project.) A second avenue I'm pondering is making dali more high-level, a.k.a. a "macro assembler", potentially as a Nim DSL via macros. Maybe even taking it as far as some kind of half-baked language targeting Dalvik; maybe even Java-like? (Given that Java appears to be not really that far from the raw Dalvik bytecode.) A third direction I thought of would be to maybe actually try and write some real apps in Nim with JNI, and see how it works for my needs. Maybe try using nimx for GUI? Explore the status of touch events support for nimx and/or JNI apps in general? I would also love to get rid of the NDK in this case if possible; I started experimenting with maybe using Zig as a cross-compiler instead of the NDK, but I am not there yet. Personally, I'm mainly interested in GUI apps. On the other hand, the current state of the project should be an especially good fit also for exploring writing Android games in Nim. I believe it might be not far from just changing the hellomello project to use a NativeActivity (in AndroidManifest.xml and in the .dex). But I don't know anything about writing games for Android; so, either you'll have to explore this further on your own (but feel free to ask me on any aspects related to dali/marco/...), or, as I said, you can consider sponsoring me to help you — see below.
Sponsoring/hire. If you're interested in hiring/sponsoring me to work on this project, and to take it further in some direction interesting for both of us, please contact me by email (czapkofan@gmail.com). I'd be very happy to get involved in such a collaboration. Please do it ASAP, as I'm currently in the process of searching for a new job. I'm super open and eager now (we can agree on starting a bit later this year), but once I start working at some other place that I'll like, I won't expect to be open to switch any longer for quite some time.
References & internals — dali. I wrote the dali .dex assembler based primarily on an awesome PDF with an annotated diagram of a hello-world .dex binary, by Ange Albertini. I consulted the official .dex spec and the official Dalvik bytecode spec along the way. The specs are small, clear, readable and simple, so it was really fairly straightforward. I'm very happy that I initially said "let's try how far I can get," instead of worrying what could go wrong. Especially the Dalvik bytecode is surprisingly (to me) high level and reasonably easy to write. At high level, the main function rendering a .dex file is basically 155 lines long, while the whole assembler, including types is only 600 lines of Nim, including comments. I'm very happy how Nim worked for me on this project. I see it as having a great balance of expressivity and readability, with static typing being of immense help, while also non-intrusive.
References & internals — .apk files. I built my understanding of .apk files based on the ApkGolf project, and the accompanying article, which aims to construct a simplest possible .apk file. Thanks to this project and its author (Jamie Lynch), I managed to understand that there's not really much needed to package a .dex into an .apk. So I went on and wrote the rest of the necessary tools (marco & apksigner) in Nim.
References & internals — marco. This app currently can only compile the simplest AndroidManifest.xml files (notably, it can't emit binary resource files yet), but extending it should be easy enough. The project is currently just 250 lines of Nim, and I listed some most useful references I found in the readme. My own initial mode of operation was analyzing a minimal binary .xml file, together with the sources of a Java parser/clone of aapt for reference.
References & internals — apksigner. This app is the simplest one at the high level, as it just takes a .zip file (containing classes.dex & binary AndroidManifest.xml), then calculates SHA1 hashes of the archived files, and signs them with a user-provided private key + cert. Finally, it writes the hashes & signature as a few new files in a META-INF/ directory in the .zip. Unfortunately, to keep prototyping fast, I had to write it in Go, as I had trouble finding appropriate crypto libraries for Nim, and being a Nim newbie, I didn't know how to correctly wrap OpenSSL (?) for this either. In Go, I fortunately managed to quickly find some packages that worked for me, so I was able to quickly verify if writing this app makes sense, which was crucial at that time and phase. I'd be super grateful if someone could help me rewrite this in Nim now, at least by helping to get access to some necessary crypto primitives. Specifically (as listed earlier), I desperately need to be able to:
- parse a PKCS#8 file, to load a private key;
- parse an x509 PEM file, to load a certificate;
- sign a message and emit a PKCS#7-formatted signature.
Could you help me with that?
Nimble & hellomello. Apart from some help porting apksigner to Nim, I would really appreciate if someone could help me make the sample hellomello project "self contained", with Nimble. I understand Nim uses NimScript for building, but I never used it, and am not sure where to even start and how to arrange it, between Nimble and nim.cfg. I would love if it could be possible to tell a new user to, say, "just clone hellomello and run nimble build", and have all the remaining steps done automatically by Nimble/NimScript. This way, I would hope forking hellomello could be made the whole procedure required to start a new Android project in Nim. Could you help me with that? (Notably, the jni_hello.nim file, emitting the bytecode wrapper for the JNI should be moved from the dali repository into the hellomello repo.) Other than that, I'm not sure how to best publish dali+marco+apksigner via Nimble. Should I publish them separately? Or would it maybe be a good idea to eventually merge them into a single tool, with subcommands? But dali is currently only a library... Or, maybe I shouldn't publish them yet to the Nimble directory, if they're still at a very early phase?
That's all for now — hope you like it! :)
a hobby project I've been working on recently
This is an impressively large and complex project for a "hobby" project lol!
I don't have anything to add other than this is a really cool project, congratulations, and thank you for sharing.
Thanks for the good words! :) I don't really find it large nor complex, each of the tools proved to be surprisingly simple and not doing too much after exploring their code; though now that you say so, I maybe haven't conveyed this well enough in the code of the dali assembler in particular; maybe I should at least put in more of some encouraging & clarifying comments. Also so I won't forget the details myself in the future...? :P That may actually be a really good idea for some most immediate work on the project. Though I'd love to try and automate the "step by step" instructions for hellomello using Nimble too; if no one has time to help me with this, I'll probably try to do it myself.
By the way, a quick update: I did a successful experiment, where I managed to cross-compile a simple C JNI library with Zig's builtin C cross-compiler (Clang), targetting armv7, and to verify it runs successfully in an .apk. I believe this is very close to confirming that it should hopefully indeed be possible to get rid of the ~2.5GB Android NDK and replace it with the ~100MB Zig package for cross-compiling Nim programs to Android. Though I didn't have enough time to test it on the Nim-generated C files yet.
@akavel, from your comment on lobste.rs:
this may be solved if there’s some IDE support (i.e. “go to definition”) available; I haven’t searched for it yet.
I've successfully used go to definition, both in VSCode and Neovim (with nim.nvim plugin).
Things I dislike are: too hard to find stuff/browse in the standard library docs (Go docs are the gold standard for me in this area); stdlib APIs are a mixed bag — sometimes great, usually acceptable, sometimes messy, occasionally downright braindead.
Could you go into more detail about these?
What could be improved to make docs better? Did you use the index for browsing/finding stuff?
Can you give some examples of APIs in every mentioned category? I'm most interested in the latter two categories, i.e. maybe we can fix some of those for v0.20.
As to the problems I see with docs, please note I already took some effort to try and explain them, but it feels to me like it was dismissed or ignored. So I find it hard to muster more energy to try and explain it again, maybe in different words, if it looks like no one's really interested in hearing. I know it's not a very glamorous area to contribute, so I'm trying to avoid moaning with critique while not contributing actual code for this myself. But trying to analyze the essence of what problems I see is also an effort for me, so if someone really wants it, I'd like to at least feel heard and understood. In other words, in UX, a solution to "I have a problem" is not in saying "because you're doing it wrong", a.k.a. blaming the victim, but rather it should be assumed the UX has a problem, which needs to be actively understood, identified, and attempted to be fixed.
As to the index... first of all, I didn't even realize it's there and I should use it, until I was told about it. Secondly, personally I still haven't really found a good way to make it useful to me. The most painful problem I see with both the index and the grouped docs is that of discoverability, that is, browsing for something I need, I know what it's related to, but don't know how it's named. For example, for string-related functions, I must check strutils, but if not found there, then also search for all occurrences of : string in the "system" package. In the index, it's even worse. Whereas in Go, at least for more complex types, methods are grouped by the type they operate on. Also, the index in godoc is very useful for taking a quick glance at what's available in a package. I hunger to see better grouping in Nim docs. And less of the meaningless, noisy repetition (i.e. tons of the repeated == operators making the "side panel" index in "system" library useless -- I mean not The Index, but the per-package index on left side, as shown in the image in the thread I linked to above).
As to the APIs, I can't quickly invoke much specifics now, I'd have to browse through the docs to try and list what I mean. Though as to braindead, yes, I'm afraid I feel so about the part of the API of the sha1 package, where AFAIK I can't easily get the actual bytes of the SHA sum, in non-serialized format (i.e., not converted to hex). I cannot understand why I seem to have to go through parseHexStr($secureHash(s)), or how could I avoid it. If it's possible to do it easier, I would love to see this documented explicitly, as I wasn't able to grasp it. I considered asking about this in an issue, but, somehow, didn't collect enough courage to do it yet.
GNU Affero General Public License v3.0 😤👎👎👎
Pretty much the best license out there, for network apps.
Hi
Richard here! Thank you so much for mentioning me! I am very happy to see that you continued on the project! The reason that I have not gotten back to you is because I was in an accident for two months ago and lost a month of time, my computer and of course all of my logins. Along with that two months or so worth of nim code. Nim code that was part of my open source research on input methods.
As akavel mentioned I participated to a small extent to this project. In particular in investigating on how to create a dalvik backend for NIM! I do sincerely believe that this would be a very interesting milestone for the nim-sphere as a whole.
I would argue against native interface for two reasons:
- use of x86 android for convinient debugging in native virtual machines
- the plethora of x86 android devices
The fundamentals for implementing a third language is already there as most of the hard work already have been done in a sense for C and JS. Also there are a lot of cunning helpful people here with knowledge that they can share.
As for the PR-aspects this could be a huge deal for nim! Practical non bloated development that can target web, backend and mobile would make a killer deal. Think how many apps that in practice only are web pages wrapped in thin layers. node.js got it's success from joining backend and frontend in a convinient package. If a language / environment could join these three I see a lot of potential.
As akavel goes into the dalvik is not as an scary beast as it would seem. Being a virtual machine there are a quite a few nice high level features there to make the job easier. I do not think the PR aspects for supporting this platform should be underated. Also it is a lot of fun to get into the gritty details of the subject!!
Still one option that would be more in line with the project in general would be to simply generate plain java code and pass that along to a java compiler. I'm sure that there are no shortage of java developers here.
Unfortunately I am due to abovementioned accident stranded and broke in Turkey and am forced to do my work on a Android 5.1 tablet with 1 GB RAM. Even though it''s good to develop for low end devices as to make resource efficient code I can tell you it's not to pleasant as an development machine. More importantly it rules out coding in nim. So before I can help the nim project I have to find a way to help my self :S
For my new github: https://github.com/TBF-RnD/alpha
Needless any support/donations/comments would be really apriciated!