I begun to port Nim to a custom operating system (here mentioned as "myOS"). The initial goal was to be able to compile a small hello world program and run it on the custom OS. In order make this happen I had to do the following steps.
In the file compiler/platform.nim I had to add an entry for the OS.
type
TSystemOS* = enum # Also add OS in initialization section and alias
# conditionals to condsyms (end of module).
osNone, osDos, osWindows, osOs2, osLinux, osMorphos, osSkyos, osSolaris,
osIrix, osNetbsd, osFreebsd, osOpenbsd, osDragonfly, osCrossos, osAix, osPalmos, osQnx,
osAmiga, osAtari, osNetware, osMacos, osMacosx, osIos, osHaiku, osAndroid, osVxWorks
osGenode, osJS, osNimVM, osStandalone, osNintendoSwitch, osFreeRTOS, osZephyr, **osMyOS**, osAny
...
const
OS*: array[succ(low(TSystemOS))..high(TSystemOS), TInfoOS] = [
...
(name: "Zephyr", parDir: "..", dllFrmt: "lib$1.so", altDirSep: "/",
objExt: ".o", newLine: "\x0A", pathSep: ":", dirSep: "/",
scriptExt: ".sh", curDir: ".", exeExt: "", extSep: ".",
props: {ospPosix}),
#added entry
(name: "myOS", parDir: "..", dllFrmt: "lib$1.so", altDirSep: "/",
objExt: ".o", newLine: "\x0A", pathSep: ":", dirSep: "/",
scriptExt: ".sh", curDir: ".", exeExt: ".exe", extSep: ".",
props: {}),
#-----------
(name: "Any", parDir: "..", dllFrmt: "lib$1.so", altDirSep: "/",
objExt: ".o", newLine: "\x0A", pathSep: ":", dirSep: "/",
scriptExt: ".sh", curDir: ".", exeExt: "", extSep: ".",
props: {}),
]
This solution is really nice, that many of the OS dependent settings are in one place. Imagine if this would be scattered around in several files, you wouldn't be able to find it. What would perhaps be even more convenient is to have a configuration file that the compiler parses, but this is good enough. One question I have is if this information is for the host compiler only or is this also used for the actual binary target?
In lib/system/platforms.nim you also need to add an entry for the OS.
OsPlatform* {.pure.} = enum ## the OS this program will run on.
none, dos, windows, os2, linux, morphos, skyos, solaris,
irix, netbsd, freebsd, openbsd, aix, palmos, qnx, amiga,
atari, netware, macos, macosx, haiku, android, js, standalone, nintendoswitch, **myos**
...
const
targetOS* = ...
elif defined(nintendoswitch): OsPlatform.nintendoswitch
elif defined(myos): OsPlatform.myos
else: OsPlatform.none
## the OS this program will run on.
A small change in lib/system/dyncalls.nim was needed. We can wait with the dynamic library support for now.
elif defined(nintendoswitch) or defined(freertos) or defined(zephyr) or defined(myos):
proc nimUnloadLibrary(lib: LibHandle) =
cstderr.rawWrite("nimUnLoadLibrary not implemented")
cstderr.rawWrite("\n")
rawQuit(1)
To support memory allocation changes in lib/system/osalloc.nim was needed
proc osDeallocPages(p: pointer, size: int) {.inline.} =
if bumpPointer-size == cast[int](p):
dec bumpPointer, size
elif defined(myos) and not defined(StandaloneHeapSize):
include myos/alloc # osAllocPages, osTryAllocPages, osDeallocPages
else:
{.error: "Port memory manager to your platform".}
First the people who develop Genode really showed how it should be done. Instead of inserting everything in the same file, just include another implementation file which is much cleaner. I decided to copy this way of doing it. The implementation of osAllocPages, osTryAllocPages and osDeallocPages was very easy by calling the custom OS interface and it doesn't need to be more complicated than that. This is a very low bar to reach to enable the memory management.
The file config/nim.cfg was changed in order to give custom parameters for the C compiler.
...
arm.myos.clang.exe = "clang"
...
@if myos:
clang.options.always = "--target=arm-none-eabi ..............."
clang.cpp.options.always = "--target=arm-none-eabi ............"
@end
I didn't specify any linker because I use cmake to link the binary files. This was because I have already an infrastructure to link to different types of targets as well it opens up for mixed Nim,C,C++ projects. Nim is a bit weird that it outputs several C and object files with for humans unpredictable file names. Also build systems don't like this because they want to work with files that they know about before hand. In order overcome this I set Nim to compile a static library with a specific name. Now cmake has a file name it can trigger on.
This was basically all that was needed to produce a basic binary. Good thing that nim uses standard file write with stdout when you call echo so that worked right out of the box since I have a C library that partially works. Compared to other languages I have worked with this porting was very quick and in this case Nim didn't drag in anything from standard library so that I had to port everything in one go. Often other languages use a runtime where you have to port (or stub) everything in order to get it to compile.
Now this is where it starts to become difficult, even with Nim. I decided to try out passing command line parameters. By doing so you need to "import os". This imports a complete ball of files. Many of these files are like oscommon.nim, ossymlink.nim which contains code for file system functionality. Do you need the file system in order to read command line parameters?
Here it would be more beneficial if the standard library had a more pay as you go approach. You don't include things you don't need which should be possible with Nim as the standard library is a bunch of source files. Operating systems have a wide variety what they support beyond basic memory management. They might have a file system but do not implement any access rights management. They implement basic file system functionality but don't have things like moving, copying files implemented because they don't care about that. Network support is not always there etc. To have a more pay as you go approach makes it easier to port as you can do it gradually.
Did you look into our os.nim refactorings? It's now split up into different topics.
Nim is a bit weird that it outputs several C and object files with for humans unpredictable file names.
Yes but it also generates a nimcache/myproject.json file that lists these C and object files. cmake could be patched to make use of this json file.
it outputs several C and object files with for humans unpredictable file names.
It's predictable, if you know how to. For a module named test1.nim, it generates ~/.cache/nim/test1_d/@mtest1.nim.c, and for ../../path/to/test1.nim it generates @m..@..@path@[email protected], so it just replaces / with @, then prefix with @m.
Thanks for the writeup! I've ported Nim to two operating systems with pretty similar results. Though I can add a few notes.
While importing os brings in a lot, Nim will skip the parts of a module you don't use. You might still need to ensure the posix constants are defined or faked. In a few cases you might want to extend some when defined(...) blocks. Otherwise if you don't use a function it'll be ignored.
Overall it works better than you'd expect, though if a programmer tries to use a file system proc it may not give intuitive errors. The move to split up the os module is handy for that alone.
Also, at work I've had to learn a lot of cmake recently at work. I do think theres a few ways to make Nim play a bit nicer with cmake.
A simple route would be to include the nimcache/myproject.json as an input that cmake knows about. Then if the json list of C files changes, cmake builder backends can trigger a cmake reconfigure stage. I'm confident this will work for at least the Ninja outputs from cmake. Its actually the same technique that cmake projects like the esp-idf use to trigger a cmake reconfigure.
Ideally we could make a CMake module to parse the Nim project.json files and directly produce a list of source targets. It'd be nice to to trigger Nim compiles from CMake builds as well. Hopefully this week I can make some examples.
Did you look into our os.nim refactorings? It's now split up into different topics.
No, what are these and how do I obtain the information about this?
A simple route would be to include the nimcache/myproject.json as an input that cmake knows >about. Then if the json list of C files changes, cmake builder backends can trigger a cmake >reconfigure stage. I'm confident this will work for at least the Ninja outputs from cmake. Its actually >the same technique that cmake projects like the esp-idf use to trigger a cmake reconfigure.
Ideally we could make a CMake module to parse the Nim project.json files and directly produce a list >of source targets. It'd be nice to to trigger Nim compiles from CMake builds as well. Hopefully this >week I can make some examples.
I'm not an expert at cmake and perhaps what you describe is possible if you dwell deep enough within the cmake world. Normally cmake want to know about files when generating the makefile target. It cannot create rules for filenames it doesn't know.
However, you can trigger on nimcache/myproject.json if Nim updates this file after each build. It's a bit coarse though. I solved it by forcing the Nim build each time because cmake doesn't know about if I change any library files either.
Maybe Nimble is better as it can trigger on files change in the library as well. However, I couldn't get Nimble to output a static library, only executable.
However, you can trigger on nimcache/myproject.json if Nim updates this file after each build. It's a bit coarse though. I solved it by forcing the Nim build each time because cmake doesn't know about if I change any library files either.
Good point! I created a PR to help with this. Initially I modified the json build cache file logic to only write the file on changes. However, you'd still need to do post processing of the json file, which isn't great in CMake.
Instead I went with generating a list of newline separate C source files that works nicely with CMake: https://github.com/nim-lang/Nim/pull/20950
I'm not an expert at cmake and perhaps what you describe is possible if you dwell deep enough within the cmake world. Normally cmake want to know about files when generating the makefile target. It cannot create rules for filenames it doesn't know.
I got a working example with the esp-idf library and the above PR!
Here's the setup:
set(CDEPS "${CMAKE_CURRENT_LIST_DIR}/.nimcache/main.cdeps")
set(NIMBASE "${CMAKE_CURRENT_LIST_DIR}/.nimcache/nimbase.h")
set_directory_properties(PROPERTIES CMAKE_CONFIGURE_DEPENDS ${CDEPS})
file(STRINGS ${CDEPS} CFILES ENCODING UTF-8)
idf_component_register(SRCS "${CFILES}"
INCLUDE_DIRS ""
REQUIRES lwip newlib nvs_flash mdns pthread app_update i2cdev)
Running it with nim c --compileOnly ... && cd build/ && ninja only rebuilds the C files which changed, and will call CMake configure stage if the input .cdeps changes.
I got a working example with the esp-idf library and the above PR!
Here's the main part of the CMakeLists.txt:
(CDEPS "${CMAKE_CURRENT_LIST_DIR}/.nimcache/main.cdeps") >set(NIMBASE "${CMAKE_CURRENT_LIST_DIR}/.nimcache/nimbase.h") >set_directory_properties(PROPERTIES CMAKE_CONFIGURE_DEPENDS ${CDEPS}) >file(STRINGS ${CDEPS} CFILES ENCODING UTF-8) >idf_component_register(SRCS "${CFILES}" > INCLUDE_DIRS "" > REQUIRES lwip newlib nvs_flash mdns pthread app_update i2cdev) >Running it with nim c --compileOnly ... && cd build/ && ninja only rebuilds the C files which >changed, and will call CMake configure stage if the input .cdeps changes.
How does this work when you first generate cmake? When you have a completely new build there is nothing in the nimcache directory and cmake sees nothing, no c files or dependency files.
Otherwise, making nim output dependency files is a nice addition.
Hacking on the compiler to enable this seems extreme to me.
However, you'd still need to do post processing of the json file, which isn't great in CMake.
#!/usr/bin/bash
if [ $# -ne 1 ]
then
echo "usage: $0 /path/to/main.json"
exit 1
fi
jfile=$1
changed_files=$(jq -r '.compile[] | .[0]' "$jfile")
echo $changed_files
add a config.nims that exports nimCacheDir() & "/" & projectName() * ".json" somewhere cmake can read it pass that into cmake with execute_process or so
Hacking on the compiler to enable this seems extreme to me.
Nah, overall it's the cleanest solution IMHO. Also the compiler PR is pretty trivial and self-contained, however it helps simplify the CMake side a great deal. CMake is a de facto standard for C/C++ projects so making Nim play nice with it just makes Nim more attractive.
add a config.nims that exports nimCacheDir() & "/" & projectName() & ".json" somewhere cmake can read it
That doesn't work (well) for the reason @RodSteward first pointed out. Every time you run nim c it'll update the json which triggers full CMake configure.
The key to optimizing this in modern CMake is the set_directory_properties(PROPERTIES CMAKE_CONFIGURE_DEPENDS ${CDEPS}) line. CMake sets up the target build system to only call CMake configure if that file's timestamp changes.
With this setup running nim c ... && ninja now takes me ~2 seconds, whereas before it'd take ~20seconds.
How does this work when you first generate cmake? When you have a completely new build there is nothing in the nimcache directory and cmake sees nothing, no c files or dependency files.
Yah that's a bit of a sticker. :) Currently I have a Nimble task that runs nim c and also runs cmake if it hasn't been run the first time (e.g. no build/ folder) and finally it always runs cd build && ninja.
Alternatively you could run an execute_process("nim c") in CMake configure stage to bootstrap the setup. Then a mkdir build && cd build && cmake .." followed by just `make or ninja would be all that's required for pure CMake builds.
Using the JSON parser of CMake you can do this:
# Read the nimcache JSON file to get the source files
set(NimSources "")
file(READ "${NIMCACHE_JSON_FILE}" NIMCACHE_JSON_DATA)
string(JSON cfilelength LENGTH "${NIMCACHE_JSON_DATA}" compile)
math(EXPR cfilelength "${cfilelength} - 1")
foreach(IDX RANGE ${cfilelength})
string(JSON CUR_FILE GET "${NIMCACHE_JSON_DATA}" compile ${IDX} 0)
list(APPEND NimSources ${CUR_FILE})
endforeach()
# Suppress gcc warnings for nim-generated files
set_source_files_properties(${NimSources} PROPERTIES COMPILE_OPTIONS "-w")
add_executable(${OUTPUT_NAME} ${NimSources})
See this CMakeLists.txt for a real-world example (Pico SDK in Nim). In my case, during the initial CMake configure it doesn't run because Nim code hasn't been compiled yet (see if(EXISTS imports.cmake), which is generated using a nimble hook)
And if you want to make it work before Nim has generated the C files, use a dummy source file, such as nimbase.h:
# Get the Nim include path to get nimbase.h
execute_process(
COMMAND nim "--verbosity:0" "--eval:import std/os; echo getCurrentCompilerExe().parentDir.parentDir / \"lib\""
OUTPUT_VARIABLE NIM_INCLUDE_PATH
OUTPUT_STRIP_TRAILING_WHITESPACE
)
if(NOT NimSources)
# Nim project hasn't been built yet, so we need some source file... nimbase will do!
# It won't actually get compiled, it's just during initial configure step
set(NimSources ${NIM_INCLUDE_PATH}/nimbase.h)
endif()
...
add_executable(${OUTPUT_NAME} ${NimSources})
...
target_include_directories(${OUTPUT_NAME} PUBLIC ${NIM_INCLUDE_PATH})
Is there any difference between running Nim or nimble when you are executing it from CMake when it comes to dependencies? CMake has not clue about the dependencies of a Nim build. Nimble can probably handle it but can Nim alone detect any changes in any files when you only compile to C/C++?
When Nim compiles to C/C++ files only without any compile to object files, can you then omit all C include files (with --cincludes) and wait with those for the C/C++ compilation process and just use CMake include_directories(....) just as you would with any other C/C++ build? That would make things easier. Nim doesn't seem to open any C include file but passes the option on to the C/C++ compiler.
Nim doesn't seem to open any C include file but passes the option on to the C/C++ compiler.
I can confirm this to be true.
I made a slightly modified version based on the CMake file suggested by @djazz.
set(OUTPUT_NAME TheExecutableName)
set(NIMCACHE_DIR "${CMAKE_CURRENT_BINARY_DIR}/NimCache")
set(NIMCACHE_JSON_FILE "${NIMCACHE_DIR}/${OUTPUT_NAME}.json")
if(NOT EXISTS "${NIMCACHE_JSON_FILE}")
execute_process(COMMAND nimble build --compileOnly:on --nimcache:${NIMCACHE_DIR} WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}")
endif()
if(EXISTS ${NIMCACHE_JSON_FILE})
# Read the nimcache JSON file to get the source files
set(NativePathNimSources "")
file(READ "${NIMCACHE_JSON_FILE}" NIMCACHE_JSON_DATA)
if(NIMCACHE_JSON_DATA)
string(JSON cfilelength LENGTH "${NIMCACHE_JSON_DATA}" compile)
math(EXPR cfilelength "${cfilelength} - 1")
foreach(IDX RANGE ${cfilelength})
string(JSON CUR_FILE GET "${NIMCACHE_JSON_DATA}" compile ${IDX} 0)
list(APPEND NativePathNimSources ${CUR_FILE})
endforeach()
endif()
endif()
cmake_path(CONVERT "${NativePathNimSources}" TO_CMAKE_PATH_LIST NimSources)
add_custom_target(nimcompile ALL COMMAND nimble build --compileOnly:on --nimcache:${NIMCACHE_DIR} WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}")
include_directories(pathToNimDirectory/lib)
include_directories(...)
add_executable(${OUTPUT} ${NimSources})
What is changed here that I use execute_process if the desired json file in the nim cache directory is not found. Then it will generate the files during the cmake configuration phase.
Then I use the JSON parsing example that @djazz provided. Only change here that a conversion to native CMake paths was added. This is required if you run on Windows.
Then add_custom_target command will run every time you run your build process. It is the 'ALL' parameter that ensures this.
After that it is like a normal C/C++ build and it triggers correctly when a file was changed. You can use include_directories just like normal.
In this case I used nimble and it seems to work. I'm not sure what is the best here nimble or just the nim compiler. With Nimble you can keep things separate from CMake though with its own configuration per nimble project.
Here I had to manually add the path to nim /lib include directory in order to obtain nimbase.h. There were path conversion problems again that I haven't bothered with yet.
What I have NOT tested is what happens if you add or remove files in the nim build. I suspect it will not detect this and fail.