Hi, coming from a background of mostly dynamic languages, when it comes to compiled ones I always struggle to translate some common patterns that heavily rely on dynamicity (e.g. user configs especially when "scriptable", plugin systems, etc. ).
I apologize if some of my doubts and intuitions may sound naive, but I'm looking forward to hear your opinions on the matter. I will list a couple use cases I often encounter, and would like to know how they are usually handled in idiomatic nim and/or what are some opinionated different approaches to those problems. I think nim may have some "outside of the box" type of solutions, especially given the flexible macro system, expressive and readable language, and possibly integration of nimscript in the process.
This is probably the easiest one. Any configuration language could be chosen and a parsing library can be used to read the file and use its data.
What I was thinking, though, is using nim itself as a config language. Using few macros is possible to get enough readability and flexibility that an additional language seems superfluous.
I am wondering if there is an easy and commonly used way to read a nim file at runtime and parse it using some predefined macros and functions mapping to predefined types. My intuition is that it should be feasible, but would be nice to be able to embed the nim parser (ideally a very stripped down version not intended for development) in a binary that can be distributed. Is this an explored approach at all?
If I can use nim syntax and my own dsl on top of it in a user config, why not going to the next level and accept user defined functions to build a solid plugin system?
Obviously this gets tricky in a compiled language, naively I can think of some possibilities, but not sure if they are feasible or not:
Those are just common approached I can think of but would love to know if there are more.
To make the context clearer I can share a very simple practical use cases:
for example the user could write a file like:
html:
div class="foo":
text "hello"
and then a distributable binary (no nim compiler needed locally), can read that file as an argument and output an HTML file.
Can we avoid a custom parser (and/or using the one from the nim compiler)?
as next step, can we use any nim function there and execute it at runtime (e.g. to iterate lists, call and/or define utility functions, etc.)?
The same could work for a custom CSS language e.g.
class "foo":
decl "width": "100%"
@media "screen and (min-width: 768px)":
decl "width": "50%"
In this case it would be interesting how to let the user define plugins (like in postcss)
how would you do it idiomatically in nim?
Is it mandatory to host some VM?
Or can something like a "DLL" be generated on first use?
Or is it in the end fast enough to give up the idea of distributing a binary and just have a local nim project and recompile when needed?
Is there any advantage on using nimscript instead of any other hosted language?
Thanks.
I wrote a three part article series and held a presentation on using Nim as a configuration language, should be of interest to you: https://peterme.net/using-nimscript-as-a-configuration-language-embedding-nimscript-pt-1.html. When embedding NimScript within your program like this you're actually importing parts of the Nim compiler into your project. So there's no need to ship the Nim compiler separately, it's just a part of your program. Of course NimScript in this way doesn't have the ability to compile and create binaries, but for running configs and such NimScript should do just fine.
Since I wrote that ElegantBeef has created the great nimscripter which makes the whole process much easier, and continues to push the boundaries for what can be done easily with NimScript.
NimScript is said to run about as fast as Python, but I haven't verified that claim. Of course you can also very easily define logic in your Nim binary and simply expose the function to NimScript to get native performance for that part.
The way I would build a config language, or a full blown HTML/CSS thingy in NimScript would be to create a library of macros and such for the DSL, then auto-import that into the NimScript module the user provides. Then they would run the DSLs in NimScript and create the output they want, returning it simply as a string to the main binary.
Another approach to the HTML thing would be to recompile things into dynamic libraries as you said. I've explored that possibility in a simple experiment. It worked fairly well, but of course this requires the user to have Nim and a C compiler installed and properly configured (and matched with the versions expected). In the end it all comes down to whether you need the extra speed or not.
Sounds like an interesting project you have in mind, would love to see how it works out.
I made https://github.com/elcritch/ants to make a YAML Nim config.
It may need updated to work around an upstream change.
I got something basic working using nimscripter.
One part I am maybe not fully grasping is how to embed a small stdlib.
@PMunch in your article you mention using nimsuggest to get a list of needed files, but ultimately those files still need to be shipped together with the binary executable and that doesn't seem ideal. Is it possible to manually select some stuff from stdlib plus some custom defined functions and macros and embed them directly in the executable?
I found this issue: https://github.com/beef331/nimscripter/issues/19 that also does not provide much more clarity: what does this comment (https://github.com/beef331/nimscripter/issues/19#issuecomment-1249784452) means? Even if I can find a small subset of the library that I need, I would still need to have it in a local folder where the executable runs right?
@elcritch interesting project, I also saw you seem to have the same issue? You are using a shell call to nim dump to get the library path, does that mean that the cli would not work in a system where nim is not installed?
Well there's no getting away from having to ship some kind of library for NimScript. Whether this library comes shipped with your program, or gets automatically extracted from the binary on runtime however is up to you. ElegantBeefs comment there means that it is possible to build a system in nimscripter which does this automatically, but that it doesn't exist yet. The system would be something like reading in the standard library files with staticRead and then storing them in the binary. On runtime it would write them out into a temporary location and point the scripts import path to that folder. Another way is of course to pre-bake some standard library which you simply string append to the input script before executing it. This would technically make it more like an include than an import, but for most things it would serve the same purpose.
To summarise you do need to somehow ship a library of the features you need. This can either be installed with an installer, included in a simple zip file, or any other means of distribution; or it could be loaded into your binary during compilation and then placed somewhere on runtime. Pros and cons with both approaches.
makes sense, I'm not there yet so I am just forethinking about future roadblocks. I can imagine a typical use case would be to distribute the binary in npm and then being able to resolve the /lib from node_modules correctly, also distributing via other package managers e.g. for linux distributions, homebrew, or whatever may present similar challenges. I think that is where the "appeal" of single executables is, but I'm also not a fan of very fat binaries, they are fundamentally a hack to solve the issue of inconsistent distribution methods.
Are there any best practices for those scenarios?
Is common practice to have a "lib" folder alongside with "src"? And does it get preserved when binary is distributed via nimble? Are there utility functions to get the library path, and what is their order of precedence? Will "lib" in the same folder always win?
There's no need for "compilation" or "executable" or "dll" for scripting language. The "executable" could be an oneline bash file with nim -r app.nim. And the "dll" is just a folder /app/plugins/some_plugin.nim.
Sadly, Nim doesn't support well such use case.
would nim r -d:release src/app.nim on a second run without changes be as fast as nimble build -d:release and then ./app ? Or is there any additional overhead with this approach?
This seem to be the easiest approach for projects where relying on the availability of nim compiler is not an issue, and probably great for initial iterations and experiments.
when you ship it you'd select the libraries you want in your environment by default and ship them in a folder that installs at like ~/.config/yourProgram/stdlib
I always disliked that unix madness, splitting program into arbitrary /etc, /etc/bin, /usr/local/bin, ~/.config/whatewer and similar nonsense.
As for NimScript, in its current state it has too many problems, currently it's much easier to use nim r script.nim
That's not nimscript and defeats the purpose so... good job. If you need the Nim compiler on the machine, you do not have a portable program.
I always disliked that unix madness
Ok, sorry install it to ProgramFiles, ProgramFiles (X86), ProgramData, or %AppData%, .... saying it's wholly a unix issue is silly when no OS has a always used place to install dependent files :D
If you need the Nim compiler on the machine, you do not have a portable program.
Hmm, maybe I miss something here. How having "nim compiler" is different from having "nimscript vm"? In both cases you need to have some binary, why binary containing "nimscript vm" is portable and binary containing "nim compiler" is not portable?
Using nim r bleh.nim uses Nim directly which requires a C compiler
Thanks, I forgot that Nim uses C compiler underneath.
Not to mention using nim r bleh.nim requires you to setup inter-process communication, you cannot just call your host procedures in an environment.
No. You have file structure
my-app/
plugins/some-plugin.nim
run-template.nim
run.sh
And the run.sh does the following
$run = read content of 'run-template.nim'
for every plugin-file-name in plugins directory:
append 'import ./plugins/$plugin-file-name' to $run
compare $run to content of `my-app/run.nim` and:
if it doesn't exist create it
if it's different overwrite it
nim r my-app/run.nim