I have been using Nim and Nimble for a few months now, and in general interacting with the tooling has been a very positive experience. More recently, though, I have started seeing some limitations of Nimble, and I started to wonder whether it would be worth to have a radical evolution.
Everything I say here is just a suggestion: ultimately Nimble is a project of dom96 and of course he has the last word. Still, since it is such a fundamental building block of the Nim ecosystem, I think that it is a good idea if everyone gives ideas about its evolution.
The limitations that I have found include:
It also worth mentioning a previous discussion that raised several very interesting points.
My typical workflow when I start a new project is to define its name and dependencies in a file, and then start working, using some tool to build, run and test my project along the way. It is crucial for me that dependencies for a project are isolated from dependencies for a different one. This means that whatever build tool I use has to understand the concept of dependencies, so in practice building and handling dependencies are often conflated (as in, say, sbt)
Nimble supports this workflow partially, but there are several things that could be improved:
I understand that other people have different opinions and needs. This is why I think that best things would be to evolve Nimble into something extensible. People would be able to write plugins in Nim for many common tasks, and final users could use these plugins declaratively. Plugins have been used in sbt-land for many different tasks: signing artifacts, publishing to sonatype, including ANTRL grammars, compiling to JS, formatting code, and many more.
A plugin architecture for Nimble may allow, for instance:
These are only a few ideas, but I am sure that many more would come to mind to other people.
Of course, it would be a lot of work, but if Nimble gets the plugin architecture right, many tasks can be delegated to plugin authors.
Managing more than one artifact would also mean changing the Nimble format. I think this would be a good idea anyway. An alternative format which is declarative but is also valid Nim has been discussed here and it seems a good idea.
I am aware that there are already libraries using the current nimble format - 212 right now. I think that migrating those libraries to a new format would be a minor task. Active library authors could do this themselves. For other projects, it should not be such a big work to translate the current syntax to the new one, especially with some tooling, and then we could make PRs to the original projects. There would remain a few projects whose authors do not bother to incorporate the PR. I think this would be mainly about abandoned experiments, and we could just link to a fork with the updated nimble file.
I think that it is important to evolve Nimble sooner rather than later, as the number of existing projects will certainly grow, and what would be a little work with 200 projects may become difficult with 2000 and impossible with 20000.
I hope to spark some discussion about Nimble. The previous discussion seemed very promising, but it is no more active. It would be interesting to hear about all common workflows, so to be sure that every use case is covered.
In fact, when you look at my lexim project that already has the problem that it uses an external helper nim program to speed up compilation and yet lexim.cfg doesn't support staticExec and neither does Nimble so that I have to document the building steps or instead I could use scons, make, cmake or could resort to nonportable bash or batch scripting ... in other words resort to tools that I like to erase from my memory.
That would be surely make it easy to extend! :-)
The only disadvantage I see wrt to a plugin system, is that it would be difficult to add new tasks. I can imagine a plugin that enforces a directory structure, providing tasks such as nimble test or nimble benchmark, or nim generate-dependency-graph.
I am not sure how anything written in Nim could provide a command like nim test. Maybe there could be a generic nim do such that people can attach handlers for various actions? Like nim do test, where in the configuration script one has
import nimtasks
registerTask("test", someProc)Just thinking out loud...
I am not sure how anything written in Nim could provide a command like nim test
Easy: nim -d:test foo. foo.nimcfg can then test for when defined(test). Clearly not optimal, but it's an emergent feature of the whole system.
Having said that, I don't believe in these standardization efforts, especially in "enforcing a directory structure". That's just more to learn for users: People already know benchmarks are likely under "benchmarks" or perhaps "bench" or "tests/benchmarks" and now in addition to that people need to learn that every name except "benchmarks" (always with the trailing 's' even if there is only one benchmark?) is verboten in Nim's ecosystem...
@Araq:
Nim, not Nimble, will be Nim's build tool. Let's face it, nim c -r foo won't go away and nim already supports staticExec (now even with caching!), so people can throw away the barbaric autotools.
I wanted that from the beginning. Nice to see you changing your mind. This would be a very natural change and would work well, and do what the user expects. Of course, we should think about it carefully, i.e. it should still work even if Nimble isn't installed, but give the user meaningful warnings or errors in those cases.
I hope you realise what this means though. Here is how I think it should work:
nim c foo.nim:
- Check for .nimble file in the current dir (parent directory too I suppose).
- A .nimble file is found:
- Execute nimble c foo.nim
- Nimble checks to see that all dependencies are satisfied, if they are not it attempts to install them.
- Nimble executes nim c --path:/path/to/dependencies foo.nim.
- A .nimble file is not found:
- Proceed as normal.
Or something to that effect.
Nim should call Nimble only for dependency management as this is Nimble's primary reason of existance anyway. So nim install foo will delegate to nimble install foo, for example.
I don't think that's a good idea. Let's only do explicit "dependency management" through Nimble.
@andrea thank you for taking the time to write this proposal! Let me try to tackle some of your propositions:
I'm uneasy about the idea of changing the .nimble format, I don't see what is wrong with it. Evaluating the .nimble file using the Nim VM seems like overkill, but then again most package managers to have a Setup.ext file (Setup.py, Setup.hs, and likely more).
Plugins are a good idea. But we need to seriously make sure that we design plugin support well in Nimble.
The limitations that you've mentioned are already probably in the Nimble issue tracker. Please take a look and create a new issue if it turns out that they don't exist in the issue tracker.
Andrea, your proposed registerTask() code seems to be a lot like nake, have you seen what it does? I've had no problem using nake to run tests for projects like lazy_rest. I just type nake test which goes through a test subdirectory compiling and running tests. I can also run nake vagrant which will start a Linux virtual machine with the purpose of either testing or building binaries.
I'm curious because I've had no problems doing what you seem to have trouble with, and I can't see a reason to overload nim or nimble even more. But if you already knew about nake excuse the interruption.
dom96: Nimble checks to see that all dependencies are satisfied, if they are not it attempts to install them.
No. No. No. Simply no.
Running a compiler should never implicitly download and execute (staticExec!) arbitrary code from the internet. If you want to use a package manager to do this, this should be an explicit action.
Jehan: No. No. No. Simply no.
So you're saying that nimble c shouldn't automatically install dependencies either?
@gradha: I have heard of Nake, although I did not actually try it. The problem that I see with Nake is that it does not understand dependencies, so I cannot write a Nake task to actually build my project - the only option is calling nimble build as a string, in which case I run into the same limitations of Nimble itself, in particular that it does not understand how to manage multiple artifacts (say a library vs its tests).
In fact, the very library that you link shows two of the issues I am worried about. First, the .nimble file mentions Nake itself as a dependency. This means that if lazy_rest was a library, people using that library would end with Nake as a dependency, while it only serves to build the library itself.
Second, I tried to make a clean download of lazy_rest and run nake test: it failed with Error: cannot open 'bb_nake'. Now, I see that bb_nake is a dependency in the nimble file, but Nake itself does not understand dependencies, and leads to this error. I guess that it would be enough to run nimble install to get the dependencies, but this is exactly what I am trying to avoid: having global dependencies instead of managing dependencies per project.
I guess that what I would like is a tool that offers both the possibilities of Nake and Nimble together. For other languages we have Maven, Sbt, Leiningen, Npm, Bundler, Cargo and I miss a similar tool for Nim. Nimble is almost there, but as soon as you run into a slightly more complex situation, it shows its limitations.
@Jehan: I understand your point. I do not really care that it is the compiler itself doing this - in fact my first proposal was a plugin system for Nimble
@dom96: I can certainly open issues, but there are some limitations that seem to be workable only by rethinking the .nimble file itself. Namely:
dom96: While this is an issue, and I recognise it, we simply don't have the resources to solve it right now. There are bigger fish to fry. I'm sorry, but I think we need to prioritise, signing packages is not high on my list.
I completely understand that. I wasn't demanding anything (plus, it would be hypocritical, given that I've been short-changed on time myself). Note also that there are a fair amount of trade-offs (for example, cryptographic signatures can be a barrier for entry and don't by themselves solve anything, since anybody can create one).
My intent was to lay out some thoughts about the long-term perspectives and to enumerate the various and sometimes conflicting concerns involved. I think for the time being it is perfectly fine if "nimble build", "nimble install", and "nimble c" simply remain optional parts of the Nim ecosystem (hence, why I was so emphatic about nimble not getting hardwired into the compilation process). Personally, I use nimble mostly as a repository list ("nimble search", "nimble list"), but build and install packages using different tools (which is easy, because packages are git/hg repositories).
I haven't seen this raised in the discussions of nimble that I've seen, so I thought I'd raise it here to see if there is any agreement about whether it's needed.
I'm a long-time user of package management systems, although in the OS (Linux) context - apt for Debian/Ubuntu, yum/rpm for RedHat, pacman for Arch, etc. They all have a number of capabilities that I sorely miss when I use nimble:
Most of the above depends on on one fundamental but missing capability: the ability to query the transitive dependencies of a package without installing it (and its transitive dependencies). Unfortunately, implementing this capability requires changing the way that nimble package metadata is currently stored.
Dependency information for a package is currently stored as a file in the package's repository. Github is (almost) the exclusive repository where Nim packages are stored, and it does not allow download of individual files. This means that, as it is currently implemented, the only way for nimble to learn all the dependencies of any package is to install the package as well as all of its transitive dependencies. This effectively rules out the ability to learn a package's dependencies with a simple query.
The only way I can see to break this impasse is to store the package dependency information in the same place as the other package metadata - in nim-lang/packages. There are many ways this can be implemented, of course, but that is for another discussion.
If this is done there would be a couple of impacts:
The benefit would be a package management system that is much easier to use effectively for everyone.
If there is agreement that nimble should go in this direction, I would be happy to contribute what spare cycles I have to the effort.
There are actually some tricks to checkout a single file from Github. Unfortunately, Github doesn't allow "git archive", but since it supports SVN you can do for instance:
# this checks out the todo.txt from the Nim repo
svn export https://github.com/nim-lang/Nim/trunk/todo.txt todo.txt
Considering that I've rarely seen a module with a huge repository size, I think that even a clone + sparse checkout won't hurt that much.
I'm not sure if my concerns are valid, but somehow I don't like the idea of putting all meta data into nim-lang/packages. It feels like important information is missing in the users repository. The project themselves are no longer self-contained. Each minor change to the meta data produces a significant overhead (PR + merge), which always means that there will be delays. Example: I want to make a minor change to the dependencies of one of my modules, and want to use the modified module immediately in another project. Do I have to wait for the PR to get merged now? It also feels weird that changing a dependency does not necessarily produce a modification in the module repository itself -- the corresponding commit message "updated module X" would go to nim-lang/packages instead, and I would have to make an empty commit to make it clear.
I actually would prefer it to be the other way around: All meta-data in the repository, and nim-lang/packages is just a cache of the meta data, generated fully automatically. Yes, resolving transitive dependencies is a bit more complicated, but it is still just a few git clones. And if the generation of nim-lang/packages is fully automatic, it is even possible to make the lookup "on publish" instead of "on query" and include them in the cache.
I think moving to NimScript from .nimble is a huge challenge but may make sense since it is the most flexible.
But that means any tool has to run the script or script sections right? Which it seems tough to keep things declarative (but possible).
Seems a huge challenge to migrate existing packages to a NimScript system. Practically speaking, how would one convert the existing key nimble sections like version and dependencies into NimScript, and how would a tool 'read out' these values without running the whole script?
Just to throw this out there again, within the next few years you are going to see a massive shift from server-oriented networking to content/data-oriented networking. Here are a few reasons: the network has many leaves and so to scale you get that branching many leaved shape.
Take a look at the architecture for npm. It is using a big CDN/cloud infrastructure being run by a few companies. But the architecture is essentially content/data oriented amd distributed. This is necessary because of the scaling requirements.
Or take another example of where data-oriented networking is necessary: China, and soon to be other countries with increasingly authoritarian governments (UK/US) enforcing communications controls to stifle dissent. In these cases encryption and sideways/mesh-based networks are becoming more and more important and prevalent.
So that is a long way of repeating myself from before and adding another big potential challenge which is rather than relying on some central website or service, especially when that is one that a small group controls or maintains, when moving away from a massively funded and resourced technopoly like github, the next frontier is a fully distributed content-oriented peer-based package repository. Could almost be as simple as bittorrent or maybe IPFS/Swarm or something along those lines. That solves the problem of who controls/maintains the package registry and scales it. It sort of like choosing bitcoind instead of banks.
Actually I have the code ready since quite some time, but it needs testing and polishing before I can create a PR out of it.
But that means any tool has to run the script or script sections right? Which it seems tough to keep things declarative (but possible).
That is easily solved: Just run nimble dump foo.nims which produces something easily parsable, maybe even the old nimble syntax. That means you need to have Nimble installed to process nims files but that is not much of a requirement, is it?
So that is a long way of repeating myself from before and adding another big potential challenge which is rather than relying on some central website or service, especially when that is one that a small group controls or maintains, when moving away from a massively funded and resourced technopoly like github, the next frontier is a fully distributed content-oriented peer-based package repository.
I think we are in really good shape here as we don't plan to move away from the Github based solution but instead will support multiple sources for packages.json files.