Inviting review and feedback from the community on some Nimble changes that are being planned for the near future. The problem statement and changes are documented on the wiki so as to retain change history.
Link: Package structure and Interop
The goal:
Please share feedback right here since the wiki doesn't support commenting.
Thanks in advance.
Rather than "warning that exporting multiple packages is imprudent", the default should just be a similar reporting & resolution mechanism for name management of symbols/procs in modules. Specifically, if both pkgA/modA and pkgB/modA (or pkgC...) exist then the compiler tells you to qualify the import to pick the one you want (or warns you). The qualification just uses a slash (/) rather than a period (.).
Like symbols in modules, qualification should also always be allowed for those who want to be explicit about their dependencies. Just as you can always call strutils.find you can always say import pkgA/modA even if there is no ambiguity. This seems like the internally logically consistent name management way to go.
If you really want a nimble warning -- when no installed packages are known to have a conflicting top-level module name -- it could say "only module pkgA is protected from a need to package-qualify". (It is really only protected via the centralized package database not having duplicate package names, but people can always have some private code with --path causing a collision. So, it's not really ever "guaranteed".)
Please be aware that @bobeff's work on reforming Nimble to support lockfiles addresses some of the presented issues and it's already quite close to completion. Please take your time to study it by examining the test cases. @bobeff is also likely to start working on better documentation quite soon.
I'll try to briefly describe the main ideas in the workflow with lockfiles and then I'll comment on each point in the wiki:
The lock file is a json file that precisely specifies the version of each transitive dependency of your project. It includes information about git revisions and directory content checksums, so the dependencies can be obtained both through git cloning and through tarball downloading. The integrity of all downloaded files is verified through the checksums before the package is installed in the global Nimble cache to prevent package hijacking attacks.
Within the Nimble cache, it's possible to have many versions of the same package installed side-by-side. Within your project, Nimble generates a config file with the precise active paths at any moment. This file is not intended for committing in git, but it will be usually included from your project config file which is stored in git.
Thus, the typical workflow is to invoke Nimble only when something about your dependencies changed while normal builds and IDE plugins continue to use plain Nim, which results in the correct paths being sourced from the config files.
An import new feature is the localized develop mode. Within each project, you can use the nimble develop command to create a local nimble.develop file that includes path overrides for the dependencies that you are currently actively working on. This file is not intended for committing in git. It can "include" other develop files to facilitate a common directory layout that is well-suites for large teams. Here is an example demonstrating how my workspace at Status would look like:
status/
nim-libp2p/
...
nim-chronicles/
...
nim-eth
...
nimbus/
...
nimbus.nimble
nimbus.develop # includes ../status.develop
nim-beacon-chain
...
beacon_chain.nimble
beacon_chain.develop # includes ../status.develop
...
status.develop # puts all of the Status packages in develop mode
In other words, the presence of a local develop file tells Nimble to prefer the local directories for certain packages instead of referring to the global cache while generating the config file with the active paths.
The Nimble commands associated with the develop mode are designed in a way that makes them easy to integrate with git as pre-commit or post-merge hooks (i.e. executed after git pull). The overarching goal of the lockfile feature is to make it impossible for you to push a broken build to your teammates, so the nimble check command will verify that all of your dependencies are in a clean and published state, that your committed lockfile is up-to-date and so on. On your daily pull, another command will help you get your working copies in a state that matches the fetched lockfiles.
Let's see how these features address the pain points from the wiki:
@bobeff's work is relatively orthogonal to these, but it does preserve the conceptual separation between a repository and the installed package directory. This is considered an important feature if we want to support monorepos with multiple Nimble packages appearing within one. The src directory quirk is just handled behind the scenes by automatically specifying the correct added path for the package. Just like now, the nimble develop command can be used to create a complete development environment for the package in a local directory where you can run the tests, prepare a PR and so on.
I don't think there are any functionality changes related to this, but the dependency traversal algorithms have been largely rewritten to support parallel downloads and installation of packages. Any development based on the current Nimble source files is likely to conflict with Bobeff's branch, so it may be wise to directly try to implement the changes there.
Addressed through my description of the minimal config files integration.
Addressed through the precise paths specified in lockfiles and config files.
Responding to some points raised by @zahary.
Regarding package structure, I've added some info in the nim.cfg interop section detailing how multi-package repos can be handled with the same nim.cfg path mechanism. But the main motivation of these changes is to simply the lives of library writers and not require them to fumble around with installExt and related settings or worry about the repo looking different after install.
As for adding support for installing packages from tarballs or other download mechanisms, it can certainly be added but can be discussed separately. Package integrity checking should not be affected by package structure as you have noted.
Parallel downloads sound great and so do any improvements around dependency handling but neither has anything to do with package structure. It is best to break all these improvements into individual PRs which can be reviewed and merged into Nimble master which has over 2k changes since @bobeff's last sync.
Last, the nimble develop enhancements as proposed seem complicated especially from a user perspective. Today, you can develop multiple packages by just running nimble develop pkg1, pkg2, pkg3. By just adding a --recurse or similar flag, nimble could be instructed to setup all dependencies also in nimble develop mode in sibling directories.
Local deps mode might need a little more than that but adding a .nimble-link capability that extends to $nimbleDir itself might solve that puzzle. Introduction of a separate nimble.develop file along with 6 new CLI flags seems unwarranted. Maybe I'm missing something but the diff does not help with understanding the behavior intended. More importantly, I don't see the nimble develop behavior interfering with the package structure proposal.
It's not "pollution" IF there is any "Nimonic" package-qualify the module import ability, however that is done, any more than exported procs in a module "pollute" the global symbol namespace. I don't see anything incompatible about setting things up so qualification can work. It seems to me that whatever magic happens to have foo-version/ installed but just import foo work should be adaptable. Some version has to be selected, but that is already the case.
The incredibly common "measured" case in the current nimble DB is no need for qualification. It's debatable if it will stay that way, of course. Unknown future evolution should not hold hostage present day usability.
@cblake - this design is basically Araq's idea with all the consequences documented and mapped to the issues that will get resolved.
As for using git worktrees and space efficiency, as interesting as it sounds, it does not seem warranted since it will require quite significant changes to how Nimble functions and introduce additional git awareness into Nimble which will need maintenance. The value add is not significant compared to just having many independent clones. And for the rare large repos, the user can simply nimble develop a single directory.
@dom96 - I agree this proposal is not too different from what we have today as far as namespaces are concerned. Part of the effort here is to make it clear what nimble is pushing for while improving other packaging aspects in the identified issues:
Once again, as soon as we require import x / x rather than import x there is no "pollution" of module names. Nimble introduced the pollution, not Nim.
Right, but then this proposal should explicitly state that this is the goal and how we plan to achieve it. The rest of this proposal isn't as important (nor controversial).
Meanwhile, two more nimble issues will be addressed by the proposal: