When creating a software project, it is important that it is reproducible. This means that the dependencies for that project are not, for instance, globally installed on the developer machine, but there is an automated way to retrieve them and make it available.
Many language-specific package managers (e.g. Maven, SBT, Leiningen, NPM, Cargo) handle dependencies through a configuration file. This allow, for instance, to clone a project on a fresh machine and (usually) be able to build it locally with a single command. For languages where such package managers are not widespread (e.g. C), many people resort to solutions such as Docker to create an isolated, reproducible environment to build the final artifact.
Nimble is able to handle dependencies as well: when starting a new project one needs only to create a .nimble file and declare dependencies there, and then run nimble c ... - or a custom nimble task - to build the project having the dependencies in scope.
For some reason, though, I often see libraries that recommend to do nimble install to make them available. For instance, out of the latest 10 libraries published on nimble (one of which is my own), the following 4 suggest a global nimble install:
(not to pick on those in particular).
The problem is that this will install libraries globally. Suggesting newcomers to use nimble install for anything but executables (such as c2nim) will lead to an ecosystem where projects are built with unknown dependencies at random versions.
My suggestion is:
I think this is important, because nowadays being able to handle dependencies in a package manager is more or less given for granted by people coming from other languages, and while technically Nim is able to do that, there is still mixed advice in the community.
See this issue for more discussion of the topic
I think one thing that would hlp with your first point would be to add a new option to nimble, such as nimble install --save tempdir (using one of my packages for example). This is stolen from NPM (node's package manager) which uses both --save and --save-dev to mean that the package argument is added to the package.json file (or <xyz>.nimble in Nimble's case).
In addition to what you've mentioned, Nimble also really could do with having the ability to vendor packages within a package/application rather than always relying on a global package store. Go have struggled with taking the approach of installing places globally and they still haven't fully fixed it (though they recently added stuff to their package manager to vendor stuff, in the meantime several third party alternatives such as govendor and gom have sprung up). Without vendoring, I can very easily release a v1.0.0 tag which you might then install - there's nothing stopping me from then later moving that tag to an entirely different commit and force pushing it to the repository. With vendoring, you can always check in the exact package versions you're using to build your application.
Lock files would also be great, and I know they're on @dom96's roadmap of things to get around to.
@andrea
For some reason, though, I often see libraries that recommend to do nimble install to make them available.
I believe the reason this recommendation is so widespread is that developers very often prefer to test out packages before adopting them into their project. Often times they don't even have a project. Creating a .nimble file just to play around with a package would be tedious.
Suggesting newcomers to use nimble install for anything but executables (such as c2nim) will lead to an ecosystem where projects are built with unknown dependencies at random versions.
Have you seen packages that just assume dependencies will be installed globally? I haven't seen a single one, so I don't think this danger is real.
that people stop mentioning nimble install as a way to depend on libraries (it is fine for executables though)
Maybe I am mistaken but developers are not suggesting that nimble install be used "as a way to depend on libraries". They are just showing this command without any comments.
Looking at an npm package for example, I see a similar command shown in the right sidebar. Cargo packages on the other hand show a snippet that can be added into the cargo.toml file.
I do agree that library authors should be showing requires "pkg >= 0.1.0", but I'm not sure it's such a big deal if they don't.
that library authors git tag each new version of their libraries - this is needed for nimble to retrieve the library at that particular version
Agreed. This is a requirement and is described in Nimble's readme.
that library authors bump versions frequently enough that one can depend on numbered versions
Yes, but how often is often enough?
Anyway, thanks for starting the discussion. I'm happy to discuss anything and everything about Nimble even though I may disagree with you sometimes :)
@euant
In addition to what you've mentioned, Nimble also really could do with having the ability to vendor packages within a package/application rather than always relying on a global package store.
It seems that a lot of people really want vendoring. But to me it just seems like a poor man's lock file.
Without vendoring, I can very easily release a v1.0.0 tag which you might then install - there's nothing stopping me from then later moving that tag to an entirely different commit and force pushing it to the repository. With vendoring, you can always check in the exact package versions you're using to build your application.
So the idea is to also commit these vendor folders? I guess hard drive space is cheap these days, but still... at this point you can just as well copy and paste the dependencies into a vendor folder, add --path:vendor/ to your config.nims or nim.cfg file and call it a day. Is there even a need for a package manager?
Speaking of which, it's useful to track what dependencies (and exact versions) were used in a build and ship this information with the binary.
This can be used to automatically identify if a binary was built against libraries affected by known vulnerabilities or that use obsolete protocol/file formats and needs updating.
Often times they don't even have a project. Creating a .nimble file just to play around with a package would be tedious.
it isn't, as long as it's just a nimble init away. AFAIK stack and other modern build tools work this way and they're doing just fine; globally-installing packages is often regarded as a bad practice in other ecosystems as well.
this post about cabal v. stack gives a good explanation https://www.fpcomplete.com/blog/2015/06/why-is-stack-not-cabal
Looking at an npm package for example, I see a similar command shown in the right sidebar.
that's not good advice, though: https://www.smashingmagazine.com/2016/01/issue-with-global-node-npm-packages/
@dom96 The idea is that any release zip of source code would contain the vendor libraries. Most of the time you wouldn't check them in.
The NuGet package manager for .net does something similar (as does nom), in that packages for a project are local to the project, and stored in a "packages" folder. The NuGet package system relies on centralised package feeds though, rather than decentralised Git repositories.
A good lock format would solve the concerns that vendoring solve for definite though. Both approaches aim to solve the same thing in the end :)
I love vendoring. I will vote for vendoring. I think some thing like this would be cool:
It would be cool if nimble install X just installed stuff in the current folder's 'lib/pkgs/deps' directory and updated/created .nimble to reflect the installed dependancies and their versions.
Its magical because you just do the thing that you would do anyways but all of the deps and .nimble and stuff is just handled for you.
Its magical because .nimbile file would contain the exact version you installed. You can choose to just git ignore the 'lib/pkgs/deps' folder and commit your '.nimble' file.
It would be magical because your can edit, fix, or just add echos the stuff in 'lib/pkgs/deps' and it would not impact other projects.
You would use nimble install -g or --global to get them to install globally for the user, the thing it does now.
If you just run nimble install without params it would look for .nimble file and install everything for that and recreate the 'lib/pkgs/deps' folder.
I already run into nim's version of "dll hell issue" where one project I have uses one library version and another project I have uses the other (I think it was this one: https://github.com/ephja/nim-glfw) and it was really confusing.
Vendoring ftw!
It would be cool if nimble install X just installed stuff in the current folder's 'lib/pkgs/deps' directory and updated/created .nimble to reflect the installed dependancies and their versions
It would be much better if nimble could recognize two different dependency lib that depend on the same dependency instead of copying it in 'lib/pkgs/deps/dependA/lib/pkgs/deps/sharedMany' for dependency A and 'lib/pkgs/deps/dependB/lib/pkgs/deps/sharedMany' for dependecy B
Hard drive is sure cheap now (it's different for different countries and regions though), but duplication is still duplication and I cringe at that :(
If the package manager could know that sharedMany (example above) is depended by two different lib and installed at 'lib/pkgs/deps/sharedMany', instead of other respective dependA and dependB, I'll definitely go with local dependency ;)