Mirror of forum.nim-lang.org

2209 :: Nimble enhancements - discussion

• euant (orginal) [2016-04-14T10:20:32+02:00] view original

  Hi,

  I'm pretty new to Nim and just started learning how to use it. I've noticed some small things mostly revolving around the Nimble package manager that I would like to discuss with the community. I've talked with @dom96 via Twitter already about some of my ideas and he suggested creating a topic here or an issue on GitHub. I went with the former for now, and any ideas that come out of this thread can move across to GitHub later.

  Nimble is a good start to a package manager and works well for now, but there are a few things that we could adopt from other languages:

  • Vendoring dependencies into projects: Currently Nimble installs all dependencies into a per-user store. This is an approach similar to old(er) versions of Go that has caused them major headaches and that they're currently (at the time of writing - Go 1.5 started the process) solving. Basically, all dependencies should be installed to a vendor (or similar) folder within the project root. This allows discreet package versions per project rather than every project having to use the single installed version in the per-user cache (which causes problems if one project uses v1, and another uses v2 which breaks backwards compatibility).
  • Introducing a dependency lock file: Currently you include just a myproject.nimble file describing your dependencies and versions that are then resolved at compile time. This means that if you used a version mask for something like 1.1.*, one suer might get 1.1.1 and another might get 1.1.2, which again could cause compatibility issues. If Nimble generated a lock file to nail down specific dependencies (combined with suggestion #1), this would make builds much more reproducible. For more details, see this blog post about Composer's (PHP package manager) lock file: https://blog.engineyard.com/2014/composer-its-all-about-the-lock-file
  • Use vendor prefixes for packages: This is somewhat based upon the left-pad debacle that hit npm recently. Currently packages in Nimble are organised just by their name, which has to be unique. For instance, you can only have one json module, all others have to be named something else. If packages were prefixed with their vendors (excluding core packages) (and again much like Composer), you could easily create a new package with an existing name but with a different vendor. For instance, my pledge package would become something like euantorano/pledge (the separator is completely up for debate here!).
  • A centralized repository of packages: Packages are all currently referenced via a JSON file hosted upon GitHub. GitHub often has outages (which Nimble gets around by keeping a cached copy of the list locally). It would be a good idea to have a central repository of packages that provides a user interface to browsing and adding packages without having to use the nimble publish command every time. There is already a site that provides a very rudimentary browser for Nimble packages (http://nimble-explorer.com), but it could be enhanced and ideally should use HTTPS and provide things like checksums and the possibility for signing for packages. I've already started thinking about building my own implementation of something like this, but it would need official support to make it work properly. This would be similar to other package managers such as Ruby Gems (Ruby), NPM (node.js), Packagist for Composer (PHP), Crates.io (Rust), etc.

  I'm sure there are other enhancements and ideas too that would benefit Nimble. Having a decent, reliable, reproducible package manager and build system is key for success. I'd like to hear some feedback on the ideas presented here and some input.

  I'm still starting out with Nim, but would like to get involved wherever I can, especially with anything web based which is where most of my skills lie.

---

• dom96 (orginal) [2016-04-14T15:55:36+02:00] view original

  Hello @euant, and thank you for taking the time to write this. Here are my thoughts:

  • I'm not sure what the situation in Go is, but I don't think that vendoring dependencies into projects is necessary for Nimble. With Nimble you can easily have package Foo 0.1 and 0.2 installed and used at the same time by different packages (as long as you use Nimble to compile your program). I don't understand why this was a problem for Go.
  • Dependency lock files are planned and somebody has recently shared their desire to implement them for Nimble, here is the relevant issue: https://github.com/nim-lang/nimble/issues/127
  • I support the idea of vendoring.
  • I also support the idea of a centralised Nimble website. Here is a relevant issue: https://github.com/nim-lang/nimble/issues/219. As I already mentioned to @euant on Twitter, a centralised repository will give many advantages with one of the most important ones being a guarantee that packages will continue to be available (we should only allow removal of packages under strict circumstances, npm learned this the hard way). But in addition to the centralised repository, we should encourage the use of other package repositories as well as standalone packages on Github/Bitbucket/etc as we do now.

---

• euant (orginal) [2016-04-14T16:43:21+02:00] view original

  @dom96:

  Regarding the vendoring, Go used to have "one true version" for every package, which would be under ~/go/src/github.com/euantorano/pledge (or whatever). This was the only version you had. I assumed Nimble operated similarly, but hadn't taken the time to check - my bad! It still may be useful to have the option to vendor packages (perhaps a flag to nimble build or whatever like --vendorDir) so that you can send a full zip file including all dependencies to people that can be built easily without relying upon the package database/GitHub/the internet at all.

---

• wulfklaue (orginal) [2016-04-15T10:51:32+02:00] view original

  @dom96: The vendoring was a major issue with Go. Go has no support for version packages. It means that the package a person wrote last year and this year are from a Go point of view, the same. People add there own system of version management but its not a centralized or system in the code Go language ( mostly manual work ).

  Lets say last year i wrote a program. This depends on a external library dated 2015.10.15. This year i am writing a new program and i want to use the new functionality of the same external library dated 2016.04.10. I fetch this and my new program works great. But i like to update my other program ( bug fixing, whatever ). Now i run into a problem, because the external library has been overridden with a newer version and my old program trows up a bunch of errors because of functionality changes or maybe even features that get removed or totally rewritten.

  The whole vendoring is Go's "solution" for not supporting version'ing of projects from the get-go. They refused to have any version support in there modules and it bit them in the behind now. So they solved it by vendoring ( having external packages linked to a project per project base, instead of the global scope ).

  Nim already supports vendoring more then Go ever did because of the way Go enforced there directory structure. I can easily have separated packages in Nim without fighting the directory structure.

  This is one of the reason why i picked Nim over Go. Because the whole darn language ( while nice & popular ) is so darn restrictive, when your opinion does not match the developers, it can really lock you into dealing with annoying problems. The more the scope of a project expands, the more you run into problems like that with Go ( in my opinion ), and i hate to fight the system.

  Its like dealing with PHP and the 7 year people fought the developers for having array() = [] ... That is the nice thing about Nim. Do not agree with the developers? Simply write a macro and deal with it yourself. Disadvantage: On some parts Nim is more difficult to understand the logic ( especially when you have been stuck writing PHP code for the last 10 years lol ).

---

• dom96 (orginal) [2016-04-15T11:03:27+02:00] view original

  Thank you both for explaining.

  Based on what i've read I think I can safely conclude that this isn't a problem in Nimble. In Nimble, you can have ~/.nimble/pkgs/foo-1.0, ~/.nimble/pkgs/foo-1.2, ~/.nimble/pkgs/foo-2.0 etc. installed side-by-side. So we have versions and multiple versions of the same package can be installed at the same time :)

  It still may be useful to have the option to vendor packages (perhaps a flag to nimble build or whatever like --vendorDir) so that you can send a full zip file including all dependencies to people that can be built easily without relying upon the package database/GitHub/the internet at all.

  I don't think this is necessary, because you can actually do this very easily without Nimble: nim c --path:<vendorDir> file.nim.

---

• Stefan_Salewski (orginal) [2016-04-15T11:24:32+02:00] view original

  That sounds interesting. So nimble can be used now for my ten gtk3 related packages too? The problem was not GTK3 itself, because its name is unique, but the corresponding cairo, pango, glib, gobject... modules. Some people suggested calling them cairo-salewski in the past, but I think nobody likes to write "import cairo_salewski as cairo" in all their modules. I will try to read your new nimble chapter soon. GTK 3.20 will be available on github soon, I will create a new top level directory for that and the ten related files, intended toplevel name is NGTK3.

---

• euant (orginal) [2016-04-15T12:39:09+02:00] view original

  @dom96: Fair enough, I should probably check what's there already a bit further ;) I'm glad versioning is supported out of the box, definitely a big plus!

---