Compared to Rust's cargo install
it looks like nimble install sequence is
foreach package do:
download()
check()
install()
foreach package:
download()
foreach package:
check()
install()
Yes, right now nimble installs all dependencies and resolves them at the same time. Main difference between cargo and nimble is that latter one uses script (nimscript) to define list of requirements, making it harder to statically (only using package manifest file) reason about (i.e. to get list of requirements you need to download package and eval it's manifest).
Also see https://github.com/nim-lang/nimble/issues/890#issuecomment-774476074
Once that manifest has been evaluated, though, couldn't nimble.directory cache the results and provide them over some API?
Or am I being naiive and is there no one to one mapping from a package version to a dependency graph, since nimscript.
If that's the case then followup question: wtf?
I think so too - it could, and should cache results. And implementing this part of package manager as an API has its own benefits too (like not having to reimplement package manager dependency resolution for tooling that needs to read package dependencies for example).
Even more - I did some testing and found that 96% of all packages can be statically parsed (although there is no 100% guarantee that some package does not contain some weird logic that cannot be statically parsed, like https://github.com/juancarlospaco/plz/blob/4a2f243ca6af7290c3b8851edf8534899f8f4c5d/plz.nimble#L1)
A centralized package index that defers all version/dep metadata to the packages is nice & flexible, but is also costly.
Maybe I'm wrong, but as I understand it, you would need to clone each whole repo anyway to get all versions of its .nimble file..expensive to build the graph for the whole nimbleverse. My clone of the nimbleverse is only about 5 GB (3 GB for just the .git). Not all users have fat network pipes and that can be slow.
We could maybe create a "meta-repository" that had just all the .nimble s across all their versions and all packages that got updated daily or something. Just brainstorming. Very unsure what the best solution is here. Delegation to package authors is good, but OTOH it definitely complicates global package graph analysis.
To get 100% reliability for parsing .nimble it is necessary to have full repo cloned - but that is another topic of having absolutely unconstrained turning-complete language that can easily do anything just when you install the package. Which is the source of major pain regardless, and it will be more and more difficult to deal with as number of packages that rely on this would only go up over time.
But so far the number of such packages was relatively low - as I already said, simply parsing PNode from package is sufficient for almost all cases.
total package count: 1565
processing ok: 1382
not using github: 51
http error when getting: 87
configuration parse fail: 19I've updated implementation a little since I got these numbers, so it should work even better, but 19/1565 packages is approximately one percent. And believe it is absolutely useless to try and work around every concevable way someone might write requirements, and instead try to contain this into something that can be statically reasoned about.
Note that I'm not trying to completely remove nimscript from .nimble. Of course it is necessary to have nimscript eval for task and other, but I think it is totally unnecessary for requires = ""
that might look something like the crates.io index I cloned a snapshot branch out of interest, it was a 90MB download, du -h shows 502MB. this is for ~39k packages.
this article "So You Want To Write a Package Manager" behind the chain of links referenced by @haxscramper is worth bringing to the top as well.
Yes, this can definitely be done and eventually we should get to a point where it is possible.
The only question is deciding how to do it and having someone implement the necessary changes to make it work.
An API in nimble.directory (or another service) is a good option to speed up the downloads, the only downside is that it makes Nimble dependent on this service, but if it's just used as a cache then in the worst case the downloads will just be slower if the service is down.
But to be honest it seems that in the long run we want a centralised repo anyway. Git/hg repos can easily disappear or be changed in nefarious ways without us reviewing the changes, so we'd want to at least have a centralised repo of all package releases as a backup.
Made some mockup here
It downloads the .nimble file and extracts the dependencies. If the package dependencies were in packages_official.json it would be a lot faster
You should look at how nimp implements dumpIni/dumpScript.
Also fetching just the version control head of .nimble will give you today's requires but not yesterday's or last years. Someone should maybe study how dynamic those are over the years.