I made a packages archive with all official Nim packages. You can see it at https://www2.elcritch.net/atlas-packages/ (servers in Frankfurt, cheap vps ftw)
So why? Atlas does the hard part of parsing historical Nimble packages. Adding a repo caches was pretty easy with OpenAI Codex. It's about 15GB with repos and tar files for each package version.
The nice bit, my Atlas branch #package-cache-installer supports this and can download the package info, package metadata, run SAT, and download the deps for my sigils library in 0.657 seconds! A normal run of Atlas takes ~4s. This is on a cloud VPS with fast networking.
It installs Mummy's dependencies in 1.026s vs 14.423s for plain Atlas.
Here's the example:
-> % rm -Rfv deps && time ~/atlas install --pkgs --curl
[Notice] (atlas:project) Using project directory:
[Notice] (atlas:project) Detected project directory: /home/elcritch/sigils
[Notice] (sigils) expanding root package at: /home/elcritch/sigils url: atlas:///home/elcritch/sigils/sigils.nimble
[Notice] (atlas:expand) Expanding packages for: sigils
[Notice] (sigils) Initializing packages: sigils
[Notice] (sigils) Processing packages: sigils
[Notice] (sigils) Initializing packages: variant, threading, stack_strings
[Notice] (variant.yglukhov.github.com) using package archive cache: /home/elcritch/sigils/deps/_caches/variant/package-cache.json
[Notice] (threading.nim-lang.github.com) using package archive cache: /home/elcritch/sigils/deps/_caches/threading/package-cache.json
[Notice] (stack_strings.termermc.github.com) using package archive cache: /home/elcritch/sigils/deps/_caches/stack_strings/package-cache.json
[Notice] (atlas:expand) expandGraph took: 74 milliseconds, 645 microseconds, and 590 nanoseconds
[Warn] (atlas:toFormular) ForceUseHead sorting..
[Warn] (atlas:toFormular) ForceUseHead sorting..
[Warn] (atlas:toFormular) ForceUseHead sorting..
[Warn] (atlas:toFormular) ForceUseHead sorting..
[Warn] (atlas:toFormular) ForceUseHead sorting..
[Warn] (atlas:toFormular) ForceUseHead sorting..
[Warn] (sigils) dependency: mummy is lazily deferred and not loaded
[Warn] (sigils) dependency: cborious is lazily deferred and not loaded
[Notice] (atlas:resolved) inactive packages: mummy, cborious
[Notice] (atlas:resolved) selected:
[Notice] (atlas:resolved) [ ] (variant, 0.3.0@cf8aa946)
[Notice] (atlas:resolved) [x] (variant, 0.3.1@097031a4^)
[Notice] (atlas:resolved) [ ] (variant, #head@097031a4^)
[Notice] (atlas:resolved) [ ] (variant, 0.1.0@aa3a7d3e)
[Notice] (atlas:resolved) [ ] (threading, 0.2.0@b692590d)
[Notice] (atlas:resolved) [x] (threading, 0.2.1@c5a39a03)
[Notice] (atlas:resolved) [ ] (threading, #head@ee77a27e^)
[Notice] (atlas:resolved) [ ] (threading, 0.1.0@49562fa0)
[Notice] (atlas:resolved) [ ] (stack_strings, 1.1.3@59ffb200)
[Notice] (atlas:resolved) [x] (stack_strings, 1.1.4@8fa11c46)
[Notice] (atlas:resolved) [ ] (stack_strings, #head@acb6bd76^)
[Notice] (atlas:resolved) [ ] (stack_strings, 1.0.0@df5f9dbf)
[Notice] (atlas:resolved) end of selection
[Notice] (atlas:solve) loadWorkspace solve took: 2 milliseconds, 950 microseconds, and 648 nanoseconds
[Notice] (atlas:graph) Activating project deps for resolved dependency graph
[Notice] (variant) Getting package archive cache
[Notice] (variant.yglukhov.github.com) using package archive: /home/elcritch/sigils/deps/_caches/variant/variant-0.3.1-097031a4.tar.xz at: /home/elcritch/sigils/deps/variant-0.3.1-097031a4
[Notice] (threading) Getting package archive cache
[Notice] (threading.nim-lang.github.com) using package archive: /home/elcritch/sigils/deps/_caches/threading/threading-0.2.1-c5a39a03.tar.xz at: /home/elcritch/sigils/deps/threading-0.2.1-c5a39a03
[Notice] (stack_strings) Getting package archive cache
[Notice] (stack_strings.termermc.github.com) using package archive: /home/elcritch/sigils/deps/_caches/stack_strings/stack_strings-1.1.4-8fa11c46.tar.xz at: /home/elcritch/sigils/deps/stack_strings-1.1.4-8fa11c46
[Notice] (atlas:graph) Running build steps
[Notice] (atlas:graph) Wrote nim.cfg!
~/atlas install --pkgs --curl 0.36s user 0.07s system 65% cpu 0.657 totalThe nim.cfg:
-> % cat nim.cfg
############# begin Atlas config section ##########
--noNimblePath
--path:"deps/variant-0.3.1-097031a4"
--path:"deps/threading-0.2.1-c5a39a03"
--path:"deps/stack_strings-1.1.4-8fa11c46"
############# end Atlas config section ##########Relevant: https://nesbitt.io/2025/12/24/package-managers-keep-using-git-as-a-database.html
inb4 this is called NimPI (nim package index)
inb4 this is called NimPI (nim package index)
Yeah and there used to be nimble.directory. Though for a PM you also need not just the index but also the package tarballs for releases. I don't think that was ever a thing.
Very cool, I also just finished creating something similar:
I run a simple forgejo docker container, set up a "Rex" (spoilers) organization, and then run a (python, it's what I'm good at) script to call the forgejo api, and mirror every official nim package that uses git (iirc 4 use hg), so I could self host sr.ht instead but ugh. One could use any git service that for this, but forgejo enables auto-pulling at intervals, so it could enable a global update every X hours.
The result is here, with the plan to generate a new and automatically up to date packages.json with extra attributes that the official one.
This also could enable people to self-host pypi or cargo equivalents without the need to trust central services (so no leftpad or whatever incident)
I suppose this is better for nimby rather than atlas though, but still very cool to see others come up with a similar approach.
As an update, I also noticed that forgejo autogenerates tarballs for git tags so this also comes for free.
@elcritch: Seeing your comment here
Setting up a proper packages server isn't that hard. IMHO the harder bit is user logins and all the crap associated with that. However a server with a cron job to do a git pull on the couple thousand nim packages every couple of hours ain't too hard either. Perhaps there's some github tricks for watching repos. :Shrug:
It seems that my approach works because you have the ability to do all of those, you just need to run my script with a cronjob to fetch new packages declared in the packages.json, which could be also handled by a mirror.
As an update, I also noticed that forgejo autogenerates tarballs for git tags so this also comes for free.
Nice! Though "nimble packages" aren't exactly equal to the git folder. It's whatever the Nimble file's srcDir points to that Nim cares about. If you use the repo tarball then for packages like zippy or pixie you get lots of extra test data. Proper archives are easy to generate the archives with git archive once you know the commits.
It seems that my approach works because you have the ability to do all of those, you just need to run my script with a cronjob to fetch new packages declared in the packages.json, which could be also handled by a mirror.
That's pretty similar to what I was thinking! Actually the Atlas tool I made is multi-threaded and will do a git pull and update the individual package-cache.json files.
I was thinking that really all that'd be needed would be an endpoint to "trigger repo update" if someones releasing something and wants the package now. No auth, just an REST endpoint trigger with some timers to prevent spam.
It did cross my mind something like an ssh-key based signature, but really why? Just update from the official git repo.
I run a simple forgejo docker container, set up a "Rex" (spoilers) organization, and then run a (python, it's what I'm good at) script to call the forgejo api, and mirror every official nim package
Nice! I've always been partial to bazaar like approaches and federation.
Also I'd bet that forgejo repo cache could be shared with my Atlas based package-info tool with a few tweaks. Essentially just re-run the tool after triggering forgejo etc, and it'd re-generate the package-cache.json's pretty quickly. Probably < 5 min? It's fairly smart and checks the git HEAD, runs multi-threaded, etc.
The result is here, with the plan to generate a new and automatically up to date packages.json with extra attributes that the official one.
You know ping me on discord with this username or email (see my github profile) if you'd like to collaborate a bit on a meta data format or more!
It'd be cool to have a shared format different tools could use. Like here's cloned repos and also a package cache with extra metadata. Especially if there's a couple of server folks could use it with it'd be more reliable than one solo person. Then it's just a set of URLs folks could add to their tools.
For example, the static releases.json improved a couple of tools already!
Note: I found that adding too much into packages.json makes it become larger which seemed inefficient. So I went with the atlas-packages/<package>/package-cache.json which seems simple and scalable going to that article @mig posted.
I suppose this is better for nimby rather than atlas though, but still very cool to see others come up with a similar approach.
It'd work well with Atlas which can do URL redirects. I have the Atlas PR out for handling multiple git remotes too which would make federation easy.
Though "nimble packages" aren't exactly equal to the git folder. It's whatever the Nimble file's srcDir points to that Nim cares about. If you use the repo tarball then for packages like zippy or pixie you get lots of extra test data. Proper archives are easy to generate the archives with git archive once you know the commits.
If I understand correctly, you're saying that rather than use git tag releases, it makes more sense to see when the .nimble file was updated. I agree, in fact it's how I maintain python libs, I just referenced it because I understand the "soft" necessity that cloning an entire git repo+history+... might not be a good idea. To my understanding there are two (three) ways of doing version bumps. One is on the repo level via git tags (where the tarballs offered by forgejo are useful), and the other is on the project level, by bumping the version on the project definition file (what I do on python, and where your solution fits much more naturally, if I understand it correctly).
Proper archives are easy to generate the archives with git archive once you know the commits.
I guess the issue then is to figure out which are the good commits, which I guess is down to git blame and other automation.
I was thinking that really all that'd be needed would be an endpoint to "trigger repo update" if someones releasing something and wants the package now. No auth, just an REST endpoint trigger with some timers to prevent spam.
It did cross my mind something like an ssh-key based signature, but really why? Just update from the official git repo.
Yes so what forgejo allows you to do is to set up runners that trigger when a repo updates, so that we could auto-trigger running the script to add the auto-updating mirror to the org.
I am linking relevant parts of Forgejo's docs, because I am not fully familiar, and you might get a better understanding that me trying to explain things I only half-know.
https://forgejo.org/docs/next/user/repo-mirror/ https://forgejo.org/docs/next/user/webhooks/ https://forgejo.org/docs/next/user/api-usage/ https://forgejo.org/docs/next/user/releases/
Also I'd bet that forgejo repo cache could be shared with my Atlas based package-info tool with a few tweaks. Essentially just re-run the tool after triggering forgejo etc, and it'd re-generate the package-cache.json's pretty quickly. Probably < 5 min? It's fairly smart and checks the git HEAD, runs multi-threaded, etc.
I think this is achievable with webhooks, if I can set it to send automatically a message on a discord server, I can make it run a command I guess.
If I understand correctly, you're saying that rather than use git tag releases, it makes more sense to see when the .nimble file was updated. I agree, in fact it's how I maintain python libs
Pretty close except that git tags are preferred otherwise use the commit when the Nimble version changed. Makes sense to me as well and it's how Atlas works. Nimble does ... something.
Many Nimble packages don't have git tags. Without using the Nimble version changes or something you'd have provide tarballs for each git or fallback to using full git clones.
I am linking relevant parts of Forgejo's docs, because I am not fully familiar, and you might get a better understanding that me trying to explain things I only half-know.
Thanks! I'll check it out. I'm curious about the folder structure it uses and all.
I think this is achievable with webhooks, if I can set it to send automatically a message on a discord server, I can make it run a command I guess.
Awesome, well yeah with a webhook it should be easy. It'd be easy to add a simple API endpoint for the atlas-packager side. I fibbed and looks like the atlas-packager doesn't do git pull currently. Perhaps that could/should be a separate program that could be wired in like forgejo.