After a few weeks of effort, I've completed the curation of all nimble packages and I've added extra information to the package descriptions. Particularly, there are now new optional fields like long_description, code_quality, doc_quality, project_quality, categories that can be set in packages.json and used by nimble to search or list packages.
For instance, search all packages related to hash functions. How many of them mention "hash" in the description or tags?
$ nimble search hash | grep -E "^[-_a-zA-Z0-9]+:$" | wc -l
29How many of them mention "hash" in the description or tags, or the long description?
$ nimble search hash --full | grep -E "^[-_a-zA-Z0-9]+:$" | wc -l
41
How many of them have a maturity over 2, meaning that there is minimal documentation and code is readable?
$ nimble search hash --full --mat=2 | grep -E "^[-_a-zA-Z0-9]+:$" | wc -l
31
Adding these features to nimble forced me to break the structure of the previous package.json file. I've tried to find relevant long descriptions about packages in the packages documentation, when I did not have to spend a long time. If it took me too much time, and some package authors are not good at describing their work, I let it empty, meaning that these packages will be less searchable than the ones with more descriptions.
Similarly, I've classified the packages into 25 categories: Algorithms, Audio, Cloud, etc. See the full list in the documentation. A package can pertain to multiple categories, and sometimes it was not easy determining where a package fit.
Here is a list of categories with the number of packages associated with each category:
| Category | Number of packages |
|---|---|
| *Dead* | 55 |
| Algorithms | 175 |
| Audio | 35 |
| Cloud | 12 |
| Data science | 20 |
| Database | 50 |
| Development | 91 |
| Education | 5 |
| FFI | 287 |
| Finance | 10 |
| GUI | 60 |
| Games | 51 |
| Hardware | 32 |
| Image | 49 |
| JS | 32 |
| Language | 55 |
| Maths | 54 |
| Miscellaneous | 146 |
| Network | 72 |
| Reporting | 28 |
| Science | 22 |
| System | 70 |
| Tools | 80 |
| Video | 19 |
| Web | 91 |
Last, I've estimated the maturity of the package based on 3 indicators: code_quality, doc_quality and project_quality. These indicators are in the range [1 .. 4], with 1 being the lowest quality and 4 the highest. I've estimated these values based on a few glances at the code, looking at issues or contributors, reading the documentations, etc. These indicators are subjective and depend highly on the mood I was when I looked at a package, even if I tried to keep a scientific evaluation methodology... Again, see the documentation for the ratings used.
If you, as the author of a nimble package, disagree with the description or the maturity ratings I gave to your package, you have the opportunity to change these metadata before I generate the final version of packages.json. You are the best one to know exactly what your package is doing and I expect you can objectively explain it and rate it. In order to do that, just go to the Google Sheet and change the information. On September 22, I'll generate the final version of packages.json for inclusion into nimble packages..
The Google Sheet is sorted on the url field and you should be able to find your packages from your Github/Gitlab or bitbucket account. Don't change the sort order while others are editing the file.
Thanks
doc_quality, project_quality, categories
That is an interesting effort, but you may admit that above categories can be very arbitrary. And unfortunately it can discourage people when they get not maximum point count. At leasts I would become really angry if I get not nearly maximum point count for my packages, as it was so much hard work :-)
Do you list which packages compile/work with latest devel compiler, with latest stable compiler and with new runtime? That info seems to be really interesting. For example I recently installed the great yaml, but still have to test for myself if it will work with latest compiler.
Generally it would be interesting which of the 1000 packages have at least one user. Maybe a plain polling system would be nice, where users can tell: I have just tested successfully package xyz on OS abc with compiler version ijk. That is already great info. And it is only positive feedback, which I generally prefer. Other package without active users may still be great.
Let me take as an example my own project neo, which is rated with 2 on the provided sheet. It is kind of demoralizing to see this number after all the time I dedicated to the project.
I don't know if the code is well structured (I find it easy to maintain, other may disagree) and it is certainly not commented. On the other hand, it has test sets, it is tested and run on multiple platforms and it has multiple examples provided (shouldn't this last points affect docs quality?). On the other hand, another project of mine is csvtools which was more like a couple of day project, yet it has code quality 3 (is it because it has comments? It definitely has less structure and less tests).
I don't take this personally, I know you had to rate a lot of projects on a lot of aspects, and maybe a 2 is deserved here, I don't really give it much weight. But giving official votes to the quality of various aspects of projects can carry a lot of negative feelings
Unittests are not the only kind of Tests that exists, theres BDD, Integration, Design By Contract, etc, so using Unittests as measure is wrong by definition.
Nimbles description does not have a hardcoded limit of length (that cripples usability), making long_description kinda pointless. What if description can optionally have RST instead?.
Ideally it should not need a generic Miscellaneous category, it always end up being the kitchen sink of random stuff. Tools kinda feel the same too.
@bobd
There's one package, for example, that I was planning to use that gets a 2 for code quality
That's precisely my point! You are making a decision based on a cursory assessment by a person who has looked at hundreds of packages in a month. You should not reconsider using that package because the code quality is written to be 2! If you, and other people, do that, the author of that package may lose motivation, essentially for no reason at all. Moreover, say the author uses this as an opportunity to improve the code quality (which may already be high!): should they update packages.json to signal that they have done improvements? It feels awkward. I don't know any other platform that gives votes to packages in this way.
Example on my own packages: I know for a fact that - say - the code quality on Neo is much higher than that on CSVtools or memo. This is because I have written them. But judging from the spreadsheet you may think that it is a good idea to use CSVtools and a bad idea to use Neo.
My only package that gets a 4 on code quality is Alea. While that is not especially bad, I have no clue why that should get a 4 while others get a 2. From what I can tell on my own packages, these votes are essentially arbitrary.
Code/Doc quality is a must indeed, but I refuse to be part of some biased unclear scoring system that threatens to remove my packages I'm doing so in my free time NOT PAID because of unclear process.
code_quality - (Optional) An integer evaluation of code quality between 1 (low) to 4 (highly mature code).
doc_quality - (Optional) An integer evaluation of documentation quality between 1 (no documentation) to 4 (professional documentation).
project_quality - (Optional) An integer evaluation of project maturity from 1 (Abandonned project) to 4 (Living project with community).
Well, Let's check how well did @Araq and @dom96 do to follow their lead. many of their packages code/doc/project quality (2-3) and there're ones even. How would you expect people to create quality projects even if the core contributors score 1,2,3 they don't have a 4 in your scoring system ? Even the official nim-lang account has only 2 projects with 4 :D
Let me be clear, that would be fine if it's a project wanting to be part of stdlib then you can have your committee or whatever do rating process and whatnot, but threatening people to remove their packages from nimble index isn't wise. Actually it might be a better idea that we officially ask people to use full github url from now on and boycott the whole thing now.
If you want to make people's lives easier just show how many packages depending on the package you intend to use to give them confidence.
Some question for you who controls that sheet? who's opinions are considered? on what bases?how frequent that changes? do you open issues on the repos you find not up to your standards to clearly state what are the problems? or does that happen on PR against nimble?
A few more optional field suggestions for the dot-nimble format:
Author Info:
Fetching The Source:
Pre-Installation Requirements:
Other:
See also: Rust's cargo manifest format.
I did not wanted this work to be polemical and I've already described my feeling in a previous thread. Let me explain my personal motivation for this work.
When I'm looking for a Nim package to be used in a development, I turn to nimble search function. But as a package description is a one-liner, frequently I don't have enough information to determine if the package is useful or not. With the --full option, I offer package authors the possibility to describe more extensively the main features of their package. This extended description can be displayed in with nimble list --full or used in search with nimble search -full.
You could argue that I could use Google search or Github search to find more about Nim packages, but why resort to external tools when we have a package manager?
Then, when I've found candidate packages for inclusion in my project, I need to evaluate which package is the most future-proof for inclusion. This evaluation is highly subjective as sometimes one prefers to favour performance against code clarity, for instance, but as a first hint I would like to have a rating I could base a quick decision upon, in order to select the packages I want to evaluate first. I decided to create a maturity indicator in a range of 0 (no more maintained package) to 4 (no risk package, well documented and tested, with active community, etc.). You can use nimble search --mat=level keyword or nimble list --mat=level to find or list packages with a rating over the level given in parameter. Now, how do you calculate that maturity level? As nimble package repository is static in a single packages.json file and that this file is maintained manually, I couldn't use ratings automatically computed. As a first shot, I decided to create the 3 subjective ratings code_quality, doc_quality and project_quality and calculate the resulting maturing with a magic secrete formula. I spent many hours browsing the >1000 packages web sites to set these (again I totally agree subjective) 3 values for each package in order to have starting maturity values.
Regarding the categories, I wanted to add fixed attributes that can be used to filter packages. I'd rather select a pure Nim package than one based on an external library, for instance, and that's the reason why some package are categorized FFI (though it means pure instead of using FFI...). I selected a basic set of categories inspired from Debian ones and discovered later on that's not the best set for the work, but it was to late and I did not want to start again... For the moment, the package categories are not used by Nimble. I hope they can be useful.
I must say that I've seen scrap projects like I've seen many gems. Sometimes, I could not give a high score to a very valuable projects because it did not fit into the scoring grid that I had set when I started the work. Like @andrea said, if some code did not contain a single comment, I had to evaluate it as not easily accessible and its rating was lowered. I certainly committed many errors spending an average of 5 minutes by package. If I missed tests and examples, the package rating should be corrected and increased.
That's the reason why I appeal to the package authors to correct these errors. The Google Sheet can be simultaneously accessed while I can't do it with the JSON file. Do it for your packages or the packages you know well! Try to respect the evaluation grid so these maturity indicators are fair against other packages. This Google Sheet is only a temporary tool easier to use than a JSON editor.
What is the plan?
I don't take this personally, I know you had to rate a lot of projects on a lot of aspects, and maybe a 2 is deserved here, I don't really give it much weight. But giving official votes to the quality of various aspects of projects can carry a lot of negative feelings
That's the reason why I ask the package authors to correct the errors I have done, in the Google Sheet. They are the best ones to know the value of their projects. And even if they over-rate these ..._quality variables, this will only rate the care they put in their projects. It will be indirect measurement for "compiles with a recent Nim version" for instance.
Don't see the resulting maturity evaluation as a vote but more as a statistics. Yes, statistics can be seen negatively or positively, like grades at school, as they synthesize multiple facets of reality in a single number. Use it only as an abstract indicator.
Perhaps the names ..._quality are a bad choice and I changed them a few times. In the long term, these ..._quality variables are planned for removal when the maturity can be computed automatically from project repository and code analysis.
So, first of all, thank you @spip for the effort! Very illuminating.
My 2 cents: may I suggest decoupling formal measurement (which, as some mentioned, should be numerical and rigorous) from best practice guidelines on Nim coding. Personally, I am more interested in the latter first and then in the former.
Maybe this effort can become the beginning of an official "best practices" guide for the Nim package maintainers? There are some proposed guidelines for standard library (https://nim-lang.org/docs/nep1.html), but none exist that I am aware of for Nim package maintainers or Nim coders in general.
I also believe that numerical analysis should be predicated on more exact definition in the guidelines. For example the guideline "Code is well structured" might have a rule in it: "A single function should have no more than 6 levels of indentation." (just making it up, but you get the point). This is a) clear to a human and b) should be fairly easy to implement in a code analysis tool. Eventually I can even see a VSCode plugin that does this type of analysis on the fly and highlights offending code.
The right way to go about this is to allow the community to vote, the herd can do a better job than one person's opinionated scrub followed by the package author's biased adjustment.
Agreed. Github et. al. also already have the better tools implemented for that.
Meaningful quality measurement of software projects is a hard problem. People have written theses, books and sophisticated tools for it. We should leave it to these people and to the developers who know what they need and choose packages based on that. Some bits of info from my experience with useless-to-dangerous pseudo-objective quality metrics in software projects:
Comments: Heavy code comments tend to go out of sync with the actual code. Wrong comments can be worse than none. Nim code can speak for itself very often: the language is concise and mostly clutter-free; choose meaningful identifiers and you won't need a lot of code comments. Doc comments for the public API of a module are of course mandatory except for obvious cases.
"Well maintained" projects: Metrics can include code frequency, number of contributors/LOC, age distribution of PRs, opened bug level issues minus closed ones over time and others. Which ones are important varies by developer preference and intended application: for a lib with very specialized functionality, a "dead" project with low activity may just mean that the code is rock solid and needs no more work. In a project implementing sophisticated scientific calculations, a single contributor doesn't have to be a bad sign. In a high-profile open source project, letting PRs submitted to beef up someones resume slowly die is probably not a bad thing. It depends.
Tests: while no tests are often a bad sign, their existence is not necessarily a good one. I've seen scores of tests which just exist to look good when they go green. Test coverage and orthogonality are better criteria, but again: these are hard and there are tools for that.
In PYPI they are all marked as Beta, it seems people is lazy to categorize their own stuff. (Someone scrapped PYPI and made stats)
NimScript already has predefined task for Test that can take arguments, kinda nimble test integration I guess.
What I don't care about in this lib:
How do I find is it good library?
References from other users are most valuable to me.
People who use libs should be the judges.
When you put numbers to quality attributes people will compare them. In this case quality is measure of how lib conforms to some standard. It is not quality in sense how well it solves problem.
To me it implies that you know what library does, what business problem it solves and how well it works.
Suggestion:
The ..._quality variables were probably bad named as English is not my native language and I couldn't figure out better terms.
Based on people feeling uncomfortable with these variables and indirectly with the package maturity indicator, I'm going to remove this information from the packages list. At this stage of the project, it is easier removing something than adding a missing information. Including it into the packages.json file had a negative impact on the maintenance of that file as it is dynamic information. But now we lose the ability to judge the value of package (put into the term value the basic meaning from the decision grid I've published or the one you imagine). For instance, we won't be able to find which projects are dead (well, we still can with the *Dead* category, 55 out of 1040 packages) or which packages are worth including in a "Nim distribution". This seemed important to me as many packages are of poor quality (that's a personal jugement) and newcomers to the Nim environment could feel deceived as the language itself aimed at code efficiency and performance.
A last comment on these quantitative variables:
Doc comments for the public API of a module are of course mandatory except for obvious cases.
On the documentation measurement, everybody has her own standard for documentation. In this forum, lack of documentation is a constant recrimination by newcomers and old nimers. That's the reason I thought that having at least one public proc documented was a mandatory minimum to get above 2 in doc_quality. It did not seem to me an impossible target to reach for package authors...
As I'm going to remove these quantitative variables, this should not be a problem anymore. Let's talk now about the other improvements I've added to nimble. The only one implemented at the present time is the --full option that displays an extended description (the values in column "long_description" in the Google Sheet) with the search and list commands. I've tried to find descriptive sentences of the main package features from the package documentation. Sometimes, even if the package has a long documentation, I was unable to extract a short description as the documentation is not really user oriented or synthetic. For these situations, I would love package authors to write a short introduction about their packages into the long_description column in the Google Sheet. This makes their work searchable into nimble.
As an example, let say you want to find a web server written in Nim. nimble search web will return a list of packages including microasynchttpserver, mofuw or nimhttpd. Have you spotted that httpbeast is not part of the results, nor httputils or whip? You can find them with nimble search --full web.
The other information added is the categories but they are not used by nimble. They could be used as search filter too or in a packages browser in the future.
PS: I had to revert to the version of the Google Sheet as of September 13th, as someone deleted the "long_description" column. If you have updated the data since that date, check that your input is still there.
For instance, we won't be able to find which projects are dead (well, we still can with the Dead category, 55 out of 1040 packages) or which packages are worth including in a "Nim distribution"..
when I'm searching packages on Github, I can easily see the files date, or search by filtering with "most recently updated", so it's easy to see which one could eventually be dead. and dead don't necessarily mean not functional. if it's the only one package to solve a problem, it could still be useful...
to see which package could be worth including in a "Nim distribution", still with my Github search, I would look at the number of download and the number of stars.