Hey there, is anybody using choosenim on Linux or macOS here and VS Code?
You probably have heard of or experienced the problem, where nimsuggest processes started by VSC or nimlangserver are zombified. This is a known problem with the proxy executables, which choosenim installs for nim, nimble, nimsuggest etc. These start the actual programs from the Nim toolchain version currently selected and are used in lieu of symlinks, probably because these are problematic on some OSes, as I understand. choosenim proxy executables use startProcess to start sub-processes, which means that you get two processes for each instance of nim, nimble, nimsuggest setc. that is started, the proxy executable and the sub-process it starts. And if one of them is accidentally killed, the other can stick around in a non-functional state, in some circumstances.
I have created a PR, which replaces startProcess with execv, which means that the proxyexe process is replaced by the process it calls. This, it seems to me, solves the issue, at least on POSIX systems.
Anyone wants to test this on their system?
Do do so, you would need to install choosenim from my fork with:
nimble install "https://github.com/SpotlightKid/choosenim@#9f2735a"
and then remove any previously installed choosenim proxy executables with:
rm ~/.nimble/bin/{nim,nimble,nim-gdb,nimpretty,nimgrep,nimsuggest,testament}
Lastly, install / re-select the Nim version you want, to install the new proxy executables again, e.g.:
choosenim stable
I'd be very interested, if this successfully fixes the problem with hanging nimsuggest processes and VS Code for those using choosenim on posix systems.
Thanks from the user who raised this issue in first place.
I'm now (manually) using symlinks instead of shims, but I will be happy to switch back to choosenim as soon as this gets fixed. I'll try it asap
Would this fork work on windows?
I don’t understand why those proxies are needed in the first place. Couldn’t choosenim simply use junctions on windows and symlinks everywhere else. AFAIK junctions work well and are quite similar to junctions. Perhaps there are some issues with antiviruses or something like that?
Well this just has the old behaviour on Windows, so it would work, but it wouldn't be any different from what choosenim does today.
I've also been wondering about why choosenim doesn't use some sort of links, it would be a much cleaner solution than these proxy exes.
But great work @SpotlightKid, solving a long-standing issue with choosenim here!
I've also been wondering about why choosenim doesn't use some sort of links
Because on windows the user needs administrator privileges to create symbolic links.
Because on windows (by default) the user needs administrator privileges to create symbolic links.
See Native Windows Symlinks in MSYS2 – Symlinks. That section is a "short primer for native Windows symlinks", it's not specific to MSYS2.
TL;DR it's been possible since 2016 to create symlinks in Windows with a normal user account, but Developer Mode needs to be enabled. Even then, they're quirky compared to Unix symlinks.
@didlybomb wrote:
Couldn’t choosenim simply use junctions on windows...
Junctions can only be created for directories. Windows also has file-to-file hard links, but I don't know if they're suited to this purpose; they also have some quirks, e.g. you can't hard link from a file on C: to a file on D:.
but I don't think these are wrapped in the Nim standard library, and AFAICS you need the Microsoft C runtime library, which is problematic for open source software
Can you clarify?
To my understanding, linking to MSVCRT or UCRT on Windows happens all the time when you build software on Windows, open source or otherwise. Those are system libraries, and e.g. the GPL makes an exception for them, see Can I link a GPL program with a proprietary system library?. For more liberal open source license (MIT, etc.) an explicit licensing exception doesn't seem to be necessary.
What am I missing? (...probably a lot)
To be honest, my knowledge about Windows development is very limited and I'm only parroting what I've read from other, without having thoroughly researched the topic myself. As far as I understand it, you can't rely on the MSCRT being installed already, but you also can't distribute it with your app. So you would need some kind of installer to ensure it gets installed with your app.
But if somebody can show me an example of how to use _exec* functions in Nim on Windows and how to test this on a Linux system, in docker or in the GH CI, I can try to do another PR implementing an equivalent change in choosenim for Windows.
My Windows dev knowledge is very limited as well, but I've picked up on a few things over time. I welcome any corrections from readers here who know better than I do.
MSVCRT has been distributed with (as part of) Windows since Win 95. It's still in wide use by old and new software built for Windows, but at least in theory developers should be / should have been migrating to UCRT, which started shipping as part of Windows 10 in 2015, and is also available for older versions going back to Vista: Download Windows 10 Universal C Runtime.
There are a number of reasons to prefer UCRT, including proper support for the UTF-8 locale, but I'm not sure if it matters re: having choosenim make use of the _exec* functions.
But if somebody can show me an example of how to use _exec* functions in Nim on Windows...
jiro4989/setup-nim-action@v2 is used to install the Nim compiler to build choosenim:
.github/workflows/release_build.yml#L60
.github/workflows/release_build.yml#L90
But it's not clear to me what C toolchain is used in that environment.
It seems like it should be possible in Nim to importc the desired _exec* functions from process.h, but because I'm unsure about the toolchain, I don't know exactly how one would go about it, i.e. does one need to account for MinGW vs. Microsoft's toolchain.
Here's what's supplied by MinGW: process.h#L72-L79
To my mild amazement, using _exec* is rather straightforward.
I have created a small test project, which builds a simple test executable and has a test to test it and a GH workflow for Linux and windows.
Heres the Nim code wrapping execvp / _execvp:
https://github.com/SpotlightKid/nimtestchamber/blob/master/src/nimtestchamber/common.nim
And here's the test:
https://github.com/SpotlightKid/nimtestchamber/blob/master/tests/test.nim
And the successful GH CI run:
https://github.com/SpotlightKid/nimtestchamber/actions/runs/11028200280
TL;DR it's been possible since 2016 to create symlinks in Windows with a normal user account, but Developer Mode needs to be enabled. Even then, they're quirky compared to Unix symlinks.
I don't think you need to have developer mode enabled to use the mklink command, and it does not need admin privileges either.
I just checked that developer mode is disabled on my Windows 11 laptop. Then I opened a (non admin) command line and ran mklink linked.txt existing.txt which worked fine without ever prompting me for admin permissions. Doing the same for a folder also worked fine.
Maybe this could be an option but I suspect that using links would work much better than the current choosenim solution (I don't know if the patch that was the reason for the creation of this thread would work even better, though).
I don't think you need to have developer mode enabled to use the mklink command, and it does not need admin privileges either.
From what I've read, this is actually governed by a group policy. On your system it might be set the wright way. Or all the documentation I've found so far is inaccurate and it doesn't require elevated privileges or a correct policy setting (at least for Windows 11). Anyway, until further confirmation of your findings I will continue to assume that symlinks on Windows are not a solution that works everywhere where choosenim should work.
I have updated the choosenim PR with an implementation for Windows based on _execv:
https://github.com/nim-lang/choosenim/pull/38
Next: write tests
Is this now the correct first step?
nimble install "https://github.com/SpotlightKid/choosenim.git@#3b884e2"
Do I need to rebuild nimlangserver after your proposed three steps?
If you're on an Arch Linux variant, you could just install https://aur.archlinux.org/packages/choosenim :-)
Please note that the patch for https://github.com/nim-lang/choosenim/pull/38 currently has a broken test (when run in the CI with a Ni dev version).
Sorry, I don't currently have time to provide proper tests. I'm looking for help by someone, who can write proper tests for the feature and get this PR merged!