It might be that my idea here is quite stupid and doesn't really work out. I am also very tired, but before I sleep I would like to get out my idea here.
On a Linux machine the function fork() duplicates the entire process, including all memory and pointers and address space. But this memory duplication is done lazyly with copy on write. Meaning as soon as one of the processes, child or master, writes to memory, that memory section is first mapped to another memory region. This means all memory that is just used for reading will not be cloned at all. My idea here is to use fork() in a smart way to save the state of the compiler, not for parallelism.
My suspicion why it takes so long to run all the tests is, is not because of big tests, but because of meany small tests that have a lot of overhead to start the compiler. So my idea to solve this would be to run the compiler to a certain point and then fork it. The master would then just idle and the child would continue to run the test, for example an expected failure. And then the the next test could just take another fork from the master without processing any imports etc.
To start, I would try simple Nim programs and see if forking works properly. This could be a good way to do things...
@twetzel59 - I use fork (and even vfork) all the time. They work just fine.
@Krux02 - I have never used compiler-as-a-service mode which seems originally targeted at IDEs, but that feature may apply to testing as well. See compiler/service.nim and tests/testament/caasdriver.nim..maybe search for "caas".
That said, I suspect a lot of the time is actually the C compiler, not the Nim compiler. If you can set up a nim.cfg to run tests with TinyCC/tcc (the mob branch), things could probably go much faster.