I agree with most of your points, but I mind
Until now, macros can only deal with Nim-parsed code and therefore the Nim parser is in the way.
How is it in the way? You can easily write compile-time parsers that work on a triple quoted string literal and produced NimNodes, see how strformat does it.
Other points read a bit like "Within Nim there is a much smaller and cleaner language struggling to get out"... Which I agree with, and we will get this better Nim language -- eventually. Proposals like https://github.com/nim-lang/RFCs/issues/369 are highly appreciated.
My ideal language is something like a modern C or a minimalist C++. A language with built-in concurrency library, unicode support, basic OO constructs, generics, intuitive metaprogramming, clean syntax, easy interoperability with C, and performant. Being memory safe is a plus, but I am fine with constructors/destructors, smart pointers, and manual memory management.
Nim is already close to this. There is no more header files, and other clunky C/C++ stuffs. The problem is Nim tries too hard to be memory safe. Nim ends up with all kinds of GCs and each has its own limitations to ensure memory safety. We have ref and ptr, alloc and allocShared, etc. It seems that Nim is now exploring the ownership ala Rust (sink, lent, move). I certainly hope Rust lifetime stuff does not creep into Nim.
It is okay to be memory unsafe. C and C++ are not memory safe and they are still popular. It is better to pass the control back to the users than to complicate the language. They know what they sign up for when they learn the language.
Note that with all that fancy memory model, Rust still need smart pointers to cover for corner cases. I believe Rust has six of them (probably more), while C++ only have three. In C++, smart pointers are not necessary and more like convenience tools, while in Rust they are required.
Even though Nim has lots of GC options, most seem like toys or are for special use cases. Like @Araq mentioned here, --gc:orc changes a lot for Nim's coming evolutions/additions.
Simplification of Nim can easily come in the form of better defaults (like using the orc GC over the refc GC as a default), and also in PRs like the one above which will remove an unnecessary part of the language.
Also, if I am not mistaken, @Araq, you have expressed time and again that Nim will not follow in the footsteps of Rust, when it comes to lifetime annotations.
Having the options for sink, lent, and move I have seen as being most helpful for those who need/desire total control of memory with Nim without a GC or with their own GC and these become explicit annotations for the code's behavior compared to implicit copies and moves like we see in C/C++.
It seems that Nim is now exploring the ownership ala Rust (sink, lent, move). I certainly hope Rust lifetime stuff does not creep into Nim.
So far there are no plans to go beyond sink/lent/move/cursor/acyclic. I know these 5 new things can be a bitter pill to swallow but they are opt-in and also address issues existing since Nim's initial design, not only "memory safety". To the best of my knowledge the design is reasonably complete and the implementation is catching up. Don't worry. :-)
And don't please underestimate memory safety -- there are valid reasons why Rust is taking off and why the entire rest of the industry moved to memory safe languages years ago when they could.
Just imagine you work in a Java/C# shop and you try to sell them to use C(++) instead: "Hey, we get more control over the code and it uses two times less memory then. Occassionally there will be ridiculously hard to track down bugs that can take up months to fix but they are fine because we know C(++) is not memory safe." -- "Er, you know, maybe you should look for a new job..."
To the best of my knowledge the design is reasonably complete and the implementation is catching up.
Cool! Quick question about that: in the destructor manual, the mostly complete example (needs 3 line implementation of resize) of defining a myseq calls the =destroy on all the elements of the sequence. What's supposed to happen when you have a sequence of elements that don't have an =destroy, like ints? Can value types get an implicit no-op =destroy?
And don't underestimate memory safety please -- there are valid reasons why Rust is taking off and why the entire rest of the industry moved to memory safe languages years ago when they could.
So true. Rust got adopted by Mozilla because it handled a lot of the bugs in their C++ codebase. And other large C++ shops are looking for safer alternatives. IMO Nim could be one alternative. For C++ programmers, those 5 new things are hardly a bitter pill. You'd have to add many more things even to catch up to C++ initialization...
I do think @hankas has a point though. Ada, amongst others, is not "memory safe" but it's a lot safer than C++ in practice.
What's supposed to happen when you have a sequence of elements that don't have an =destroy, like ints? Can value types get an implicit no-op =destroy?
Every type has =destroy but it can be "trivial", like in C++.
Every type has =destroy but it can be "trivial", like in C++.
When I took the code from the destructors manual, added a missing resize
proc resize[T](x: var myseq[T]) =
x.data[x.len] = cast[typeof(x.data)](realloc(x.data, x.len * sizeof(T)))
x.cap = x.len
Error: type mismatch: got <int>
but expected one of:
proc `=destroy`[T](x: var T)
first type mismatch at position: 1
required type for x: var T
but expression 'x[i]' is immutable, not 'var'
proc `=destroy`[T](x: var myseq[T])
first type mismatch at position: 1
required type for x: var myseq[=destroy.T]
but expression 'x[i]' is of type: int
For those who use C++, C++ 20 concepts fail this test as well
Well, concepts were newly introduced (Dec 2020), the draft paper had 380+ pages, so give C++ a try, the language will mature soon.
BTW, nice example! I hope you file an issue.
IDK. Probably not. The typing world has changed considerably between ... 2006 and 2021. Type polarity is now well understood. If we obey type polarity, we'll obtain stable substitutions. We can bind early and reduce later. Very similar to the bare-metal lambda calculus. If we do not obey it, the compiler will then constantly have to check the environment and even backtrack if substitutions failed.
Concepts will add a considerable amount of boilerplate , so in some sense, they try to cure a problem with a problem. They can led to combinatoric explosion. They add an (expensive) Meta-DSL to the language. However, if they worked biporarily, they would give us a nice high-level macro-language for almost everything, e.g. the construction of virtual function tables and much more.
I gave some examples above where generics did not work as expected. Something elementary is going on here. A change in the type system (precisely: type inference system) would break existing code. So, a simple add-on needs to be done. Basically at the module (= file) level. We have a clever developer who wrote a baseModule and a user programmer who writes a userModule by importing the baseModule. The pair (userModule, baseModule) should compile and work as expected. That's all about it.
and therefore extend Gstack with a new implementation type Ustack. The implementation of Ustack is in the user's responsability. The user might write now:
import basemodule
type
Gstack[U] = Ustack[U] or Ustack[type unit] # {.prototype.}
Ustack[U] = object
data : seq[U]
len : int
proc len*(st : Ustack) : int {.inline.} = st.len
proc push*[U](st : var Ustack[U], item : U) =
st.data.add item
st.len = st.data.len
proc pop*[U](st : var Ustack[U]) : U =
let len = st.data.len-1
if len < 0 : st.len = 0
else : st.len = len
result = st.data.pop
Really interesting, as your comments usually are. I've started diving into type theory and things you, @timothee and others have contributed are part of the reason.
A modification proposal: the minimal one-pragma approach is elegant, but let's not break import visibility semantics: the meaning of * could remain as it is and one could pass an optional parameter to prototype to denote extensibility:
{.prototype(open).}
or
{.prototype(extensible).}
These could also be valid inside the defining module.
A general thought: prototypical/exemplary definitions seem to go against most peoples' intuition in the context of statically typed languages. It feels JavaScript-y. Comparable to the witness construct you proposed during the discussion about symbol aliases started by @timothee. Back then I felt that supplying "explanatory example code" to the compiler sounded really weird. But now I think that that was irrational, especially with meta-programming making generating various kinds of types possible, at least in principle.
If it was easier to experiment with types, people could get creative and I believe some substantial practical progress could be made. A DSL for type calculus and transformation rules which produces the compiler code for type verification and resolution and defines new type implementations with Nim's existing types as building blocks; is that a pipe dream or a realistic goal?
Well, concepts were newly introduced (Dec 2020), the draft paper had 380+ pages, so give C++ a try, the language will mature soon.
If you read the code at the end of the message you replied to, you'll see a translation into C++ 20 which compiles with both clang and g++ (--std==c++20), and demonstrates the problem.
If you're not going to file the issue with concepts, I guess I will. As for the rest, currently even though type Foo = int or string is valid Nim, it's a lie, Foo isn't a real type, much less a sum or union type. As the manual documents
Whilst the syntax of type classes appears to resemble that of
ADTs/algebraic data types in ML-like languages, it should be
understood that type classes are static constraints to be enforced
at type instantiations. Type classes are not really types in
themselves but are instead a system of providing generic "checks"
that ultimately resolve to some singular type.
Me:"Well, concepts were newly introduced (Dec 2020), the draft paper had 380+ pages, so give C++ a try, the language will mature soon."
If you read the code at the end of the message you replied to, you'll see a translation into C++ 20 which compiles with both clang and g++ (--std==c++20), and demonstrates the problem.
I'm sorry, but you didn't get the irony here....
If you read the code at the end of the message you replied to, you'll see a translation into C++ 20 which compiles with both clang and g++ (--std==c++20), and demonstrates the problem.
Yes, I've read your C++ code. And I appreciate it, because it was much work, way more difficult than my Nim example.
Foo isn't a real type
Of course it isn't. It is used contravariantly only (you can't construct smth with it), and therefore, as a type, it could only be used as a common subtype (that in most cases doesn't exist). Sumtypes are either unions (enforced coercion, like in C, so they can fail) or ADTs. Nim's ADTs are Variants.
I'm sorry, but you didn't get the irony here....
I guess the joke's on me. I couldn't decide if you were serious or not :-)
A DSL for type calculus and transformation rules which produces the compiler code for type verification and resolution and defines new type implementations with Nim's existing types as building blocks; is that a pipe dream or a realistic goal?
A required goal if we want to get out of today's whack-a-mole type system development. ;-)
Whack-a-mole ? Type system ? Really?
Imperative languages do offer product types for both datatypes and function types and typechecking via identity. Some coercions, e.g. int8 -> int32, and conversions, eg. int -> float, are standard. C allows for some form of parametricity , for separate type definitions and type dependencies. Moreover, the C preprocessor allows to split them further, with a "#define" part and a declaration part, rendering all types by forehand. The .c file provides function body and data instantiation and typechecks. The code could then be rewritten with type parameters (see the NimC example above). Without function overloading, it is a parametric (but monomorphic) module instantiation. The .h files add modularity to C. This advantage paved the way for C's success.
If several module instantiations are needed at the same time, function overloading needs to be implemented and we obtain syntactical polymorphism and sets of typesets, allowing for first order type reasoning (inclusion and instantiation order). I call this PMI : "polymorphic module instantiation" PMI gets handled via modules in ML/SML or typeclasses in Haskell.
C++ went another way: It offered function overloading first but without the module-wide type parameters, therefore, no PMI, no type sets. I call this PPMI : "pointwise polymorphic module instantiation", because the extent of the module instantation is completely dependent from the programmer. (There might be no module at all, even if it looks like one). A "whack-a-mole" dependent type system was added later with type generators aka templates. Types can now be "calculated", but need to be restricted too. Therefore, some time (30yrs) later, concepts were introduced. They are the counterpart to templates: Concepts are destructors and templates are constructors.
Nim's generic parameters, like "T" in proc xxxT are nothing else than convenient comprehensions of differing overloads. They are still part of PPMI, they do not add anything new to the type system.
The same with concepts. They are redundant. Example: A programmer implements with a concept that requires a proc myprocex. The proc is not implemented though, because it is not used. Compilation will stop because myprocex is not there. But it could compile because the implementation does not depend of myprocex. Now, the programmer writes a dummy myprocex and the prog compiles. However, the compiler will throw a warning : "myprocex not used". Otherwise, if myprocex had been called, a missing myprocex had led to a failed compilation with and without the concept. Well...
Some people think that a set-like representation of a module is missing, therefore "interfaces". Let's see what we have:
https://dev.to/xflywind/zero-overhead-interface-exploration-in-nim-language-2b3d
nothing is really convincing here, in principle, some boilerplate , an intermediate layer that is not needed and not appropriate because "it is not in the language".
So, what to do?
Modules are missing in Nim. They could be added easily though.
{.prototype(extensible).} These could also be valid inside the defining module.
Yes but you have submodules then. How to make it explicit in recent Nim, it's not in the language yet. Your syntactic extension looks reasonable.
A general thought: prototypical/exemplary definitions seem to go against most peoples' intuition in the context of statically typed languages. It feels JavaScript-y.
A prototype in Javascript is a convenient way to construct local data and local functions via delegate bindings. The prototype provides the constructors. You don't need to know about the concrete types the prototype is working with. The types remain abstract for you. Prototypes can be regarded as (sub)modules. Let's say you are writing an user app Uapp and you have a prototypic module Pmodule providing an abstract type T. So, you have the pair : (Uapp,Pmodule[exT]). JS will bind Uapp to the prototype. The "exT" stands for "existentially bound". This is the abstraction. Since you never can define a free floating [exT] within JS, you always have a monotyped implementation type :
Comparable to the witness construct you proposed
The witness. (It is called this way, not my invention though...) At this point, the abstraction is a mental model only. JS doesn't know about the exT, nor it can declare it. JS determines (standard) types at runtime and is typeless otherwise. Now the other way round (Uapp[exT],Pmodule), where [exT] is abstract for Pmodule. JS is typeless, so you can always pass your own types to PModule. Because the prototype is complete (by definition), PModule will work "out of the box" with your exT. However, due to JS's dynamic contexts (the key feature of JS) and runtime name lookup particularly, you can overlaod PModule's functions and therefore "extend" it.
Now to Nim. A Nim pair (Uapp.nim,Pmodule.nim) to consider with the two abstract cases (Uapp,Pmodule[exT]) and (Uapp[exT],Pmodule). Let us postpone the former for good reasons. Then we have (Uapp[exT],Pmodule) and this is idiomatic Nim, it should work always even if it does not. {.prototype.} will come to help here. If some overloads are specified, it will check the overloads. The most general overload is the unit type - it is completely abstract. If Uapp wants to specify its own overload, it can do this by "extending" the prototypically bound type e.g. Gstack[U] with its own exT and an appropriate implementation. Again, {.prototype.} will check this. If something is missing, it will tell you what to do.
Why are these checks useful? An existentially bound Type together with its implementation is a dependent sum type, a pair (a,B(a)) where a type a produces an implementation type B(a) that contains a . The unit type aside, this will not work in general. In fact, you can expect to get (a and C(a), B(a)) where C(a) extends a with constraints, (a and C(a)) being a subtype of a. If C(a) is already in the context, you are lucky. If not, compilation will fail. Another issue : Since the generic parameters are analyzed pointwise, function-per-function only, C(a) is not stable, it depends of how many B(a) are involved (no module-wide generic annotations, this is a severe problem).
In this regard, concepts are a misconception. They add new constraints (they are destructors). They do define new subclassing properties (examples in the concept RFC), very nice, but they don't remove the implicit constraint subtyping. If they could, they had to construct the appropriate context so that (a & C(a)) always matches with the context.
{.prototype.} would clear up the mess. It provides a (very) basic framework for type abstraction. It defines the set of a valid for B(a), a common supertype "Shape". So it gives a pair (a:Shape,B(a)) without further constraints. BTW it can help for vtables too.
I will not claim that {.prototype.} is a perfect solution, but a solution compliant with nim. A rather cheap solution, cheaper than concepts.
If it was easier to experiment with types, people could get creative and I believe some substantial practical progress could be made. A DSL for type calculus and transformation...
Creative and complicated... Well, Haskell relies on higher-order unification (restricted to keep it decidable) for typeclass reasoning. It might be possible to do it simpler. Perhaps Araq could demonstrate how monads get verified with the current concept design. I tried it but I could not figure out how to name type abstractions like (U->V)->V, so I failed.
The last round now: (Uapp,Pmodule[exT]) . Here, the User app doesn't know the witness type exT. It knows about the supertype only, let's call it Shape again. So we have (Shape,B(a:Shape)) then. Example : Uapp can define var x : Shape = Pmodule.new() where x gets "opened" with a type Shape. From now on, any specific operation related to x has to be done with functions that are imported from Pmodule. Again, we could use {.prototype.} for this. Example with a string witness type:
Exttype {.prototype.} = string
The implementation type is a monotype and therefore it can be used in Uapp for declarations. As said, any function that gets applied to Exttype has to be taken from Pmodule for the sake of type abstraction. No functions containing Exttype may be declared in Uapp and the compiler should check this.
TL;DR : Generic annotations [T] introduce implicit and unpredicted constrains/downcasts in Nim that make type reasoning unfeasible. Downcasts can be removed with abstract types. Abstract types are missing but can be brought to Nim without any runtime cost, especially no virtual functions needed.