I've returned to playing with a parser combinator toy project I used to learn some Nim in the past https://github.com/CircArgs/parsnim/blob/master/src/parsnim.nim now looking to make it better after reading a bunch more about Nim on a trip.
First, I am trying to move from having to cast everything to seq to some more generic and performant alternative.
openArray seems like an obvious and pragmatic solution and I also wanted to play with concepts. I've encountered a few issues though that I'm not quite sure how to get around.
In essence, I think I want a type that satisfies these constraints:
type
Stream[T] = concept s, t
s[Natural] is T # can index
s[Natural..Natural] is Stream[T] # can slice
for i in s: # can iterate
i is T
s.len is Ordinal # has length
s & t is Stream[T] # has concat
This does not compile with the recursive constraints though. I tried backing out the base functionality without recursion into a StreamBase and then something like:
type
Stream[T] = concept s, t
s is StreamBase[T]
t is StreamBase[T]
s & t is StreamBase[T]
A less complete concept does compile and works with the types I would expect (those that I expect to work with openArray too).
I'm trying concepts to learn this aspect of the language but also because as I understand it, using openArrays to type regular variables is behind an experimental flag. I also was not completely sure whether using openArray would force copies in some scenarios like strings. Moreover, I wasn't clear what benefits openArray would provide over a functioning concept.
A pattern I like to use when manually implementing PEG parsers is to accept the input in some indexable form, and carry the caret as an inout parameter.
proc read_literal(x: string; here: var int): bool =
let bookmark = here
# .. do everything ..
Failures can put here back to marker but otherwise the cursor is advanced on a successful consumption. So far I've found this has the least fangly bookkeeping (compared to, say, returning tuples and having to unpack them and update cursors that way.)
I haven't really used concepts at all. The times I've needed to do something like this I just put templates in place, since the templates use the index operator [] and the compiler will figure out the right thing for that to mean.
The first part of your reply, I think, is mildly similar to what my code does. It maintains the cursor in a state with the stream. The reason I am trying for this generic Stream is because I fancy the idea of the library as just a means to convert between streams. In the most common sense, this would be something like string -> seq[Node], but then you could go seq[Node] -> seq[SomeOtherNode].
I was under the impression that concepts were little used and buried in Nim, but the idea does seem really powerful.
As for templates, it seems like you're proposing switching from the proc s like I have to templates and just allowing the compiler to resolve things. That makes sense to me. I need to change my way of thinking, I guess.
You can remove the recursive part by re-using the type being matched by the concept.
type
Stream[T] {.explain.} = concept s, t, type S
s[Natural] is T # can index
s[Natural..Natural] is S # can slice
for i in s: # can iterate
i is T
s.len is Ordinal # has length
s & t is S # has concat
I think this prevents using array, since & would produce a different type (it is not implemented in std anyway), seq works fine.
For openArrays, I don't know if it is even possible to implement & with an openArray return type.
since slicing and concatenation change the array type you can split up your concepts:
import std/enumerate
type
SliceableStream[T] {.explain.} = concept s
s[Natural..Natural] is BaseStream[T]
BaseStream[T] {.explain.} = concept s
s[Natural] is T
for i in s:
i is T
s.len is Ordinal
Stream[T] {.explain.} = concept s, t
s is BaseStream[T]
s is SliceableStream[T]
s & t is BaseStream[T]
proc `&`[T;N,M:int](a:array[N,T],b:array[M,T]):auto =
var res: array[a.len+b.len,T]
for i,x in enumerate(a):
res[i] = x
for i,x in enumerate(b):
res[i+a.len] = x
return res
similarly, if you accept that concatenation can return a different type, you can implement & for openArray like so:
proc `&`[T](a,b:openArray[T]):seq[T] = @a & @b
and this is enough for
proc foo(x:Stream[int]):bool = true
echo foo([1,2,3].toOpenArray(1,2))
/usercode/nimcache/@min.nim.c:249:30: error: incompatible type for argument 1 of 'foo__in_u2091'
249 | T10_ = foo__in_u2091(T9_);
| ^~~
| |
| tyOpenArray__0HB6wOMHiJYheBmdOv6sWw
/usercode/nimcache/@min.nim.c:137:54: note: expected 'NI *' {aka 'long int *'} but argument is of type 'tyOpenArray__0HB6wOMHiJYheBmdOv6sWw'
137 | N_LIB_PRIVATE N_NIMCALL(NIM_BOOL, foo__in_u2091)(NI* s_p0) {