Hallo all,
I would like to show you a parser generator I wrote:
https://github.com/choltreppe/parlexgen
Short summary why I wrote it:
I have some projects where I need to parse things, and I was using the std/pegs module for that, but I was not realy happy with that solution. So I searched for other modules and found the nimly module, which is a way better solution, but the long compilation time anoyed me and I didn't like that the generated proc is exported automatically (dont get this wrong I dont want to hate on that package or anything, I realy like it otherwise, I actually copied the syntax for the parser definition with some tweaks).
so I decided to give writing my own one a go, with the focus on making it compile fast. Another thing I tried to improve is the compilation error messages (presenting the user meaningful errors that point to their code not my macro code). And I also integrated a system that lets the user define handling of parsing errors as easy as possible.
I already replaced the parser in one of my projects with it and it works well. It compiles in a few seconds and the error handling system works nice for me.
Interesting. Unfortunately I have never tried parser generators myself. For my simple use cases Nim's parseutils or strscan was not that bad. Or npegs. In the book I have an example like
import std/strscans
proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =
while start + result < input.len and input[start + result] in {' ','\t'}:
inc(result)
if start + result < input.len and input[start + result] in {';',','}:
inc(result)
while start + result < input.len and input[start + result] in {' ','\t'}:
inc(result)
proc stt(input: string; strVal: var string; start: int; n: int): int =
if input[start .. start + "Rect".high] == "Rect":
strVal = "Rect"
result = "Rect".len
var x1, y1, x2, y2: float
var name: string
let input = "Rect 10.0 ;20.0,100 , 200"
if scanf(input, "${stt(0)}$s$f$[sep]$f$[sep]$f$[sep]$f", name, x1, y1, x2, y2):
echo name, ' ', x1, ' ', y1, ' ', x2, ' ', y2 # Rect 10.0 20.0 100.0 200.0
Similar code I us a lot myself. Would you suggest using your parlexgen instead, and may you be able to provide an example that does the same as above code with your parlexgen?
You could write it like that:
import std/[strutils, strformat]
import /home/joel/daten/programming/nim/parlexgen/src/parlexgen
type
TokenKind = enum tkRect, tkSep, tkNum
Token = object
case kind: TokenKind
of tkNum: val: float
else: discard
Rect = object
x1, y1, x2, y2: float
makeLexer lex[Token]:
"Rect": Token(kind: tkRect)
r";|,": Token(kind: tkSep)
r"[0-9]+(.[0-9]+)": Token(kind: tkNum, val: parseFloat(match))
r"[ \n\r]+": discard
makeParser parse[Token]:
rect[Rect]:
if (tkRect, tkNum, tkSep, tkNum, tkSep, tkNum, tkSep, tkNum):
Rect(x1: s2.val, y1: s4.val, x2: s6.val, y2: s8.val)
func `$`(rect: Rect): string =
fmt"Rect {rect.x1} {rect.y1} {rect.x2} {rect.y2}"
echo $parse(lex("Rect 10.0 ;20.0,100 , 200"))
The bad:
Traditional lexers are iterator-like, they produce a single token and the storage is reused.
For the lexer generator please steal from my lexim project as much as you like. :-)
Thanks. I updated the lexer. I completly used your lexim for it now. I just have a problem with the lexe.exe (Im not using it currently). When I try to use it I get the following error:
parsejson.nim(515) raiseParseErr
Error: unhandled exception: unknown file(1, 1) Error: '{' expected for an object expected [JsonParsingError]
I tried to try it out, but ... https://github.com/choltreppe/parlexgen/issues/1 😊
Hopefully it's just something I'm doing wrong and an easy fix. Looks like a good DSL for lexer/parser generation otherwise.