Mirror of forum.nim-lang.org

14024 :: SweetSyntax - A generic parser and AST explorer

• georgelemon (orginal) [2026-07-03T15:15:28+02:00] view original

  This is a WIP generic pratt parser and AST explorer for analyzing and manipulating code in various programming languages. Language syntaxes are defined in a simple YAML format (see js.yaml), there you can add custom statement handlers that will need to be implemented at Nim level (example for js parser) - dynamic lib loader is planned via pkg/pluginkit.

  Repository: https://github.com/openpeeps/sweetsyntax

  Initially, I wanted to make a JS/CSS minifier as a built in feature for Tim Engine, but I realized it would fit better as a language-agnostic library, so here's the base, making it possible to build high-level tools for analysis and manipulation, like linters, bundlers, minifiers, formatters, transpilers, desktop apps, online editors, so on. Also, screw other parsers written in C/C++/Rust/bleah. Nim is way more productive and expressive!

  Currently, SweetSyntax can parse and generate AST for

  • JavaScript (almost done)
  • Nim (not fully supported, actually, is harder than expected lol)

  A basic CLI is built-in, you can use it to parse and print AST, sweetsyntax -h

  
  A generic syntax highlighter, tokenizer, parser and AST explorer
    (c) George Lemon | MIT License
    Build Version: 0.1.0
    
    parse <script:path>          Parse a script by extension
    ast <script:path>            Generate AST of a script by extension
            -o:bool
            -y:bool
  Run

  Context-error repporting for humans:

  
      
      usingClientEntryPoint:,
                            ^
  Error (29852:27) Unexpected prefix token: ','
  Run

  Is not perfect but it does a good job at parsing most of the code, and it is very fast (don't forget that even if YAML syntaxes are statically defined, the parser is still generic and dynamic). Benchmarks parsing some popular JavaScript libs:

  Minified Bootstrap 5

  
  hyperfine -N --runs 5 'sweetsyntax parse bootstrap.min.js'
  Benchmark 1: sweetsyntax parse bootstrap.min.js
    Time (mean ± σ):      17.4 ms ±   0.3 ms    [User: 14.6 ms, System: 1.6 ms]
    Range (min … max):    17.1 ms …  17.7 ms    5 runs
  Run

  Unminified d3.js v7.9.0, ~20k lines

  
  hyperfine -N --runs 5 'sweetsyntax parse d3.js'
  Benchmark 1: sweetsyntax parse d3.js
    Time (mean ± σ):     117.2 ms ±   0.7 ms    [User: 109.4 ms, System: 5.9 ms]
    Range (min … max):   116.4 ms … 118.0 ms    5 runs
  Run

  Unminified React DOM v18 ~29k lines

  
  hyperfine -N --runs 5 'sweetsyntax parse react-dom.development.js'
  Benchmark 1: sweetsyntax parse react-dom.development.js
    Time (mean ± σ):      91.3 ms ±   0.7 ms    [User: 84.7 ms, System: 4.9 ms]
    Range (min … max):    90.4 ms …  92.3 ms    5 runs
  Run

  Cheers!

---