nimforum mirror - Parsing binary data

dloss (orginal) [2016-03-24T18:20:18+01:00] view original

Is there a Nim library to parse structured binary data, such as PNG files, IP packets or ELF binaries?

I'm looking for a parser generator that uses a declarative, high level embedded DSL (in Nim) to specify the data format. Something like Ruby's bindata library, Node's binary-parser or Python's construct package:

https://github.com/dmendel/bindata

https://github.com/keichi/binary-parser

https://construct.readthedocs.org

It seems to me that Nim's flexible syntax, powerful macros and compile-time evaluation mechanisms should be a perfect match for this application.

Has anyone written such a parser library already? If not, how would you approach this?

rgv151 (orginal) [2016-03-25T02:40:55+01:00] view original

have you looked at my struct module?

dloss (orginal) [2016-03-25T15:35:49+01:00] view original

Thank you. Your struct module seems to be very useful for implementing the library I am looking for.

But in the end I'd like to present a higher level of abstraction to the user, i.e. give him a declarative way to specify the fields of the data structures and the dependencies between them.

As a simple example, look how an IPv4 header can be specified with Ruby's bindata library:


# IP Protocol Data Unit
class IP_PDU < BinData::Record
  endian :big
  
  bit4   :version, :asserted_value => 4
  bit4   :header_length
  uint8  :tos
  uint16 :total_length
  uint16 :ident
  bit3   :flags
  bit13  :frag_offset
  uint8  :ttl
  uint8  :protocol
  uint16 :checksum
  ip_addr :src_addr
  ip_addr :dest_addr
  string :options, :read_length => :options_length_in_bytes
  buffer :payload, :length => :payload_length_in_bytes do
    choice :payload, :selection => :protocol do
      tcp_pdu  6
      udp_pdu 17
      rest    :default
    end
  end

pugz (orginal) [2016-04-05T12:13:09+02:00] view original

This thread is extremely interesting to me. For the last couple of weeks, I've been idly kicking around in my head ways to implement Ruby's "bindata" in Nim. Like dloss, I think that Nim is a perfect match for this task.

For example, the IP PDU in Ruby above might look something like:

macro makeRecord(name: untyped, blk: stmt): expr =
  ...

makeRecord(IP_PDU) do:
  endian(big)
  
  bit4(version, asserted_value=4)
  bit4(header_length)
  # alternatively it could be written as field(header_length, bit4) if that was easier
  uint8(tos)
  uint16(total_length)
  ...
  string(options, read_length=proc() = total_length - 16)
  ...


let pdu: IP_PDU = newIP_PDU("\x00\x01\x02...")
if pdu.options == "hello world":
  echo("i love it")

I've got as far as far as a very basic implementation with no interdependent fields, but it's a tremendous hack as I'm still learning Nim macros. Right now it's still concatenating strings and calling parseStmt() to generate the AST!

I've been meaning to post a few questions to this forum about how I might tackle such a project.

Mirror of forum.nim-lang.org

2155 :: Parsing binary data