Mirror of forum.nim-lang.org

3239 :: Extract repetitive unstructured data

• alfrednewman (orginal) [2017-10-10T00:03:57+02:00] view original

  Hallo,

  Typically, I handle my data wranglers jobs using Python and Perl. I have a lot of legacy code in both languages, so I need to find some free time to rewrite all my stuff in Nim.

  Sometimes, I need to extract key information from an unstructured data file... and I would like to start using Nim to do that.

  Please, loook at this thread: https://opendata.stackexchange.com/questions/92/good-tools-to-parse-repetitive-unstructured-data

  In your opinion, how could I use Nim to solve a pattern like the one below? I mean, what is the best way to parse a great number of lines of repetitive unstructured data ?

  Maria Teresa’s Babies Early Enrichment Center/Daycare
  825 23rd Street South
  Arlington, VA 22202
  703-979-BABY (2229)
  22.
  Maria Teresa Desaba, Owner/Director; Tony Saba, Org. Director.
  Website: www.mariateresasbabies.com
  Serving children 6 wks to 5yrs full-time.
  
  
  National Science Foundation Child  Development Center
  23.
  4201 Wilson Blvd., Suite 180  22203
  703-292-4794
  Website:  www.brighthorizons.com 112 children, ages 6 wks - 5 yrs.
  7:00 a.m. – 6:00 p.m. Summer Camp for children 5 - 9 years.
  Run

---

• jlp765 (orginal) [2017-10-10T03:34:23+02:00] view original

  Taking a "big picture" approach:

  • If you are going to automate extracting data, you must know the rules that define the data "fields" you want to extract, or else no tool will do it for you.
  • most tools have some regular expression (RE) capability (not just perl), but is an RE the answer to how do I identify each field?. If you can delineate between fields without using an RE, then that may make your code faster, easier to modify, or more readable (but it also may not).
  • Are these files small enough that you can read the whole file into memory and process a single string of data, or do you need to process the file iteratively (by line, or chuck of data) to save on memory?

  If you end up using regular expressions in Nim,

  cheat by leveraging what is already done in nimgrep. It is easy to end up with RE that is slow in Nim, so if speed is an issue, make sure you benchmark it. (The RE isn't slow because it uses PCRE. The reason (I believe) is that it is easy (for a newbie) to write code with lots of string allocations that slow it down).

  The following probably applies to more structured data, but I'll mention it for completeness:

  @Araq posted about how using a tool like sqlite can be a good tool if you are then manipulating the extracted data.

---

• alfrednewman (orginal) [2017-10-12T02:25:30+02:00] view original

  Thanks for your advice. Cheers

---