Hi everyone,
I am currently taking up MS Meteorology and I want to use nim for my thesis. I need a netcdf library but I don't know how to interface existing netcdf libraries written in other languages (I am not a programmer). Could anyone write a netcdf library for nim?
Thanks in advance.
Hey!
As far as I'm aware there are no bindings to the NetCDF library so far. I personally don't have any experience working with NetCDF. However, I'm aware that since NetCDF4, it's actually just based on HDF5. So depending on your use cases it might be possible to use nimhdf5.
At least HDFView can easily read NetCDF4 files. So it should be no problem to read data from a .nc file using nimhdf5. Writing them and staying compatible with NetCDF might be more of a problem however. I don't know what the standard contains and if it's easy to manually follow it.
I'd be willing to give you some help if you want to attempt that.
I'd like to bump this just because I'm also interested in working with kind of thing. In my case, I'm interested in opening .grib2 files. I know the netcdf4 python wrapper can do it - but if I could work in nim, that'd be sweet :)
I looked at the examples section in the HDF5 repo that Vindaar linked to. I didn't see any .grib2 files there, but I guess I could test that out? I thought I'd ask people here if they've tried opening .grib2 using the nimhdf5 library.
Actually in doing some more research - I found that GDAL ( https://gdal.org/ ) opens grib2s, and some kind of netcdf formats. Not sure if its the specific version you're looking for.
Looks like somebody has already started writing headers for gdal in nim, too: https://forum.nim-lang.org/t/3417.
Maybe I should start there.
Up to right now I didn't even know of grib2 files. I need to read up on what kind of file format that is first. I suppose it's based on HDF5 files, too? If it is, I should think we should be able to make it work.
Regarding other nimhdf5 users: I'd love to be corrected, but as far as I'm aware I'm the only actual user of HDF5 files in Nim land so far. Or the other people using nimhdf5 are so happy with it they don't raise any issues. Given the state of basically non existent documentation and extremely sparse examples (sorry about that :/), that'd be a surprise.
It looks like Netcdf4 is based on HDF5 (https://en.wikipedia.org/wiki/Hierarchical_Data_Format). But it doesn't look like grib2 is based on HDF5. Although their usage cases are the same.
That said, if you think the HDF5 library could load up grib2s, I'm all for it. HDF5 looks much more modern than a lot of GDAL's file-loading functionality.
@mikebelanger: Oh, I thought you were sure grib2 was based on HDF5, since you asked about it in reference to the nimhdf5 repo.
If it's a completely different file type then of course loading those files with the library does not work. In that case either wrap a specific library for grib2 files: it seems there's g2clib by the NECP: https://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/ and the readme: https://www.nco.ncep.noaa.gov/pmb/docs/grib2/download/g2clib.readme
You'd have to look at the library and see how complex it is. If you only require a limited subset of the functionality, maybe there are a few functions, which are enough to wrap. A nice interface will be a lot of work of course.
Or continue the work you quoted above wrapping GDAL.
In either case I can try to help (although I don't know how much help I'd be :) ).
If you don't mind me asking, for what do you have to read grib2 files?
@mratsim: Yes, we talked about this before. The basic reading and writing of HDF5 files is already implemented in arraymancer, as you know of course. Writing higher level procs to provide "store to HDF5" and "load from HDF5" for a complex network layout is fine.
I looked into Keras loading back when I implemented HDF5 io if you remember. The reason I stopped for the time being was two fold:
Regarding 2: The main reason just being that a runtime based model creation in Arraymancer wasn't really possible. Since the network DSL creates most of the code at compile time I had a clash with the fact that I can only read the HDF5 Keras files, which describe the layout, at run time.
Not sure if it's even possible to generate performant code at run time only or if there's some optimization happening in the network DSL (if not writing a run time interface should be fine?).
On the other hand of course one could require that the user writes out the network DSL code that describes the model stored in a Keras HDF5 file and provide the resulting context (or type, ...) to the Keras loading procedure. Then we can raise an exception if the model of the file does not match at run time.
In any case, whenever you want to tackle this (or I finally get far enough with my work to need NNs again sigh) we can discuss it and I can (at least) do the IO related implementation.
@Vindaar sorry my mistake! I made the assumption that grib2 was a kind of HDF5 format. But the more I read into it - it looks like grib2 is a totally different beast.
As to why I'm doing this - I'm always looking for more performant ways of reading grib2s. They can get big, and many existing interfaces are slow.
Regardless, thanks for offering your help. I'm probably either going with GDAL, or this neat thing called THREDDs. Hopefully nim will be in the mix, but I'm not sure now.