Reading/Writing HDF Files

[jeffwass]

Has anybody written or read HDF files from LabVIEW? Or perhaps a similar scientific data format, such as CDF or netCDF?

(For those that might not know, HDF files allow a platform-independent, and even application-independent, method for storing various kinds of data. Eg multiple-dimensional numerical data, pictures, color tables, and also descriptional meta-data to go with each data set.)

The HDF distribution comes with a shared library file that contains the input/output routines, which would need to be accessed with CIN. If anybody has used HDF, was it difficult to access the files? Or does anybody know of public LabVIEW code that already exists to read/write to HDF files? And finally, if somebody knows of a reason that CDF or other might be preferrable to HDF, that would also be useful to know at this point.

The reason I need this is because my datasets typically involve variation of several controlled parameters (current, temperature, applied magnetic field, gate voltage, etc). And from run to run I'll vary some of these parameters and hold others constant, and it's rarely the same. So I need a common sane way to save the data such that it can be recalled and processed between runs without needing an individualized data parser for each specific dataset that is written. Eg, one dataset will record differential conductance as a function of current, and repeat this at various temperatures. Another dataset will also record differential conductance as a function of current, but this time temperature will be held constant and I'll vary the magnetic field. I want a sane way to recall and parse the data such that it can be processed using the same program (most likely the parsing would be done out of LabVIEW, so I'm more interested in writing HDF files from within LabVIEW). Multiple-dimensional arrays seem like the way to go here, and with the meta-data allowed by HDF, it seems that it would make sense.

[todd]

I have not worked with those data formats. I am interested in what you learn. Here's some expounding:

The only reason I can think of not to use a relational database is the metadata. I've used a few methods of storing metadata. The attached gif shows metadata in the filename. I've also used ini files with sections for parameters and for data. These are saved as csv so excel will associate (for my colleagues).

The work I've done with databases has been focused on creating tools that make new databases based on the swept parameters. Of course, there is another tool that creates viewers of each database. It's all very clunky and makes intra-database correlations difficult.

[Mike Ashe]

Has anybody written or read HDF files from LabVIEW?  Or perhaps a similar scientific data format, such as CDF or netCDF?

(For those that might not know, HDF files allow a platform-independent, and even application-independent, method for storing various kinds of data.  Eg multiple-dimensional numerical data, pictures, color tables, and also descriptional meta-data to go with each data set.)

3153[/snapback]

Hmm, I know that the FITS file format (astronomy based) has been implemented at least twice by different groups and now that I think of it, it seems that I have read about HDF files too.

Go to the Info-LabVIEW VI archives at:

http://www.info-labview.org/the-archives/vi/lv4

and look for:

hdf_distribution.sit and hdf_distribution.txt

These are in Mac format, but even that might be a good start. Also go to the Iinfo-LabVIEW search engine SearchVIEW.net and search from 2003 and earlier for HDF. You will find a bunch of entries.

Ciao

[Mike Ashe]

You might also want to go to NIs site. They have some tutorials and files for HDF:

Managing Large Data Sets in LabVIEW

http://zone.ni.com/devzone/conceptd.nsf/we...node=dz00000_us

This has a GigaLabVIEW.llb and a couple of HDF DLLs along with some nice tutorial text. There is also a link at the bottom to:

Can I Edit and Create Hierarchical Data Format (HDF5) files in LabVIEW?

http://digital.ni.com/public.nsf/3efedde43...16?OpenDocument

Which has a couple of components that should take care of what you want.

If the links above don't work, just go to http://zone.ni.com and search for the title lines above the links.

[todd]

Here's some text from:

ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/contrib/labview/info

FYI: We have finally released the applications which use HDF5. Included

in the release is a set of LabVIEW VIs and a "glue" DLL to access the HDF5

DLLs. LabVIEW HDF5 users can get the HDF5 utilities by going to

http://www.ni.com/ and typing "scope soft front panel" into the search box.

Select the 1.5 version from the many hits resulting. Included in this

self-extracting ZIP file is another self-extracting ZIP file titled

sfpFile.exe. This contains the LabVIEW VIs to read and write the HDF5

based waveform files used by the soft front panel. Documentation is

included for the higher level VIs. Users only interested in the lower

level VIs will find they match the HDF5 C API very closely. The set of VIs

is not complete, but they handle most common functionality. They can

easily be used as templates to generate additional functions. Requests for

additional features can be directed to info@ni.com. The contact people are

Norma Dorst and Lokesh Duraiappah. If the user wants fast response,

mention at least one of them in the e-mail to ensure proper routing.

Support is through the usual NI routes.

I found 2.5 here but it's still downloading (39MB) so I have not yet checked it for sfpFile.exe.

[edit]

So now I see that Michael provided the link to sfpFile.exe.

[/edit]

[jeffwass]

You might also want to go to NIs site. They have some tutorials and files for HDF:

Managing Large Data Sets in LabVIEW

http://zone.ni.com/devzone/conceptd.nsf/we...node=dz00000_us

That one actually worked, but it seems more focused on demonstrating how to access 64-bit files and structures. But it does include a working example at doing doing some HDF-related functions.

Can I Edit and Create Hierarchical Data Format (HDF5) files in LabVIEW?

http://digital.ni.com/public.nsf/3efedde43...16?OpenDocument

That one actually didn't work for me, it came with a Windows .exe file, which just unzips a bunch of files (an llb along with several mnu files and three DLL's). I followed the instructions to install the "Soft Front Panel", involving unpacking into a subdirectory under "instr.lib" in the LabVIEW directory, and copying the DLL's to the Windows system directory.

Ultimately, it didn't work at all. It was supposed to create a Soft Front Panel pallette, which it did, but none of the sub-links worked at all. I tried playing around with various methods of getting this to work, but wasn't able to.

(Digression here -- ) I must admit I'm new to manipulating the user-defined pallettes. But I've successfully created and used those corresponding to the LuaVIEW and labpython scripting projects. It seems really weird that the NI people can't get something easily working while the open-source community makes it very smooth. Actually, the openG package format is probably the best way of setting up NI packages, it's easy. NI involved manually moving files to various directories, and rememebring where they are if you want to uninstall.

Does anybody else besides me think the NI LabVIEW documention is fairly minimal and lacking? I was so surprised after discovering these communities at how much more LabVIEW could do than the NI documentation led me to believe.

So anyway, thanks for those links, I actually found them after posting here, but wasn't able to get that soft front panel working at all.

[jeffwass]

Update :

As I said earlier, I had problems adding the NI Soft Front Panel libraries into the LabVIEW instrument pallette. Instead of dealing with these palletes, I've been looking at the NI-provided HDF5 wrapper libraries included in the library. These are pretty much a slew of LabVIEW VI's that call the corresponding HDF5 function within the HDF5 precompiled shared library.

I've only done some basic things so far, but these wrappers seem to work okay. Hopefully they'll be all I need to fully save complicated data in the HDF5 format. If anybody's interested, I'll provide some samples of writing various data to an HDF5 file when I am able to do so.

Eventually I might re-do these wrappers from scratch, such that they could be used with the openG project. But that would be a long way off, and I'd only do that if I was pretty sure NI had no plans to make HDF5 functions included in a future LabVIEW release.