Leaderboard

Hi all, I developed h5labview for storage and analysis of large, rapidly acquired scientific datasets during my PhD, because of a lack of alternatives at the time. It's been stable since 2012 and provides a robust interface to the HDF5 library. In my case, the requirement for an intermediary library was motivated by (1) need to ensure file handles were closed - especially to prevent locking of networked files, (2) to handle translating between HDF and LabVIEW memory layouts and (3) to implement library callback functions. For simple N-dimensional numeric datasets the interface is trivial, but for arrays of strings, and also for clusters, it is complex. Handles and "Pascal" strings are not HDF-native and there are significant headaches that get introduced for aligned memory on 64-bit platforms. I devoted significant time to working out the complex issues that arise for heterogeneous datatypes, and the polymorphism necessary to handle arbitrary dimensional datasets and arbitrary clusters. The result is available for all to learn from as the project is open-source, although I could probably distill the information gleamed from trial-and-error into a better document! I'm still adding features as I have time to (waveform support is about to be released) but am definitely interested in other developer feedback. My understanding is that netCDF-4 is an HDF5 file with specific applied structure, although I've never used netCDF myself. But I would encourage you to look at the examples on my project webpage and am happy to answer why I did things the way I did. Much of it is because of "pointer nirvana" as Rolf called it, and cluster memory alignment magic. Also, XNodes for arbitrary type adaption without requiring slow flattening-to-string - which is simply too costly for large datasets. Cheers, Martijn