Does anyone use netCDF file format?

[drjdpowell]

I'm developing, for a client, a wrapper for netCDF4, a format that used for large atmospheric-science data sets that is implemented on top of HDF5.  Seems quite nice so far, and I was wondering if anyone had any experience with it?  Or HDF5 in general (which I have never used)?  It's kind of TDMS-like, but with n-dimension arrays.

Edited December 16, 2016 by drjdpowell

[gb119]

HDF5 is used quite widely in big facilities based research (synchrotrons, neutron sources and such like). It's a format that supports a virtual directory structure that contains meta-data attributes, and multi-dimensional array data sets). Although it's possible to browse and discover the names and locations of all of the data in the file, it's generally easier if you have some idea of where in the virtual file-system the data thatr you are interested in is being kept. There;s a couple of LabVIEW packages out there that provide an interface to HDF files and read and write native LabVIEW data types - they work well enough in my experience, although I haven't personally used HDF5 files in anything other thn proof of concept code with LabVIEW.

[GregSands]

I've used HDF5 in two situations (so far) for quickly capturing large 3D imaging datasets (XYZ or XYt) - I've used the open source h5labview package which has worked very well for me.  The other LabVIEW package (not open source) that wraps HDF5 is Live HDF5, but unfortunately the two do not play nicely together, so it's not possible to install both at the same time to evaluate.  The HDF5 libraries are mostly wrapped directly, with a small interface dll just for handling some of the memory allocation issues.

HDF5 is similar to TDMS, but much more flexible, and not just in array dimension - H stands for Hierarchical, which basically means anything can go anywhere - data, attributes (metadata), groups, ... - even seamlessly across multiple files.  What I think is most flexible about it is that you can have different arrangements of your data in memory and on disk, and HDF5 handles the mapping between.  Needless to say that can also get very complicated, so I tend to stay with fairly simple mappings!  Used simply, it's straight-forward, but there's a lot of power underneath.  I've said more than once that I wish NI had built their data format on top of HDF5 rather than developing their own proprietary format - in the way that Matlab now uses HDF5 as the internal format for .mat files.

[drjdpowell]

So, no one has used netCDF then?  I'm trying to see if there would be any interest in a netCDF LabVIEW API.   The API I've made is incomplete, as I can only justify implementing the minimum needed for the client who's paying for it.   Seems like it's better to just use HDF5 directly, unless one already uses netCDF (which I think is mainly just the Atmospheric Science community).  

[drjdpowell]

On 19/12/2016 at 8:11 PM, GregSands said:

 The HDF5 libraries are mostly wrapped directly, with a small interface dll just for handling some of the memory allocation issues.

Just a side comment, but I think it is better to avoid a middle-layer dll if at all possible.  I have yet to find a memory allocation issue that could not be solved with LabVIEW.exe:MoveBlock.

[smithd]

The wrapper also ensures that if you hit abort, any open files are closed automatically through the callback system. How would you accomplish that without some middle layer?

[GregSands]

No I've not used netCDF.  I presume that if netCDF uses HDF5, then it is just a particular file layout (combination of dataspaces and attributes) that ensure that all the required information is stored in a well-defined place.  I agree that using/supporting HDF5 would be of more use to more people.

The h5labview code is here (<500 lines) if you wanted to see what few things it is doing.  It's all open-source, so I'm sure Martijn would be appreciative of any extra development.  From the FAQ :

Quote

Ideally, work done in the DLL should be limited to that which cannot be done in LabVIEW (global DLL objects, function callbacks), but in practice any code that becomes messy to implement in LabVIEW but easy in C should be put in the helper DLL. So far this is limited to error handling and raw data/type manipulation.

 

Edited December 22, 2016 by GregSands

[Rolf Kalbermatter]

On 12/22/2016 at 0:24 PM, drjdpowell said:

Just a side comment, but I think it is better to avoid a middle-layer dll if at all possible.  I have yet to find a memory allocation issue that could not be solved with LabVIEW.exe:MoveBlock.

While the middle-layer is indeed an extra hassle, since you have to compile a shared library for every platform you want to support, it is for many cases still a lot easier than trying to play C compiler yourself on the LabVIEW diagram. Especially since not all LabVIEW platforms are equal in that respect (with 32 bit and 64 bit being one but by far not the only possible obstacle). Yes you can use conditional compile structures in LabVIEW to overcome this problem too, but at this point I really feel like using duct tape to hold the Eiffel tower together. Maintenance of such a VI library is a nightmare in the long run.

Not to forget about performance. If you use a middle layer shared library you can often directly use the LabVIEW datatype buffers to pass to the lower layer library functions, with MoveBlock you often end up copying any and every data back and forth multiple times.

And smithd points out another advantage of a middle layer. You can make sure that all the created objects are properly deallocated on a LabVIEW abort. Without that the whole shenanigan is staying lingering in memory until you close LabVIEW completely, possibly also keeping things like file locks, named OS pipes, OS events and semaphores alive that prevent you from rerunning the software again.

Edited December 28, 2016 by rolfk

[ShaunR]

4 hours ago, rolfk said:

Maintenance of such a VI library is a nightmare in the long run.

I have found this not to be the case especially if the alternative is a statically linked "middle layer" where you have to rely on the developer to release a new one whenever the other libraries are updated. I've found conditional statements a superior solution when the main libraries are already supplied by the developers or operating system.

4 hours ago, rolfk said:

You can make sure that all the created objects are properly deallocated on a LabVIEW abort.

We need a definitive guide to using the "Instance Data Pointer" which can alleviate, if not remove, this.

Edited December 28, 2016 by ShaunR

[Rolf Kalbermatter]

1 hour ago, ShaunR said:

We need a definitive guide to using the "Instance Data Pointer" which can alleviate, if not remove, this.

It's no magic really, although I haven't used it myself yet. I make use of other features related to so called UserDataRefnums that are although not really documented a bit more powerful and flexible than the (IMHO misnamed) "DLLs Callbacks".

Basically each Call Library Node instance has its own copy of an InstanceDataPointer. This is simply a pointer-sized variable that is associated with a specific Call Library Node.

You have the three "Callback functions" Reserve(), Unreserve() and Abort(), each with the same prototoype

MgErr (*proc)(InstanceDataPtr *instanceState);

So each of them gets a reference to the the Call Library Node instance specific pointer-sized variable location.You could store in there directly any 32 bit information (it's of course 64-bit on 64-bit LabVIEW but you do not want to store more than 32-bits in there for compatibility reasons for the case where you might need to support 32-bit LabVIEW and OSes, such as Pharlap, VxWorks and NI Linux ARM targets) but more likely you will allocate a memory block in Reserve() and return the pointer to that memory block in this parameter. In addition you should make sure the memory is initialized in a meaningful way for your other functions to work properly.

The Unreserve() callback is called before LabVIEW wants to unload the VI containing the CLN in order to deallocate anything that might have been allocated or opened by the other functions in the InstanceDataPointer including the InstanceDataPointer itself.

Abort() obviously will be called by LabVIEW when the user aborts the VI hierarchy.

Now these three functions in itself are not very helpful on their own but where it gets really useful is when you add the special function parameter "InstanceDataPointer" to the parameter list in the Call Library Node configuration. This parameter will not be visible on the diagram for that Call Library Node. Instead LabVIEW will pass the same InstanceDataPointer to the library function as what is passed to the three callback functions.

Your function can then store extra information during execution of the function in that InstanceDataPointer that Abort() can use to properly abort any operation that the function itself might have started in the background, including closing files, aborting any asynchronous operation it started, etc, etc. Depending on the complexity you can probably even get away with not implementing the Reserve() function specifically but instead have each function invocation check if the InstanceDataPointer is NULL and then allocate the necessary resources at that point. It may be a performance optimization in not allocating an InstanceDataPointer on load of the VI but only on first execution, so if someone only loads the code without ever starting it, you won't unnecessarily allocate it.

If you ever had the "joy" of using Windows API functions with asynchronous operation you will recognize this scheme from the LPOVERLAPPED data pointer those functions use.

Remains to stress the fact that every Call Library Node instance has its own private InstanceDataPointer. So if you have 10 Call Library Nodes on your diagram all calling the same library function you still end up with at least 10 InstanceDataPointers. I say here "at least" since this would be multiplied with the number of clones that exist for this particular VI when you have a reentrant VI.

As to providing ready made samples with code, that is a crux with this kind of advanced functionality. As it involves asynchronous programming it really is a rather advanced topic. Anyone who understands the explanation as above will pretty readily be able to apply it for their specific application and others who don't won't be helped much with an example that doesn't match their specific use case almost perfectly. Even I get myself regularly lost in the pointer nirvanas where an asynchronous task is accessing the wrong pointer somewhere that the debugger is having a hard time to reach into.

Edited December 28, 2016 by rolfk

[martijnj]

Hi all,
I developed h5labview for storage and analysis of large, rapidly acquired scientific datasets during my PhD, because of a lack of alternatives at the time. It's been stable since 2012 and provides a robust interface to the HDF5 library. In my case, the requirement for an intermediary library was motivated by (1) need to ensure file handles were closed - especially to prevent locking of networked files, (2) to handle translating between HDF and LabVIEW memory layouts and (3) to implement library callback functions. For simple N-dimensional numeric datasets the interface is trivial, but for arrays of strings, and also for clusters, it is complex. Handles and "Pascal" strings are not HDF-native and there are significant headaches that get introduced for aligned memory on 64-bit platforms.

I devoted significant time to working out the complex issues that arise for heterogeneous datatypes, and the polymorphism necessary to handle arbitrary dimensional datasets and arbitrary clusters. The result is available for all to learn from as the project is open-source, although I could probably distill the information gleamed from trial-and-error into a better document!

I'm still adding features as I have time to (waveform support is about to be released) but am definitely interested in other developer feedback. My understanding is that netCDF-4 is an HDF5 file with specific applied structure, although I've never used netCDF myself. But I would encourage you to look at the examples on my project webpage and am happy to answer why I did things the way I did. Much of it is because of "pointer nirvana" as Rolf called it, and cluster memory alignment magic. Also, XNodes for arbitrary type adaption without requiring slow flattening-to-string - which is simply too costly for large datasets.

Cheers,
Martijn

[ShaunR]

18 hours ago, rolfk said:

Abort() obviously will be called by LabVIEW when the user aborts the VI hierarchy.

Nice write up.

I was going to write some examples but for the life of me I couldn't think of one real world problem that it solves . I keep looking at those functions and coming back to this every couple of years in case I've missed something but every time get stumped by by the per node instance nature and being unable to pass a parameter into it  Most modern APIs use opaque objects/structures and it is these we need to clear up rather than the function call instance. I guess it is meant for managing thread safety but we are concerned with a purely IDE event so we can unload a resource as the final operation. It is a design-time problem alone.

The classic requirement is to prevent error 5 when aborting a SQLite query and requiring a restart of LabVIEW to close the handle. I can do this by installing a "monitor" into the IDE but it's an awful solution. I can't think of any way to utilise these features for that use case without an intermediary - you can't even [object] reference count .

[drjdpowell]

Interesting discussion.

My bias against a middle C layer is partly related to my lack of C coding skills.  I can use a third-party library with the confidence that I can fix bugs or add features to the LabVIEW part of it, and with some confidence that the underlying dll (such as HDF5) is widely used and thus probably well tested and near complete.  But issues with the middle layer leave me stuck.  

It also matters how complex the library being wrapped is.   My understanding is that netCDF is a simpler API than HDF5 (with the tradeoff that it cannot do everything that HDF5 can do), and I haven’t found the LabVIEW code needed to call it directly is that complicated.

Regarding the Unreserve() and Abort() callbacks, I would much rather be able to register a VI callback when a VI hierarchy goes idle.  It could then do any cleanup actions like closing things.  Perhaps I should suggest this to NI.  I can, and sometimes do, use such cleanup functions using the “watchdog” asynchronous actions in Messenger Library, but I can’t add Messenger Library as a dependency for a reusable wrapper library (also, asynchronous watchdogs are problematic compared to synchronous callbacks due the possibility of race conditions).  

[drjdpowell]

...but am definitely interested in other developer feedback.

Hi Martijn,

Perhaps we should start a new topic on your library; I’ve had a look at it and could make some comments/questions.  For example, why are you making your dll calls in the UI thread?  Is HDF5 not thread safe?  

— James

[ShaunR]

15 minutes ago, drjdpowell said:

 Is HDF5 not thread safe?  

— James

Only if you compile it to be so, then perform back-flips in your code.

Edited January 1, 2017 by ShaunR

[drjdpowell]

That means netCDF isn’t thread safe either.  In a previous wrapping of a non-thread-safe library, MDSplus, I used a semaphore to serialize access, rather than the UI thread.  

[smithd]

He did the same thing with a DVR in the zeromq code, I wonder if there isn't a different reason here (or maybe he came up with the DVR idea later).

[ShaunR]

7 hours ago, drjdpowell said:

That means netCDF isn’t thread safe either.  In a previous wrapping of a non-thread-safe library, MDSplus, I used a semaphore to serialize access, rather than the UI thread.  

That wont work, especially for recursive mutexes.

34 minutes ago, smithd said:

He did the same thing with a DVR in the zeromq code, I wonder if there isn't a different reason here (or maybe he came up with the DVR idea later).

Neither will that.

You cannot mitigate thread safety in LabVIEW. It has to be done in the library itself or an intermediary that can install the lock callbacks and handle them. Serialization != thread safety and you cannot guarantee that a CLFN will always use the same thread on each call unless it is the UI thread (because there is only one).

Edited January 2, 2017 by ShaunR

[drjdpowell]

8 hours ago, ShaunR said:

You cannot mitigate thread safety in LabVIEW.

Depends what the issue is.  The MDSplus library I mentioned uses a global variable.  It doesn’t matter if function calls are made in different threads; it just matters that critical code sections that use that variable be protected from parallel access, and thus I used a semaphore for serialization.  So I may have been using “thread safety” imprecisely to include race condition issues.  What’s the proper term for a library that requires serialization but not use from a single OS thread?

[drjdpowell]

especially for recursive mutexes

I used a mutex count for that, releasing the semaphore when the count reached zero.   That allowed me to call locking functions from inside other locking functions.

[drjdpowell]

An HDF5 reference I found discussing its issues with thread safety.  From it I gather serialization would probably be sufficient.

 

[ShaunR]

32 minutes ago, drjdpowell said:

Depends what the issue is.

No it doesn't.

32 minutes ago, drjdpowell said:

The MDSplus library I mentioned uses a global variable.  It doesn’t matter if function calls are made in different threads; it just matters that critical code sections that use that variable be protected from parallel access, and thus I used a semaphore for serialization.

I don't know this library but from your description it sounds like it is "thread safe" or single threaded. The developers will have (or should have) made a statement about it somewhere.

43 minutes ago, drjdpowell said:

What’s the proper term for a library that requires serialization but not use from a single OS thread?

Well. Serialization means writing binary as strings to me (like the flatten functions,XML etc). So I'm already floundering with your question. Synchronisation maybe? But thread safety isn't about one problem. It is an umbrella term for multiple undefined behaviours when using multiple threads, including race conditions, pointer overflows, memory dereferencing, counter overruns,  and lots of nasty stuff that will crash LabVIEW-probably randomly and when already deployed to the customer site.

LabVIEW uses thread pools and there is no guarantee that a node will use the same thread from the pool for every call of a particular node. Unfortunately (or fortunately, depending on your perspective). Most of the time LabVIEW tries to use the same thread if it is available, for performance reasons, so for the most part everything seems to work great - right up until it doesn't. So I hope you can see that it really doesn't matter what you do in the LabVIEW code with DVRs, semaphores and other native synchronisation methods. The only guaranteed solutions are to use a thread safe library, use the "Run In UI Thread"  or, if the developers have supplied a callback mechanism, use an intermediary library to mediate. 

[drjdpowell]

Well. Serialization means writing binary as strings to me (like the flatten functions,XML etc). So I'm already floundering with your question. 

Serialization is the act of serializing, which is to make something into a serial form, which is a set of things one following another.  Serial is often contrasted with parallel.  Used in multiple contexts in computer equipment, not just in converting memory data structures into serial bytes.

[drjdpowell]

2 hours ago, ShaunR said:

But thread safety isn't about one problem. It is an umbrella term for multiple undefined behaviours when using multiple threads, including race conditions, pointer overflows, memory dereferencing, counter overruns,  and lots of nasty stuff that will crash LabVIEW-probably randomly and when already deployed to the customer site.

All those issues are about parallel actions; they are not specific to OS “threads”.  Only thread-specific memory is something that would require LabVIEW to use the UI thread.  Otherwise one can serialize prevent parallel calls by any number of means.

Edited January 2, 2017 by drjdpowell

[ShaunR]

10 hours ago, drjdpowell said:

All those issues are about parallel actions; they are not specific to OS “threads”.  Only thread-specific memory is something that would require LabVIEW to use the UI thread.  Otherwise one can serialize prevent parallel calls by any number of means.

Well. The libraries state categorically that you need callbacks when using multithreading and give the code that is required for you to implement (I've just outlined why it is the case and why you can't in LabVIEW). I can only tell you what I know from successfully implementing quite a few different ones; I'm not here convince you.