LVHDF5 1.0

[jdsommer]

It's been a long time in coming, but I have completed a new version of the HDF5 Toolkit for LabVIEW that was first released in 2006. The aptly named LVHDF5 Toolkit (version 1.0) is a hefty redesign of the original toolkit (version 0.9) and should solve most of the performance issues with that library while maintaining the flexibility of the the original design.

 

The toolkit is free to use but does have restrictions on redistribution.

 

http://www.upvi.net/main/index.php/products/lvhdf5

 

In particular, I'd like to thank the members of LAVA who worked on xnodes. I've been lurking on the xnode-related pages pages for the past three or four years while slowly building up this revamped toolkit. It would not have been possible without their efforts.

[hooovahh]

Is this another group doing similar work?

 

http://h5labview.sourceforge.net/?home

 

They mention using XNodes for the implementation as well.

 

I've never heard of the file format but it does sound interesting.  Without knowing much I'd still think TDMS is the better approach if you are going to stay in the NI world.  There are of course features of HDF5 that aren't in TDMS, I just don't know if I would need them.

[PiDi]

Out of the pure curiosity: if you wanted to make functions adaptable to connected data type, why didn't you use polymorphic VIs? XNodes are undocumented, unreleased and unsupported feature, so the usage of this toolkit in production code is very limited (if not impossible). I think poly-VIs would also be easier to implement for you and easier to use by end-programmer.

[hooovahh]

Out of the pure curiosity: if you wanted to make functions adaptable to connected data type, why didn't you use polymorphic VIs? XNodes are undocumented, unreleased and unsupported feature, so the usage of this toolkit in production code is very limited (if not impossible). I think poly-VIs would also be easier to implement for you and easier to use by end-programmer.

It's obvious you've never actually developed an XNode, because in my opinion it is easier than making 60+ polymorphic VIs and then having to add them.  I don't want to derail the topic but here are a few reasons XNodes could be considered the better solution:  

 

XNodes make the instance with scripting so any data type can be used, where with a polymorphic VI you need one for each data type.  I've had polymorphic VIs with 60+ VIs and it still didn't meet all the type needs.  Most XNodes are only 5-8 VIs and can cover many more cases, with more functionality.

 

Fewer VIs in memory - Polymorphic VIs load all VIs into memory.  This means all 60+ where a XNode is just a few.

 

Updating a polymorphic VI is a pain because you have to update all 60+ versions.  XNodes have the UpdateType ability which can have revisions like classes and force a mutation to update an older version.

 

XNodes appear to be inlined VIs and run as fast as possible.

 

XNodes when in a built EXE are removed and they are replaced with the VI that is made so it should behave just as a normal VI in a production application.

 

XNodes can react to other input like user interactions that normal VIs can't.

 

Icon updating, and help can be generated based on the specific usage of the XNode.

 

XNodes maybe unreleased and undocumented, but they are not unsupported.  NI has many XNodes, and even promote a few like the Math Node.  And I don't know how a polymorphic VI would be easier for the end user.  My goal is to make life easier for developers and XNodes do that which is why I'm so interested in them.

 

I do think there is a stigma associated with XNodes which maybe why people avoid them, but I saw some amazing things done with scripting in the 7.x era before NI officially supported that as well.

[JKSH]

 

It's been a long time in coming, but I have completed a new version of the HDF5 Toolkit for LabVIEW that was first released in 2006. The aptly named LVHDF5 Toolkit (version 1.0) is a hefty redesign of the original toolkit (version 0.9) and should solve most of the performance issues with that library while maintaining the flexibility of the the original design.

 

...

 

In particular, I'd like to thank the members of LAVA who worked on xnodes. I've been lurking on the xnode-related pages pages for the past three or four years while slowly building up this revamped toolkit. It would not have been possible without their efforts.

 

Congratulations on your milestone! Could you tell us more about your toolkit? Your website says, "Several attempts at LabVIEW interfaces to HDF5 have been produced over the years. LVHDF5 is the most complete and flexible interface available". What advantages does it offer over the other attempts, like h5labview? (I notice also that LVHDF5 cannot be co-installed with h5labview) 

 

 

I've never heard of the file format but it does sound interesting.  Without knowing much I'd still think TDMS is the better approach if you are going to stay in the NI world.  There are of course features of HDF5 that aren't in TDMS, I just don't know if I would need them.

 

I work for a systems integrator, so we often need to interface with the non-NI world. HDF5 is a popular format in the scientific community.  In our current project, we had to interface with the client's data logger, which provided us with HDF5 files. Our job was made much easier thanks to 3rd-party LabVIEW bindings.

[PiDi]

hooovaah,

I've looked at this code and poly-VIs looked like obvious solution for that kind of code for me. But that doesn't mean that my solution would be better

 

You're right - I've never developed any XNode. And that's why I wanted to understand, why this was the developers choice for this toolkit, why was it better to use them here. Your post gives me full answer to this, so thank you for that - and I'll leave XNodes themselves for other occasion, as I also don't want to make too much offtopic here.

[jdsommer]

@hooovahh: Yes, h5labview is similar. Version 0.9 of my toolkit which did not use Xnodes, but some horrid flattening of variants was started long before h5labview, and appears to have been the inspiration for Martijn. While I should have been aware of h5labview, I was not, and worked independently. Naturally, I'm biased, but my toolkit provides nearly complete access to the library and a number of useful utilities besides, so I think it's better. On the other hand, h5labview works with 64-bit LabVIEW, a failing I'd like to remedy in my toolkit. However, I don't currently have access to the tools (like, 64-bit LabVIEW, for instance ).

 

Regarding TDMS, I honestly don't know a lot about it, but I think you hit the nail on the head with your phrase, "it's better if you're going to stay in the NI world." You might not. My general impression is that HDF5 is much more flexible, but the flexibility comes at a bit of a performance hit in terms of streaming data. Because of the interoperability and open nature of the HDF5, I would choose it over TDMS unless I needed absolutely needed the speed. Of course, I'm also biased, having put 8 years (on and off again) into the toolkit.

 

@PiDi: Xnodes over polymorphic. It really is not possible to use polymorphic VIs in this case. The library can handle nearly any datatype that you throw at it. Particularly when you start enumerating cluster types (was it a cluster of two doubles and an integer, or one double and two ints, or something else?) it is absolutely impossible. As the toolkit developer, I can't predict ahead of time every datatype that a user might try to shove into an HDF5 file.

 

For the most part, the polys are used simply as means to pass any data type through to the DLL which does the main work. I would have chosen another path besides Xnodes, but it was really the only option available.

 

@JKSH: I've looked at h5labview's code and, while it is generally similar, I believe that my code is more likely to work with a wide range of datatypes. I have particular concern about its ability to handle strings inside of clusters--a corner case, to be sure, but one which my toolkit does handle. I'm also unclear on h5labview's ability to read and write complex selections. 

 

However, the primary difference between the two toolkits is the completeness of the interface, as I mentioned above. There may be a few functions that have been added since HDF5 1.6, that I haven't implemented, but nearly every library function that can be called is exported. Also, I provide a number of higher level tools that are not included in h5labview, such as the ability to read and write a dataset and all attributes in one VI which I find to be useful.

 

All that said, I have not really given h5labview a fair shake, and I don't want to disparage Martijn's work. It requires a deep level of knowledge of LabVIEW datatypes to make anything like this work at all. The fact of the matter is, since I'm going to be biased towards my own toolkit anyway so I'd prefer to have someone else do the comparing and contrasting.

 

As a final note, the reason that the two cannot be installed simultaneously is that we both have chosen some obvious names like "H5Dread.xnode." If your calling VI loads the xnode from the wrong toolkit, then all kinds of broken wires ensue.  LabVIEW libraries (.lvlib) were supposed to handle make this better by providing a namespace, but, correct me if I'm wrong, xnodes already are a LabVIEW library and, thus, cannot be put into one.

Edited September 17, 2014 by jdsommer

[Yair]

I saw some amazing things done with scripting in the 7.x era before NI officially supported that as well.

 

I think that the difference is that scripting never really had the problems that XNodes have. It had some holes and things that didn't work, but it was basically complete and stable.

 

My understanding is that XNodes have some serious problems which NI plans on addressing by replacing them with something else. This understanding does not come from any official NI line or from a deep understanding of XNodes (I actually have little more than a cursory understanding of their actual structure which comes from playing with them a bit), but it does come from NI and it is up to date. I can't estimate how representative it is of what will actually happen, but it's not something that I made up.

 

Something like this has happened before with external nodes, which were the 7.x equivalent of XNodes. They worked, and they did the same things as XNodes, but I guess they had issues which were too big, so NI replaced them with XNodes, and apparently it is their intention to do so again.

 

All this does not mean that you should not use XNodes. They certainly can be written to function correctly. You should just take into account that even if you write stable XNodes, you might have to rewrite them in a few versions, because they will no longer be supported. Maybe NI will create some kind of migration option. Maybe not. Maybe they'll create a tool to assist with migration and publish it without support. Maybe not.

As a final note, the reason that the two cannot be installed simultaneously is that we both have chosen some obvious names like "H5Dread.xnode."

 

...correct me if I'm wrong, xnodes already are a LabVIEW library and, thus, cannot be put into one.

 

Libraries can be nested freely (although I wouldn't be shocked if this caused some instabilities to appear) and I believe the only exception is that classes (and maybe .lvsc too?) cannot contain other libraries. I haven't checked with XNodes, but I believe they can at least be added to libraries.

[hooovahh]

I haven't checked with XNodes, but I believe they can at least be added to libraries.

They can be added to a library, but XNodes themselves are a library.  So this makes the VI name space very long "MyLibraryClassName::MyXNodeLibraryName::MyVIThatDoesStuff.vi"

 

Also I don't know if it was me but I got into a situation where I would open the library which would load the XNode into memory, which would lock the XNode, preventing me from editing it.  Not sure if it was my own fault but my solution was to just not put an XNode in a library.

[gb119]

They can be added to a library, but XNodes themselves are a library.  So this makes the VI name space very long "MyLibraryClassName::MyXNodeLibraryName::MyVIThatDoesStuff.vi"

 

Also I don't know if it was me but I got into a situation where I would open the library which would load the XNode into memory, which would lock the XNode, preventing me from editing it.  Not sure if it was my own fault but my solution was to just not put an XNode in a library.

There used to be a similar problem with XControls in classes (and I guess libraries) - loading the library would load the facade which would lock the library which would stop you removing te XControl from the library. NI eventually made it impossible to add XControls in ways that would deadlock the project like this - but I guess as XNodes don't exist there's no checks on putting them into libraries and classes :-)....

[jdsommer]

It's sounding like I should try to put the xnodes into a library, but go cautiously. Of course, that's probably good advice any time one is dealing with xnodes! In any case, I'll consider it for the next revision. In the meantime folks will just have to uninstall h5labview to use my toolkit.

[hooovahh]

It's sounding like I should try to put the xnodes into a library, but go cautiously.

Good luck.  That's one thing I spent a couple days playing with.  Several times I thought I had it, and maybe I would have if I kept trying but in the end it wasn't worth the hassle.  Now I see a need for trying to protect my data and have my XNode be in a class so I may re-investigate this later.  If XNodes aren't complete for a reason, it is probably going to be edge cases that aren't complete, and I see XNodes in classes and libraries being one of them.

[JKSH]

@jdsommer: Thank you for your detailed and open explanations. Our current project (which doesn't need a wide range of data types)  has been implemented using h5labview, so it looks like I won't be able to try out your toolkit for now. I do agree that high-level convenience APIs make a noticeable difference to a programmer's job.

[jdsommer]

@JKSH: I've rarely found that any one project requires a wide array of datatypes. However, I have found that very few projects seem to require exactly the same datatype as a previous one. This is in part because I generally find the most natural representation of data that I acquire and process an array of clusters, (which translates to datasets of compound element types in HDF5). No two cluster types are usually the same. When I set out to build this toolkit (actually, the original version 0.9) I wanted to make a toolkit that would work for any project.

 

In any case, if you get a chance to try out the toolkit, even just for fun, I'd appreciate any feedback you might have.