Article - Using XML Data in LabVIEW is Hard

[Jim Kring]

Do you find working with XML data in LabVIEW to be difficult? I sure do, and I've written an article about it here:

• Using XML Data in LabVIEW is Hard
  

What XML tools are you using to work with XML data? What tools do you wish you had?

[cross posted]

[John Lokanis]

I read your article. I agree that to the novice, using XML in LabVIEW is not easy. You do need to understand a lot about XML features, like the schema and namespaces. But it is not impossible and can be a very powerful tool for large and complex data driven applications.

Also, the best way to read and write XML from LabVIEW (in my opinion) is to use the MSXML .NET assemblies. I have tried the XML toolkit from NI as well and find it lacking compared to rolling my own with .NET calls.

I am actually working on a presentation to our local LV user group about .net and using it to work with XML files. I use XML extensively in my LV applications, mainly as a universal script format to drive data driven portions of my application.

I have tried many ways of parsing XML within LabVIEW. Given the tree nature of XML, my first method was to use recursion (via VI server) to walk an XML structure and then build a representation in a LabVIEW data structure. The first attempt used an array of clusters with elements to contain the node links and the data at the node. The second version used a Variant tree (via attributes) to construct the tree in the LV data space. This was a more natural representation of the data but was slow to access (due to the overhead of the variant attribute VIs, I suspect). This was before LVOOP. I understand there are better ways to represent tree structure in LV using the new OOP features but I have not explored that yet.

Each of these methods required you to then translate the data into a LV structure (nested arrays of clusters) to make it usable (and the code readable) in the rest of the app.

My current implementation is to parse the schema directly into a LV data structure consisting of arrays of clusters of arrays of clusters...etc... This is the fastest method so far and goes directly to a structure that can be used in the rest of your application. The downside of this approach is the structure of the schema is directly coded into the LV parsing VIs. So, if the schema changes, so must your code. For my applications this is usually not a big issue since we design schemas up front that are flexible for the design goal of the application.

However, this certainly is not a good generic solution for LabVIEW and XML. I think the holy grail here is likely some sort of LVOOP implementation that can dynamically traverse an XML file and then build a representation in memory that is simple and clear to access in the rest of the application.

Another approach is to use the .NET interfaces to interact with the XML file 'live' instead of preloading and parsing it into a LV structure. This may have advantages in some applications, but in my experience I have always wanted the data to be resident in memory and fast to access with standard LV array and cluster tools.

One caveat with using .NET assemblies to access XML is the confounding documentation (or lack thereof) of how to do this. I have spent many an hour staring at MSDN pages trying to understand how to call or what to call to get the result I wanted. Also, there are some pitfalls to watch out for. For example, if you use the MSXML schema validation methods, you cannot simply close the reference to the XML reader when you are done. Instead you must call the CLOSE method in the XMLValidatingReader and the XMLReader before you close their references. If you don't then the OS will keep a file lock on the XML file until LV is closed.

Once I complete my presentation I will try to remember to post the PowerPoint and VIs here for other to view. In the meantime, I would be happy to share my code with anyone who is interested.

I also strongly recommend looking at the w3schools website (http://www.w3schools.com/xml/default.asp) to learn more about XML.

Also, the best tool I have found for editing XML and creating schema is XMLSpy. It is not cheap, but it has paid for itself many times over on my projects.

-John

[Dirk J.]

yup, it's not easy.

I for one have been most comfortable with the MSXML activeX interface.

Nevertheless, I'm currently using a virtual class defining (my most relevent) XML related methods and a couple of child classes which implement these methods.

If I need a very safe-and-simple-backwardscompatible-etc routine, I use my LV-string based childclass which honestly can't do a lot;

most of the time I use the ActiveX implementing childclass; and for some other projects the (presently rudementary) .NET implementing class.

ps: since both the activeX and .Net stuff is "by-reference", I implemented this whole class hierarchy as "by-reference"

[silmaril]

QUOTE(jlokanis @ Jan 22 2008, 06:01 AM)

I think the holy grail here is likely some sort of LVOOP implementation that can dynamically traverse an XML file and then build a representation in memory that is simple and clear to access in the rest of the application.

That's exactly what I would like to have!

Sounds great, but I don't know where to start.

[Eugen Graf]

I use LVs XML Flatten/Unflatten instead of Config File VIs, because it's very easy to use like Flatten/Unflatten binary string, but is readable by user like INI-file.

[LAVA 1.0 Content]

I use XML Flatten/Unflatten because it's fast and easy to implement. But we have to deal with the limitation of this method.

For more sophisticated design, we use MSXML. Then, it could be more difficult depend on your XML knowledge.

[Saverio]

One of the options in the poll is LabXML, which is what I use. It inherently uses the MSXML parser, so I'm not sure which poll answer to choose in this case (went for LabXML).

The earlier comment regarding using .NET is interesting because I actually tried to use it and I found it to be abhorently slow, as compared to LabXML. I essentially duplicated the operations that were being performed in LabXML in reading an XML node, so as not to hard-code the schema, as mentioned in the post. I found that the same operation was an order of magnitude slower with the .NET XML interface.

[jzoller]

Jim, I couldn't agree more.

From my experience, the internet toolkit XML vi's are hugely inconsistent in their interface and the native LV schema is brittle to the point of being useless.

I ended up writing a private API based around the MSXML support for XPATH queries, but even it suffers from versioning problems on different machines. I will admit, I didn't investigate the LabXML interface too closely because of its dependency on yet another library (Gnome's libXML2).

Joe Z.

[gb119]

QUOTE(Saverio @ Jan 22 2008, 03:52 PM)

One of the options in the poll is LabXML, which is what I use. It inherently uses the MSXML parser, so I'm not sure which poll answer to choose in this case (went for LabXML).

Ummm, I thought the later versions of LabXML used libxml2 - the open source XML parser ? The earlier versions did use the MS XML parser though. It doesn't seem to have had active development for a couple of years - somehow I don't think that's because it's perfect already

[semmel]

I went down the "cluster of array of cluster of array of ... ... string" approach. It worked, but it was cumbersome.

[mgunning]

For writing XML files I wrote my own simple Flatten-to-XML VI, which works in a similar way to the OpenG Variant Config File tools. In fact, that is where I got the idea from.

It involves converting a cluster of strings (or cluster of cluster of strings, or array of cluster of strings) to a variant, which is fed to the VI, which then converts it to an XML string. It automatically converts the string names to the XML Tag names, and the string values to the XML element values.

Cheers,

Mark.

[Jim Kring]

QUOTE(mgunning @ Jan 22 2008, 05:07 PM)

For writing XML files I wrote my own simple Flatten-to-XML VI, which works in a similar way to the OpenG Variant Config File tools. In fact, that is where I got the idea from.

It involves converting a cluster of strings (or cluster of cluster of strings, or array of cluster of strings) to a variant, which is fed to the VI, which then converts it to an XML string. It automatically converts the string names to the XML Tag names, and the string values to the XML element values.

Cheers,

Mark.

Bingo.

[Dirk J.]

QUOTE(Jim Kring @ Jan 23 2008, 02:31 AM)

Bingo.

maybe, but reading from xml is still not trivial.

and yes, the .NET thing is whoofully slow.

[Jim Kring]

QUOTE(Dirk J. @ Jan 23 2008, 01:23 AM)

maybe, but reading from xml is still not trivial.

and yes, the .NET thing is whoofully slow.

Right, it's not trivial. There are a lot of things that can be done to make it much easier. I don't want to let the cat out of the bag, yet, but I've got some ideas that i'll be sharing, soon.

[Phil Duncan]

QUOTE(Jim Kring @ Jan 24 2008, 09:44 AM)

Right, it's not trivial. There are a lot of things that can be done to make it much easier. I don't want to let the cat out of the bag, yet, but I've got some ideas that i'll be sharing, soon.

Great Post Jim, I agree completely. I have just about completed a project that uses the native LabVIEW XML schema and found that whilst development time is reduced, the resulting xml data is not very readable by people not familar with LabVIEW terminology. And yes the structure can get quite cumbersome eg. <Cluster><Array><Cluster>.. blah blah</Cluster></Array><Cluster> etc.

I am certainly going to investigate the LabXML and/or MSXML options for future projects. Of course I am eagerly awaiting the release of the cat you mentioned !!

Incidently, I did run into an annoying inconsistancy (bug) converting class data to XML which I will post shortly.

Cheers & Beers

:thumbup: :beer:

[Saverio]

QUOTE(Gavin Burnell @ Jan 22 2008, 03:19 PM)

Ummm, I thought the later versions of LabXML used libxml2 - the open source XML parser ? The earlier versions did use the MS XML parser though. It doesn't seem to have had active development for a couple of years - somehow I don't think that's because it's perfect already

There's actually two versions available. One that uses libxml and the other that uses the MS XML parser. I happen to use the one that uses the MS XML parser.

[Dirk J.]

QUOTE(Jim Kring @ Jan 24 2008, 01:14 AM)

I don't want to let the cat out of the bag, yet, but I've got some ideas that i'll be sharing, soon.

Aha! A teaser!

[John Lokanis]

There are many things about .NET in LabVIEW that are slow. Hopefully NI will put some effort into fixing this some day. But, there are work arounds. If you go look at Brian Tyler's old blog when he was at NI, there are a few good tricks for moving large data sets between .NET and LabVIEW. I have used these tricks to make a SQL interface via .NET that is 10x faster than the NI toolkit. Unfortunatly it does require writing a little C# code (but just a little ).

Maybe if we all leaned on NI to improve their .NET overhead, things would get better.

[Val Brown]

QUOTE(jlokanis @ Jan 24 2008, 10:39 AM)

There are many things about .NET in LabVIEW that are slow. Hopefully NI will put some effort into fixing this some day. But, there are work arounds. If you go look at Brian Tyler's old blog when he was at NI, there are a few good tricks for moving large data sets between .NET and LabVIEW. I have used these tricks to make a SQL interface via .NET that is 10x faster than the NI toolkit. Unfortunatly it does require writing a little C# code (but just a little ).

Maybe if we all leaned on NI to improve their .NET overhead, things would get better.

or perhaps we could lean on NI to improve the DCT. That way we could all use the SAME interface that was directly from NI instead of having twenty different ways to implement the same functionality.

I presonally hope to NEVER use .NET, C# or C++ in my project. Pure LV makes my life much, much simpler. But, then again, I am focussed on one major project and supporting it as an individual developer so my needs are likely different from others on this forum.

[Mark Yedinak]

Personally, I have never been a huge fan of XML to begin with, at least for anything a human will be dealing with. XML is not very readable for humans without some other tool. I prefer generic ini file formats for configuration data.

With that said, I do need to work with XML since the product I am working with returns data via XML. I found the fastest method for working with this was to write my own simple parser. I mainly need to extract values out of the XML data and a simplified text parser was easier and faster than the XML tools available with LabVIEW.

[crelf]

QUOTE(Jim Kring @ Jan 23 2008, 07:14 PM)

...I've got some ideas that i'll be sharing, soon.

Well?

[Jim Kring]

QUOTE(crelf @ Feb 17 2008, 05:34 AM)

Well?

I've just announced a new article on my blog with my ideas. I hope it meets your expectations after keeping you in suspense all this time

Cheers,

[robijn]

XML was meant to be easy and human readable. Both those goals were missed.

I don't use it but use text files instead.

Joris

[Yair]

QUOTE(robijn @ Mar 7 2008, 10:53 AM)

XML was meant to be easy and human readable. Both those goals were missed.

Actually, the term was "http://www.w3.org/TR/REC-xml/#sec-origin-goals' target="_blank">human-legible and reasonably clear". XML itself is basically ASCII, so it's legible and even somewhat readable as long as you have proper indentation. That's not different from normal writing, where it's hard to read if you don't have paragraphs. It may not be easy to parse in your brain if to get to each element of a group you need to view several XML elements, but I don't think that was one of the design goals.

As for whether it's easy to process programmatically, I never tried writing a full parser, so I can't comment, but I suppose it also depends on which tool you're using. In LabVIEW, at least so far, I agree that it hasn't been a very useful option.