XML for data

[ASTDan]

Hello,

I am thinking about saving the data I generate in XML. In the past I have used a tab delimited or comma delimited file.

What I like about this idea is the following

-Open file format. This makes the file readable by LabVIEW, Excel, Matlab, Diadem, etc

-I can add tags and attributes to data

-I like TDMS, however XML is an open file format. This appeals to me more because moving forward any kind of data viewer should support XML. For example my customer likes Matlab. Using an open file format that can be easily read without finding, installing, and maintaining converters is very appealing. I am not really concerned with file size or speed.

The JKI Easy XML toolkit looks like the way to go to implement this.

My main concern about implementing this is my customer's unfamiliarity with XML. Has anyone used XML to save data? Any recommendations, tips, pitfalls, etc. in using this file format.

Thanks

Dan

[crelf]

I am thinking about saving the data I generate in XML. In the past I have used a tab delimited or comma delimited file.

What I like about this idea is the following

-Open file format. This makes the file readable by LabVIEW, Excel, Matlab, Diadem, etc

-I can add tags and attributes to data

-I like TDMS, however XML is an open file format. This appeals to me more because moving forward any kind of data viewer should support XML. For example my customer likes Matlab. Using an open file format that can be easily read without finding, installing, and maintaining converters is very appealing. I am not really concerned with file size or speed.

We use ATML - it's XML formatted in a particular way for test, and sounds like it'll do everything you're after (and TestStand writes it natively). I've been meaning to write a LabVIEW module to write ATML for a while now...

[ASTDan]

ATML looks interesting.

Can it hold an array of data for analysis?

How portable is it? Can it be used in a non NI reader? i.e. not diadem

I have been playing around with XML files and excel. Very cool!

Edited September 8, 2009 by ASTDan

[PJM_labview]

XML is a very useful format to share data between application. Additionally if you ever need to edit it, you can use Microsoft XML Notepad. This is a very nice free tool that does show the XML in a hierarchical tree control. So if your customers are a bit "scare" of this new file format, you can recommend XML Notepad as a viewer / editor.

PJM

[crelf]

ATML looks interesting. Can it hold an array of data for analysis?

Yes.

How portable is it? Can it be used in a non NI reader? i.e. not diadem

Is as portable, if not more portable, than raw XML, because it is XML but with standard defined tags.

[PaulL]

We use XML extensively here. On the other hand, TDM files are readable with any of the tools you mention. They are ideal for LabVIEW and DIAdem, of course, but there are plug-ins for Excel and MATLAB (Integrating TDMS in Third-Party Products) and the TDM format is publicly available, I'm pretty sure.

We use XML largely to share data with other applications. There are a couple challenges:

1) No complete XML tool exists for LabVIEW yet to my knowledge, although some tools offer a very good start. We use JKI's EasyXML and like it. A complete tool would handle LabVIEW objects, for instance, and I'm not sure that a truly appropriate solution is possible without having access to the LabVIEW internals. My opinion is that NI should release a suitable XML framework, but I have no reason to believe that NI is working on such a thing. (Yes, there is the LabVIEW internal XML definition, but that is not useful for sharing data with nonLabVIEW applications. In particular, if a LabVIEW object has default data the XML representation just indicates this rather than detailing the data.)

2) An issue we have encountered recently is handling units. If a control has units in say, degrees, the value on the wire is always in SI units (radians in this example), and this can create issues when writing to XML. For instance, EasyXML just writes the value on the wire to the XML output and ignores the unit (quite reasonably) but this means the value is the value in radians, not degrees. (We are developing a work-around here, but there are some pitfalls just because of the way LabVIEW handles unit information. It's more trouble than you might think.) Again, I think a "complete" XML framework for LabVIEW would handle units correctly. (This ideally would probably mean writing units and dimensions as attributes, but even writing the value in the desired unit would be a great start!)

Nonetheless, XML is a great way to share structured data between applications. Its universality makes it very appealing (and hence why I would like to see a complete implementation in LabVIEW).

Paul

[crelf]

2) An issue we have encountered recently is handling units. If a control has units in say, degrees, the value on the wire is always in SI units (radians in this example), and this can create issues when writing to XML. For instance, EasyXML just writes the value on the wire to the XML output and ignores the unit (quite reasonably) but this means the value is the value in radians, not degrees. (We are developing a work-around here, but there are some pitfalls just because of the way LabVIEW handles unit information. It's more trouble than you might think.) Again, I think a "complete" XML framework for LabVIEW would handle units correctly. (This ideally would probably mean writing units and dimensions as attributes, but even writing the value in the desired unit would be a great start!)

Since you're designing a framework, it might be a good idea to write the units in the same standard that ATML does...

[Phillip Brooks]

Some time ago I was trying to validate an ATML string; specifically units.

The regex looked like this:

(y|z|a|f|p|n|µ|u|m|c|d|h|k|Ki|Mi?|Gi?|Ti?|Pi?|Ei?|Zi?|Yi?)?(a|A|Ah|atm|at|B|BW|Bm|BV|BuV|bar|bbl|b|Bd|Bq|Btu_th|Btu|cal_th|cd|eV|C|Ci|°|d|deg|degC|°C|degF|°F|°R|degR|dyn|erg|ft|ftHg|ftH2O|F|fc|fL|g|G[bi]?|Gal|gal_Imp|gal_US|gf|gr|Gy|ha|hp|Hz|in|inHg|inH2O|J|kat|K|kn|lb|lbf|lm|L|lx|mHg|mho|min|mol

|'|Mx|mo|Nm?|Np|o|Oe|Ohm|oz|P|Pa|pc|%|ph|pt|pdl|ppm|qt|r|R|rad|rem|s|S|sb|sec|slug|sr|St|Sv|t|T|thm|ton|t|u|ua|V|var|VA|W|h|m|Wb|Wh|yd|H)

The original LAVA post is a bit munged, but you get the idea of the problem. The ATML working group worked hard to support any conceivable units.

My code never got used, and I think it wouldn't match the schemas as currently released. That's the biggest problem with doing XML in LabVIEW, adapting to the schema...

[PaulL]

Since you're designing a framework, it might be a good idea to write the units in the same standard that ATML does...

First, thanks for the ATML information. I hadn't heard of that before. I just started to look at it now.

What I was thinking about in units came from this: LabVIEW and XML.

I certainly agree that this should be standards-based!

On that note (and departing a bit from the conversation) I would really like to see a reusable, standards-based, XML parser for LabVIEW. For instance, I was talking to an AE about an issue a couple weeks ago and he commented that he writes his own XML parsing code when he needs it. While the basics of this are not particularly difficult, getting everything correct does take some time. Ideally LabVIEW developers would have a reusable, common, library we can all share, rather than each reimplementing something (with lots of variations!) ourselves. Ideally we can spend time working on code relevant to our project domain rather than on infrastructure. (That's the dream!)

This is one reason why I separately advocate using a commonly available publish-subscribe framework such as shared variables. While I don't want to ignore existing issues with SVs, I am not fond of the frequent suggestion to posters to write a communications framework using TCP/IP primitives. The latter solution is not particularly difficult, but it does require a finite amount of time and the resulting infrastructure is quite customized. Where this is necessary, this is great, but I think most often SVs fit the bill (and they have features one is not likely to take the time to develop oneself!) and they are something that is common at least within LabVIEW--and NI will presumably do the work of maintaining the infrastructure into the future. Again, I want to focus on code in my application domain. (If I am trying to sell infrastructure tools, that is another story.)

I'd really like to see XML become something that LabVIEW developers just use, rather than develop. I'm not sure anyone besides NI can create a framework to handle objects currently, so I'd really like to see NI do this.

Paul

[jzoller]

I've built a couple XML function sets. It's a good-and-bad technology.

Positives:

->Everything mentioned above.

->I love XPath support.

Negatives (which may not affect you at all):

->Everything mentioned above.

->File size bloat: there's a lot of overhead in most XML tags.

->Memory: Most (all?) libraries with LV use the DOM system, which means the entire file needs to be put in memory. This can be prohibitive if you're storing a lot of data.

->Brittle structure: simple changes to the XML schema can really mess with your code. Using a predefined standard like ATML may avoid this.

->Validaton: If you decide that you need validation, it can require a painful amount of work to get things just right.

->External libraries: most XML writing and parsing is going to require dragging along some non-LV library (ActiveX, LibXML, .NET, etc).

->Possible memory leaks: many XML libraries require a lot of references to be opened, and it's not always straightforward when they should be closed.

->Searching is relatively slow: Again, in order to search, you need to read in the file, then traverse it. This is much slower than doing something like a fixed offset column search.

Joe Z.

Edited September 8, 2009 by jzoller