TDMS for big config clusters -- is there a better way?

[Kevin P]

Ok, so I'm starting to get frustrated. I'm getting down to crunch time on a big project that includes some pretty high-speed data streaming. There's been lots of NI rah rah promotion of TDMS and I decided to buy in. And it's been just fine *FOR ACQ DATA*. But I'm having a heck of a time trying to also jam a bunch of my big configuration clusters (typedef'ed clusters containing typedef'ed clusters, enums, etc.) into any of the TDMS properties.

I spent the morning trying to "trick" the TDMS Set Properties function. I tried converting my config cluster(s) to Variants, I tried flattening to string, I tried typecasting to string, I tried flattening to string then typecasting to U8 array, and so on. Some of these things claimed to write successfully but of those, NONE would read back successfully -- I would get an error about data corruption.

I'm very close to bagging the whole attempt and living with the uglier solution of storing acq data and config info in 2 separate files. Has anyone out there found a reasonable way to store complex data structures in a format that TDMS will accept?

Note: I don't want to have to unbundle every single element of the clusters manually as that will be a maintenance nightmare. Also, anything involving LVOOP wouldn't be practical now -- I don't have time for the needed learning curve. Thanks!

-Kevin P.

[Herbert]

Kevin,

TDMS properties are reserved for "human-readable" strings. The reason is that they need to play well with other software accessing them, in particular DIAdem and the NI DataFinder.

If you want to store cluster information, I recommend flattening it into a U8 array and storing it as a channel. You can also store the data type as an array of I16 to a separate channel in order to handle changes in your cluster data type. See the attached screen shots for how to do that.

Hope that helps,

Herbert

Write cluster type and value to TDMS channels:

http://forums.lavag.org/index.php?act=attach&type=post&id=5503

Read cluster type and value from TDMS channels:

http://forums.lavag.org/index.php?act=attach&type=post&id=5504

[Kevin P]

Herbert,

Thanks for the quick reply. I can see how that can work if I write exactly one time. A couple followup questions:

1. Please tell me if the following is correct: When using TDMS properties, each Write operations *overwrites* old data related to that named property. When using TDMS channels, each Write operation *appends* to old data related to that named channel.

Until now, I intended to make use of the (expected) *overwrite* behavior of properties. I planned to write the default clusters prior to starting the test. Then during the test, when certain fields of some clusters needed to be changed due to operator choices or other test conditions, I could just (over)write the cluster to the TDMS file on each change. There are several nearly independent processes running, and I was basically planning to use the TDMS properties as a storage mechanism like a functional global.

It would appear that if I need to use TDMS channels for this data, I'd better use standard functional globals and then be sure to perform one and only one TDMS write at test completion.

2. Just curious. I don't *intend* to change the cluster datatypes, but you suggested that you were illustrating a way to handle such changes. Can you explain more -- I'm not seeing it.

I'm imagining the most likely change where some fields are added or removed from the cluster. The TDMS file was written with the old cluster and I'm now reading it with code based on a new cluster, containing 2 extra data fields. Won't the "Unflatten from String" function fail because the data string based on the old cluster now has the wrong length to be unflattened into the new cluster?

Or did you mean something different?

Thanks again! I've got a workable path forward, and understand better why the "properties" were restricted to human-friendly strings.

-Kevin P.

[viSci]

I had a similiar problem and constructed a mechansim using variant attributes to automatically convert cluster variables to named TDMS properties.

Here is my example vi.

Mike Sachs

Intelligent Systems

[PJM_labview]

Modifying the OpenG Variant Configuration Tools works like a charm. I quickly made a writer based on the write ini cluster.

PJM

VariantDataToTDMS.llb

[Herbert]

QUOTE(Kevin P @ Apr 13 2007, 02:58 PM)

1. Please tell me if the following is correct: When using TDMS properties, each Write operations *overwrites* old data related to that named property. When using TDMS channels, each Write operation *appends* to old data related to that named channel.

[...]

It would appear that if I need to use TDMS channels for this data, I'd better use standard functional globals and then be sure to perform one and only one TDMS write at test completion.

That is correct. We do not yet have a way of deleting objects or overwriting channel values in TDMS files. Using a variable sounds like a good solution.

QUOTE(Kevin P @ Apr 13 2007, 02:58 PM)

2. Just curious. I don't *intend* to change the cluster datatypes, but you suggested that you were illustrating a way to handle such changes. Can you explain more -- I'm not seeing it.

I'm imagining the most likely change where some fields are added or removed from the cluster. The TDMS file was written with the old cluster and I'm now reading it with code based on a new cluster, containing 2 extra data fields. Won't the "Unflatten from String" function fail because the data string based on the old cluster now has the wrong length to be unflattened into the new cluster?

Or did you mean something different?

You're perfectly right. The idea here is to store a piece of information to the file that can tell you which version of your cluster the file contains. I guess I went a little overboard with this when I stored the complete type descriptor to a channel. You can instead store a version number for your cluster to the file (e.g. as a property of the channel that contains the flattened cluster data). When you read the file, you can then use a case structure that has a case for all cluster versions you have used over time. In each of the cases you unflatten using the appropriate cluster type and convert to the newest version.

A crude, but working solution is to always try unflattening with the newest version of your cluster, then trying the next older version if "unflatten" returns an error, and so forth, until you succeed. That way, you don't even need a version info in the file.

Herbert

[Herbert]

QUOTE(PJM_labview @ Apr 13 2007, 04:42 PM)

Modifying the OpenG Variant Configuration Tools works like a charm. I quickly made a writer based on the write ini cluster.

Very cool. :thumbup:

Herbert

[Kevin P]

Thanks all!

Herbert: Ok, I understand. I'm pretty sure I'm going with the simple flattened U8 array as a TDMS channel in the short term.

Mike S: You gave me a little something to think about there. I've been studiously avoiding the waveform datatype for years. Most of my data acq has been in the handfuls of kHz and I didn't find the waveform datatype useful. The timestamp t0 was never nearly as precise as the sampling rate and didn't seem worth the bother. Not to mention that it could be very misleading for delayed-trigger acquisitions, marking t0 when you started the task rather than when the trigger came in. I also thought I'd read about inefficiencies in memory and CPU due to its cluster-like nature.

However, I see that you were able to tuck in some useful "properties" as waveform attributes, which would be another way to get such info cleanly into a TDMS file. In my particular case, the clusters don't readily map to any true waveform data that they could "piggyback" onto, so this time I'll probably go with the brute-force method Herbert showed.

PJM: That looks like the best solution of all. In the end, the info actually ends up as viewable TDMS properties! Hurray! Er, that is, hurray... if only I did OpenG. (Anyone remember the old Schoolhouse Rock "hurray, I'm for the other team...")

At work, only computers with standard corporate images get to ride on the network. LabVIEW goes on lab machines without any network connections. I could download the VIPM on my network PC, but don't have permissions to perform installs. I could install on my lab machine, but then can't see the internet to get the OpenG packages. And honestly, the old licensing requirements looked kinda painful. Our test config management favors big monolithic unchangeable executables.

It appears that the recent licensing changes are meant to ease much of that burden, though I've only followed Jim K's announcements superficially and don't know details about which packages have ported over and which haven't. There's probably some kind of workaround for OpenG on non-networked PC's, and now that the licensing is changing, it's probably time to give OpenG some serious consideration again. I've *wanted* to use it, I just didn't want it to have to hurt.

Thanks again to all responders.

-Kevin P.

[Herbert]

Kevin,

the cool thing about the OpenG-based solution is that it is ready for cluster structures of arbitrary complexity. If your cluster isn't too complex, you might get away with something more straightforward. Of course, none of these solutions is as straightforward as the U8 array. I'm adding this more for further reference than in order to talk you out of the U8 solution.

Examples:

If your cluster contains no arrays, clusters or containers of any sort, simply loop over the "controls" array. Passing a variant into "TDMS Set Properties" will create a property of the same data type that the control has (if supported).

http://forums.lavag.org/index.php?act=attach&type=post&id=5515

If you are using nested clusters, you can use the example VI "Queue Control Reference Info" as boilerplate code, so you end up with something like this:

http://forums.lavag.org/index.php?act=attach&type=post&id=5516 http://forums.lavag.org/index.php?act=attach&type=post&id=5517

For arrays, you would need to add a case similar to the above "30" case. And you might need a reading VI. Plus, the error handling could use some improvements .

Herbert

[PJM_labview]

I made a reader as well based again on the OpenG Variant Configuration Tools.

http://forums.lavag.org/index.php?act=attach&type=post&id=5518

PJM

Note: The attached llb does not required that the OpenG tools to be installed.

http://forums.lavag.org/index.php?act=attach&type=post&id=5519

[Grampa_of_Oliva_n_Eden]

QUOTE(PJM_labview @ Apr 14 2007, 03:18 PM)

I made a reader as well based again on the OpenG Variant Configuration Tools.

http://forums.lavag.org/index.php?act=attach&type=post&id=5518

PJM

Note: The attached llb does not required that the OpenG tools to be installed.

http://forums.lavag.org/index.php?act=attach&type=post&id=5519

PJM,

I tested this with LV 8.2 and the booleans do not seem to be working.

Ben

[bbean]

QUOTE(Ben @ Apr 20 2007, 08:45 AM)

PJM,

I tested this with LV 8.2 and the booleans do not seem to be working.

Ben

Seems like the boolean data type is not supported directly by TDMS. I modified the Read TDMS Key (Variant)__TDMS.vi to use a string type and it seems to work. It was just a quick fix so I don't know if other issues exist.

http://forums.lavag.org/index.php?act=attach&type=post&id=5562''>http://forums.lavag.org/index.php?act=attach&type=post&id=5562'>http://forums.lavag.org/index.php?act=attach&type=post&id=5562

[Grampa_of_Oliva_n_Eden]

QUOTE(bbean @ Apr 20 2007, 10:46 AM)

Seems like the boolean data type is not supported directly by TDMS. I modified the Read TDMS Key (Variant)__TDMS.vi to use a string type and it seems to work. It was just a quick fix so I don't know if other issues exist.

http://forums.lavag.org/index.php?act=attach&type=post&id=5562''>http://forums.lavag.org/index.php?act=attach&type=post&id=5562'>http://forums.lavag.org/index.php?act=attach&type=post&id=5562

I wonder why I did not get any errors?

Yes, I know I can figure this out myself, but that is not is much fun as "Gang appliance re-wiring".

Ben

[PJM_labview]

Good catch.

Thanks for the bug fix too.

By the way, there is also a bug in TDMS Get Property when type input is a complex number (CXT Type).

PJM

[Herbert]

QUOTE(bbean @ Apr 20 2007, 08:46 AM)

Seems like the boolean data type is not supported directly by TDMS.

This is a bug. Booleans are supported, but that's not working correctly. CAR# 48B7CBWJ.

Herbert

[Herbert]

I'm picking up the discussion from here, because we obviously agree that this is the thread to talk about it.

QUOTE(Ben @ Apr 23 2007, 07:27 AM)

If the data is stored as "native datatype (dbl, u32, etc) " DIAdem can do its magic without any help, correct?

Correct. It doesn't really matter if you just display properties in a report. But if you use the values to calculate something (e.g. scale a channel), or you require a particular number format, using numeric properties saves you some conversions.

Storing properties to their native data type is generally a good practice, even if you work in LabVIEW only. For the cluster use case it is obviously less important, because we already know which data type to expect.

Herbert

[Grampa_of_Oliva_n_Eden]

QUOTE(Herbert @ Apr 23 2007, 03:30 PM)

... For the cluster use case it is obviously less important, because we already know which data type to expect.

Herbert

Unless the contents of the cluster are subject to change over the life of the application.

Thank you Herbert.

Ben

[PJM_labview]

Attached is a new version (1.2) of the utiliy that read write variant data to TDMS file.

Change Log:

- Fix bug when numeric were written to TDMS. Now evrything is read back as string then converted to the appropriate type.

Read Write Variant Data to TDMS V1.2: http://forums.lavag.org/index.php?act=attach&type=post&id=5610

QUOTE(bbean @ Apr 23 2007, 07:14 AM)

...As an extra bonus it also has a new example that saves clusters to specific group/channel. This is nice because you can place your cluster properties at any level in the TDMS format. Let me know what you guys think...

Brian,

The group/channel example is interesting. The only draw back is that it increase the complexity quite a bit, but I can see use cases where I would want to do that.

PJM

[bbean]

QUOTE(PJM_labview @ Apr 23 2007, 05:01 PM)

Brian,

The group/channel example is interesting. The only draw back is that it increase the complexity quite a bit, but I can see use cases where I would want to do that.

PJM

I have already found a need for it. When I store multiple traces (channels) of data with peak detection, I store the cursor list cluster (peaks) to the channel's property. Then I can pull up all the channels and the peaks recorded during data acquisition and plop them on the graph pretty easily using the "TDMS File Viewer.vi" and the TDMS file. Its nice because it keeps everything in one file.

What do you mean by more complex? Do you mean you have to specify the channel and group every time you write a cluster? I was thinking the "channel and group" write/read functionality would be additional VIs not replacements for the default option which would be to write/read everything automatically.

Brian

[PJM_labview]

QUOTE(bbean @ Apr 23 2007, 04:03 PM)

...What do you mean by more complex? Do you mean you have to specify the channel and group every time you write a cluster?

Yes

QUOTE(bbean @ Apr 23 2007, 04:03 PM)

I was thinking the "channel and group" write/read functionality would be additional VIs not replacements for the default option which would be to write/read everything automatically.

I noticed you created separate VIs for channel/group. This does expand the API. I am not sure at this time what is best, separate VIs or additional input on the original VI (if possible).

PJM

[bbean]

QUOTE(PJM_labview @ Apr 24 2007, 11:35 AM)

I noticed you created separate VIs for channel/group. This does expand the API. I am not sure at this time what is best, separate VIs or additional input on the original VI (if possible).

PJM

I think it would work best by removing the "float number format" and adding three inputs on the original VIs.:

1) Group (string - optional): for the TDMS group. If blank chooses how to write based on 3).

2) Channel (string - optional): for the TDMS channel. If blank writes to the Group specified

3) Use cluster name for Group if group/channel blank (boolean - default true). If you set this to false and left channel and group blank, the VIs would read/write to the root of the TDMS file

I believe you can remove the float number format since the number type is stored natively in the TDMS file (except the complex at this point).

[PJM_labview]

QUOTE(bbean @ Apr 24 2007, 08:51 AM)

...

I believe you can remove the float number format since the number type is stored natively in the TDMS file (except the complex at this point).

Brian

Currently, the float number format is beeing used for writing numbers. The types are not store natively.

PJM

[bbean]

QUOTE(PJM_labview @ Apr 26 2007, 02:56 PM)

Brian

Currently, the float number format is beeing used for writing numbers. The types are not store natively.

PJM

I guess I'm suggesting abandoning storing numbers as strings and store them as their true type. I believe the only reason they are stored as strings in the OpenG is because config file format is a string. I guess I don't see why "float number format" is relevant to TDMS files, because you will probably never use a text editor to look at a TDMS file. If the numbers are stored in their native format, you can always use "format into string" primative to display in the "float number format" of your choice

[PJM_labview]

This is correct for the supported type. Currrently the set TDMS property primitive does not support every type, therefore, converting to string is a good stop gap solution until the time where all types are be supported.

PJM

[Kevin P]

Just had to chime back in with a big hearty thanks to PJM! :thumbup:

I thought I was just gonna take a quick peek at it for future reference, but it works so well I'm using it.

-Kevin P.