TDMS -- treating clusters as channels?

[Kevin P]

Ok, more growing pains with TDMS. This time, I'm looking for advice on a strategy to use for storing a cluster of info as a channel. PJM's help in this thread was terrific to help me put single-valued clusters of config info into TDMS. However, I dug myself a hole with timestamps and daylight savings time that I recently discovered.

My specific app queues up asynchronous "Events" as a cluster of info, including a timestamp, a typedef'ed enum for event ID, a couple strings and a couple numerics (DBL and I32). Since there are multiple instances of Events during the app, I need to store this info as a TDMS Channel. My misstep was to convert the cluster to an array of strings and treating the string array as a channel. Unfortunately, I converted the timestamp to string in a way that makes it disagree by 1 hour (daylight savings) with timestamps saved elsewhere as a raw timestamp datatype.

So one workaround that can solve my problem today is to separate the events into 2 channels named "event_timestamp" and "event_data" so I could store raw timestamp datatypes to their own named channel. But it feels inelegant, and I'd like a better solution if possible. Specifically, I'd like to deal with this once for a generic cluster and be able to keep re-using it for a variety of as-yet-undefined future clusters of data.

In PJM's posted code, a (presumably) small variation on the OpenG config file vi's, it appears to me that eventually, all the cluster elements get transformed into strings which are then written using 'TDMS Set Properties'. If I were to substitute 'TDMS Write' at that level, and map the "Key" as the "Channel" input, I *think* I would be able to store a (nearly) arbitrary cluster as a TDMS channel. A similar substitution on the Read side would hopefully pull the info back into the cluster. Of course, that particular quick-and-dirty method would cause all clusters to be stored as channels -- even config info, which maps more naturally into a set of TDMS properties. So I'd like to have 2 different top-level calls. One to store a cluster as a set of TDMS properties, another one to store a cluster as an element of channel data. (And of course 2 more for the subsequent Reads).

The problem: I know I'd need to change the vi's at both the highest and lowest level to support this capability. But what about all the layers of interdependent OpenG vi's in between? Hate to duplicate them all under a new name. Not sure that adding extra input(s) throughout the hierarchy is such a good way to map through from intent at the top to implementation at the bottom either.

Another possibility that's occurred to me is to separate the process of converting a cluster into a set of Key/Value pairs of strings for each cluster element from the process of writing those strings. That method would pass the raw strings at the lowest level all the way back up the call chain to the top level. Then the top level could decide how / where (/ whether?) to write them. This approach *might* result in 95% common code between writing a cluster to an INI file, writing a cluster as a set of TDMS properties, writing a cluster as a TDMS channel, writing a cluster in 99 other ways...

Thoughts / ideas / critique welcomed.

-Kevin P.

[Herbert]

I might look at this through my TDMS glasses too much, but to me, the natural way of storing the events you have mentioned would have been to create a channel for each cluster element - where the channel is of the same data type as your cluster element. I realize that this requires you to unbundle and bundle the cluster for writing and reading, respectively. But you wouldn't loose any numeric accuracy, any timestamp tidbits or other things. The only advantage I can see in storing everything as strings would be less coding. Am I missing something there?

I have thought a lot about allowing arbitrary clusters in TDMS. The problem, as you mentioned, is, that you don't know what kind of data you're really dealing with, so it's impossible to magically do the right thing. Some cluster elements are better off being stored as properties, but how would I know? If I store them as properties because they are scalar, I'm out of luck if they change their value after 1000 iterations. Similarly, what would I do with a numeric array in the cluster? Create a channel? Append the array values from the next cluster to that channel? What if these are FFT results? I have not been able to come up with a good way of identifying these things automatically. Of course, you can always come up with some fancy piece of UI that allows users to assign cluster elements to TDMS objects (smells like Express VI ), but the best interface we have for making that assignment is the block diagram.

If a cluster doesn't contain arrays or other clusters, you could make a case for that we should handle that by making each cluster element a channel. That would be a viable thing to do. But when it comes to nested clusters and clusters that include arrays, providing "automatic" handling creates expectations that can hardly be fulfilled.

Herbert

[Kevin P]

Thanks for joining the conversation, Herbert. I'll see if I can present my case more clearly.

There are 3 reasons I brought up the idea of converting clusters to strings for the sake of writing to TDMS.

1. There's no native support for writing clusters.

2. Such cluster unwrapping to and from strings has been (largely) figured out and is available as open source via OpenG and MGI. My understanding is that PJM's earlier post I referenced was only a very slight tweak to write clusters as TDMS properties rather than writing them to a text INI file. I further slightly tweaked it to handle DAQmx global channel names as human-readable strings. The point being, most of the tricky work is already done to support these conversions to and from strings.

3. One further reason I left out of the first post -- it may be an added benefit to be able to read the file from Matlab. We have a site license for Matlab but very few people use LabVIEW and no one I know has Diadem. Personal installs (such as a LabVIEW executable for handling the data, LabVIEW run-time, or even the Excel TDMS plugin) aren't allowed on our networked PC's. So, if people really want to dig through portions of the huge data files that TDMS makes it so convenient to create, I pretty much have to give them a Matlab script. I'm not real nimble in Matlab and the Matlab interface example from ni.com keeps erroring out in a weird way (in trace execution, it appears to execute each of the datatype cases in the 'switch' structure' in order as though falling through like C, then terminating with error). Sooner or later, given enough time, I expect I can work out my problems with the Matlab script. But I can anticipate that new kinds of trouble could arise if I try to use Matlab to retrieve unusual datatypes from TDMS. Binary timestamps are one such type I'm not sure of and any kind of cluster could be a whole new type of difficulty.

Now then, the alternative -- native support for clusters. You mentioned the difficulty of automatically treating cluster elements the "right" way, where some may be treated as properties and others as channels. That's not the feature I'd want. I have some full clusters that I want to write as TDMS properties and other full clusters that I want to write as TDMS channels. As the app programmer, I'll know when to call TDMS Set Properties and when to call TDMS Write. I just want to be able to wire my typedef'ed clusters directly into those TDMS primitives. Some clusters may have dozens of elements and I don't particularly enjoy having to maintain all the unique group, channel, and property strings that tend to change with every app, not to mention the extra hassle of all the unbundling and re-bundling.

As I view it, I'm defining the most important association by bundling the cluster elements together in the first place. If the clusters contain an array of data, that array of data associates to the other elements in the cluster. That's why they were bundled -- they provide one another's context. I don't need one cluster's array of data to be combined with another cluster's array of data to make them contiguous. Just treat the entire cluster as one specific value of a special datatype, just like a DBL or U32 would be treated.

Nevertheless, I can see that native support for clusters as binary data creates some difficulties for other apps like Diadem that need to read the TDMS file. So I kinda figured it might be more realistic to take an approach similar to PJM's example but where, in the end, each cluster element becomes a separate channel. The beauty is that at the app programming level, I simply wire my cluster to one simple "Write" function and all the unbundling into separate named channels happens under the hood. Similar with the "Read" function where I wire my cluster in to define the datatype. When developing the data writer and reader part of the app, it provides the functional equivalent of native binary support for clusters. The main downside is that the OpenG and MGI methods are kinda "brute force" and can't be nearly as optimized as native support could be. Also, re-clustering the individual channels might be difficult from, say, Matlab.

Ok, maybe now I see your point better. In the OpenG-like approach, if a cluster element is an array of numerics, and that array becomes its own individually named TDMS channel, then I'm not sure I can recover the various array chunks back into the appropriate cluster elements after the fact. I suppose any such array element in a cluster might need to write additional metadata such as dimension size(s) for each chunk as yet another named TDMS channel. Or something. Then again, maybe it already does? (Not near an LV PC to check right now).

Again, thanks for engaging the topic. I'm still on the TDMS learning curve, and am sorting through things as the come along.

-Kevin P.