Read data at a specific time from TDMS

[Thang Nguyen]

Hi,

I am working on a Testing project. The project will store data from the test in the TDMS file. What I need to do next is reading a sub-set of data from a start time to an end time which are specificed by user? How should I do this? I cannot find any TDMS VI provide the time selection.

Thank you for your time,

Thang Nguyen

[ragglefrock]

QUOTE(Thang Nguyen @ Jun 13 2007, 06:37 PM)

Hi,

I am working on a Testing project. The project will store data from the test in the TDMS file. What I need to do next is reading a sub-set of data from a start time to an end time which are specificed by user? How should I do this? I cannot find any TDMS VI provide the time selection.

Thank you for your time,

Thang Nguyen

TDMS VIs won't directly give you a time-based subset of your data, but all the information is there to do this with a little work yourself. Here are two approaches, the easy but inefficient approach and the more correct method (both methods assume you wrote your data to the TDMS file as waveforms):

Easy but Inefficient:

Read the whole channel from file and use the Waveform Subset VI to exctract the desired portion. Bad for very large channels.

More Correct Method:

If you write a waveform to a TDMS file, the underlying data is really stored as an array with a special channel property called wf_increment to store the dt value. You can read this property for your channel and get the double-precision dt value. The property for the t0 value I believe is called wf_offset (view the TDMS channel in the TDMS Viewer to get the exact names). Then you just calculate based on the t0 and dt the offset in the channel array where the desired waveform subset begins, and how long the subset will be. Then use the TDMS Read inputs Count and Offset to specify how many datapoints to read out. Then build an waveform from this array with the new t0 and output it.

[Thang Nguyen]

QUOTE(ragglefrock @ Jun 13 2007, 10:52 PM)

More Correct Method:

If you write a waveform to a TDMS file, the underlying data is really stored as an array with a special channel property called wf_increment to store the dt value. You can read this property for your channel and get the double-precision dt value. The property for the t0 value I believe is called wf_offset (view the TDMS channel in the TDMS Viewer to get the exact names). Then you just calculate based on the t0 and dt the offset in the channel array where the desired waveform subset begins, and how long the subset will be. Then use the TDMS Read inputs Count and Offset to specify how many datapoints to read out. Then build an waveform from this array with the new t0 and output it.

Thank you so much for your answer. The only think is although I log two channels with different rate, but the wf_increment the same and they have the value 1. And this is the dt of the waveform, not the dt of the logging rate to the tdms. I am thinking another solution to get the right value. Maybe I will read all of the channel, and divide this number for the number of data I have in the array. But this sounds stupid. Is there any better solution?

Thank you again,

Thang Nguyen

[Ton Plomp]

QUOTE(Thang Nguyen @ Jun 14 2007, 07:22 PM)

Thank you so much for your answer. The only think make me confuse is although I log two channels with different rate, but the wf_increment the same and they have the value 1. Could you tell me why?

Thank you again,

Thang Nguyen

Most likely you are storing arrays and not waveforms, sadly.

Ton

[Thang Nguyen]

QUOTE(tcplomp @ Jun 14 2007, 12:59 PM)

Most likely you are storing arrays and not waveforms, sadly.

Ton

Nope, I store waveforms. I can see the waveform in the TDMS viewer

[Ton Plomp]

QUOTE(Thang Nguyen @ Jun 15 2007, 01:19 AM)

Nope, I store waveforms. I can see the waveform in the TDMS viewer

Can you post a short TDMS?

And maybe the code of the saving?

Ton

[Thang Nguyen]

QUOTE(tcplomp @ Jun 14 2007, 11:19 PM)

Can you post a short TDMS?

And maybe the code of the saving?

Ton

I attach here the picture of VI I used to write to TDMS.

A TDMS example. But currently I don't have any input, these values just noise.

In TDMS, I have two groups of value. One is high frequency, another is low frequency. They have different number of value 1/3. Have the same wf_start_offset, but they have the same wf_increment. So they have different end time store of data.

Please modify the name of the tdms file. I don't understand why I cannot upload it.

[Ton Plomp]

QUOTE(Thang Nguyen @ Jun 15 2007, 06:04 PM)

Please modify the name of the tdms file. I don't understand why I cannot upload it.

I think Michael limited that to a few extensions.

If I look at your file with DIADem I see the same as you... Can you verify that LabVIEW stores the data correct in the array.

Ton

[Thang Nguyen]

Yeah, they are stored in the right order. And I store data at different rate. You can see the number of data in high frequency is larger the number of data in low frequency. All of them start and stop at the same time.

[Herbert]

QUOTE(Thang Nguyen @ Jun 18 2007, 12:23 PM)

Yeah, they are stored in the right order. And I store data at different rate. You can see the number of data in high frequency is larger the number of data in low frequency. All of them start and stop at the same time.

Looking at the file with the TDMS Viewer, what you have is:

• different channel lengths (high freq channels have 618 values, low freq channels have 224)
  
• same dT (1.00 for all channels)
  
• varying starting times (T0) for every channel
  

It looks like you are using waveforms to save single values. In that case, I'm not sure that DAQmx or other functions that put out waveforms will set dT correctly, because there is no second value to reference to. If you save a series of single values to a waveform channel, you need to be really sure that they are equally sampled. If you're not sure, you should rather split up the waveform data type and store the timestamps and the data values to different channels (e.g. one timestamp and one double channel).

Saving single values to TDMS like this is also not a very efficient thing to do. It is a lot more efficient to gather a bunch of values and write them as a larger array. You can have the TDMS API do that for you by setting the channel property "NI_MinimumBufferSize" to the number of values that you wish to buffer. In your case, good values might be 1000 or 10000.

Hope that helps,

Herbert

[Thang Nguyen]

QUOTE(Herbert @ Jun 18 2007, 01:58 PM)

Looking at the file with the TDMS Viewer, what you have is:

• different channel lengths (high freq channels have 618 values, low freq channels have 224)
  
• same dT (1.00 for all channels)
  
• varying starting times (T0) for every channel
  

It looks like you are using waveforms to save single values. In that case, I'm not sure that DAQmx or other functions that put out waveforms will set dT correctly, because there is no second value to reference to. If you save a series of single values to a waveform channel, you need to be really sure that they are equally sampled. If you're not sure, you should rather split up the waveform data type and store the timestamps and the data values to different channels (e.g. one timestamp and one double channel).

Saving single values to TDMS like this is also not a very efficient thing to do. It is a lot more efficient to gather a bunch of values and write them as a larger array. You can have the TDMS API do that for you by setting the channel property "NI_MinimumBufferSize" to the number of values that you wish to buffer. In your case, good values might be 1000 or 10000.

Hope that helps,

Herbert

In my program I put store VI into the loop. It scans through the array name of channel, then, read and store the correspondent data into the file in that sequence. That why those channel don't have the same start time. But the diffference is just some msec.

I have two questions for you:

A) Is the wf_increment depends on the dt? If that is the case, I understand why the data are stored with the same wf_increment.

B) Do you mean that I should gather all of the Y value of each channel into an array, then put everything into the TDMS file? Could you tell me more detail how to store everything at once into the TDMS file?

Thank you,

Thang Nguyen

[Herbert]

Thang,

A) The idea here is that users should never have to touch properties like wf_increment or even know about them. We use the wf_xxx properties to store things that are embedded in LabVIEW data types (e.g. T0 and dT are embedded in the waveform data type). If you use waveforms correctly, all of these properties should be written and read without you doing something special. That of course only works if the waveforms have the correct values in them. Since you are asking - here are the important ones:

• T0 is saved to wf_start_time (timestamp).
  
• dT is saved to wf_increment (double).
  
• If your data is not time-domain, wf_start_time will still be set, but your X0 value goes into wf_start_offset (double). This will happen for example with frequency-domain data or histogram results.
  
• If you exchange data with DIAdem, you need to set the wf_samples property to something other than 0 (we ususally set it to the number of values in the incoming waveform, so in your file, it is 1). DIAdem will use this property to determine whether a channel is a waveform or not.
  

B) That's exactly right. The only thing you need to do is set the property NI_MinimumBufferSize (integer) for each of your data channels to 1000 or 10000 or something. The TDMS API does the buffering automatically (requires LV 8.2.1). This is not crucial to the functionality of your application, but it will speed up writing and reading quite a bit.

Unrelated) I see from the flags on your account that you're from Vietnam. I just came back from 2 weeks of vacation, visiting friends in Vietnam. They took me on a roundtrip through the country, including Hanoi, Ha Long, Nha Trang and Saigon. Best vacation I had in a long time. I'm addicted to Caffee Suo Da now :thumbup:

Herbert

[Thang Nguyen]

Herbert,

Thank you so much for your meanful information.

I want to confirm this information:

Is it always true that storing the Y value of waveform and seperate it with the timestamp is more efficient than storing the waveform?

Based on the property of TDMS channel (wf_increment, wf_start_time), I think TDMS naturally used to store waveform.

But in this case, it is easier for me to read data out, if I seperate the data and timestamp.

Love to hear about this:

QUOTE

Unrelated) I see from the flags on your account that you're from Vietnam. I just came back from 2 weeks of vacation, visiting friends in Vietnam. They took me on a roundtrip through the country, including Hanoi, Ha Long, Nha Trang and Saigon. Best vacation I had in a long time. I'm addicted to Caffee Suo Da now :thumbup:

I have never come to Caffee Suoi Da, actually, there are a lot of coffee in Vietnam ( a lot ...). I lived in Saigon. . Do you feel hot there?

Thank you again,

Thang Nguyen

[Herbert]

QUOTE(Thang Nguyen @ Jun 18 2007, 04:49 PM)

Is it always true that storing the Y value of waveform and seperate it with the timestamp is more efficient than storing the waveform?

Based on the property of TDMS channel (wf_increment, wf_start_time), I think TDMS naturally used to store waveform.

But in this case, it is easier for me to read data out, if I seperate the data and timestamp.

As long as your data values are equally sampled, storing them as waveforms is a lot more efficient. A timestamp is 128bit, a double value is 64bit. So, by not storing a timestamp with every data value, you save 2/3 of disc footprint and performance. Of course, if your data is not equally sampled, this is not useful for you. In that case, you need to store time stamp and data values to different channels. If it is easier for you to read out, you can always store time and data to different channels, but it might become a performance bottleneck in your application.

QUOTE(Thang Nguyen @ Jun 18 2007, 04:49 PM)

I have never come to Caffee Suoi Da, actually, there are a lot of coffee in Vietnam ( a lot ...).

Plus, there is always the B52 :thumbup: . (If only I could figure out how to embed videos in my posts...).

QUOTE(Thang Nguyen @ Jun 18 2007, 04:49 PM)

I lived in Saigon.

. Do you feel hot there?

Not really. I live in Austin, Texas. That's about as hot as Saigon. I wouldn't dare driving a motorcycle in Saigon, though ...

Herbert