Not A Number/Path/Refnum strange behavior with TDMS file

[PJM_labview]

This is a follow up to this thread Not A Number/Path/Refnum does not work on TDMS file refnum.

Upon sleeping it over, I am still thinking that there is something fishy going on, but before I mess that thread any more than I already did, I am posting this here so people can comment.

Here is the issue.

When I create a new tdms file (using the create or replace operation input) with no error, I expect the refnum to be valid and indeed it appear to be valid since write operation occurs with no errors.

The problem is if I put a "Not A Number/Path/Refnum?" primitive to check its validity, it always return true (invalid refnum). In my opinion this is not the correct behavior.

Run the attached example and see what I mean (make sure you point the TDMS file path to a valid location).

See attachement below.

Download File:post-121-1171384445.vi

Please comment.

PJM

[Mellroth]

QUOTE(PJM_labview @ Feb 13 2007, 05:40 PM)

This is a follow up to this thread http://Not-A-Number-Path-Refnum-does-not-work-on-TDMS-file-refnum-t6234.html' target="_blank">Not A Number/Path/Refnum does not work on TDMS file refnum.

Upon sleeping it over, I am still thinking that there is something fishy going on, but before I mess that thread any more than I already did, I am posting this here so people can comment.

I've verified that this is very strange, and I would classify this as a bug since there are no other way to know whether the TDMS refnum is valid or not.

When "Not A Number/Path/Refnum?" doesn't work you would expect that there was a special method to get the status of this special refnum.

/J

PS. To clarify; my comment in the other thread was only regarding the ordinary file I/O refnums, I'm sorry I forgot to mention that I verified the TDMS refnum problem in the original thread.

[Tomi Maila]

I didn't test this but I agree with you that I think it should function the way you expected.

[aart-jan]

It doesn't stop there: a primitive such as file refnum to path doesn't work either: it actually breaks the wire!

I noticed when using probes that the refnums have very low numbers: 1, 2 etc. The workaround will have to be to cast the refnum to i16 (? not sure) and check whether you have a non-zer number.

Still, TDMS rules!

[PJM_labview]

QUOTE(aart-jan @ Feb 13 2007, 11:26 AM)

It doesn't stop there: a primitive such as file refnum to path doesn't work either: it actually breaks the wire!

Ah good catch Aart. You are correct, this primitive is not working either with TDMS.

QUOTE(aart-jan @ Feb 13 2007, 11:26 AM)

I noticed when using probes that the refnums have very low numbers: 1, 2 etc. The workaround will have to be to cast the refnum to i16 (? not sure) and check whether you have a non-zer number.

Still, TDMS rules!

Yes, I noticed the same thing. I thing the TDMS might be xnodes and not "real" primitive functions. In any case, casting the refnum to number will not let you know whether that refnum is still in use (valid) or not (invalid).

TDMS are cool. But, more than the writing speed (it is easy to write binary file very quickly) this is the read back speed that is nice. Note: The TDMS File viewer is pretty cool too.

Although I should not be too surprised of these bugs since I broke the golden rule of never using new LV features until they have matured...

Note: One of the thing missing from the API is a delete TDMS file that would delete both the tdms file and its index.

PJM

[aart-jan]

Ow, that's a wierd one! Create a new file in windows XP, say a new empty text file, and change the extension to tdms. Watch what happens...

Sometimes the file will miraculously disappear. Other times, even more jaw dropping, it will spontaneously grow an index file! We have a very active background process running here! I believe this is immediatly also one of TDMS's great strengths, because when using TDMS functions seem to execute without any delay (even defragmenting(?)). It seems to me, all these functions do, is handing over the processing load to the background process, releasing Labview from the CPU queue.

Writing and reading...and writing. Simultaneous access is the part that rocks indeed: I allready made software that simultaneously acquires lots of data, writes it to file, just to be read again for further analysis, with the results written back to THE SAME TDMS again! Throw in all possible configuration data, fail pass info...

Bring it to the office: have diadem automatically indexing the data while I sleep!

Being an early adopter is simple economics and fun this time..

[Ton Plomp]

QUOTE(PJM_labview @ Feb 13 2007, 10:40 PM)

Note: The TDMS File viewer is pretty cool too.

Although I should not be too surprised of these bugs since I broke the golden rule of never using new LV features until they have matured...

Note: One of the thing missing from the API is a delete TDMS file that would delete both the tdms file and its index.

Just one bug in the TDMS File viewer, it doen't respond on panel close..., So if you have it in your program, you load it automatically and the user closes the panel (not using quit) you have a running sub-vi you can't exit.... It is just a little extra programming but aaahhhhh

Ton

[Herbert]

First of all, thanks a bunch to everybody for digging these issues up and giving all this feedback :thumbup: . I was out of office for a conference in Orange County :beer: so sorry for not answering right away ... but here we go.

1) TDMS refnums are supposed to behave like any other file refnum. The fact that they return "true" almost all the time is a bug. For further reference, the issue is filed as CAR# 46KIDFWJ and can be expected to be fixed in the next major LabVIEW release. There is no great workaround for that, the best you can do is using a function like "Get Properties" (without group or channel name wired) or "List File Contents" and see if it errors out.

2) The "Refnum to Path" issue indeed is a nice catch. Again, TDMS refnums are supposed to behave just like any other file I/O refnum. CAR# 46KII9WJ. Sorry for the inconvenience.

3) Casting the TDMS refnum into an integer and checking for not zero will not work just like that. The list of files can be sparse, so a non-zero number wouldn't necessarily mean that the refnum is referring to an open file. On top of that, the 2 issues mentioned above might get fixed in a way that breaks this workaround, so I would rather not recommend using it.

4) TDMS functions are primitives. TDMS functions and refnums are implemented on the same core level as the normal file I/O. No XNodes or something.

5) I can see how a delete function would come in handy. We had even more requests for a merge function, I think someone actually programmed it and put it up on the ni.com discussion forums (link).

6) The rules for opening a TDMS file have been subject to some confusion. It has been a key requirement that TDMS files can be transported as one file (as opposed to TDM, which always needs 2 files). Hence TDMS is implemented so the index file is optional. However, we do prefer to have the index file, because it allows for quick random access into the TDMS file without loading much information. So when we find a TDMS file without index, we will generate the index from the TDMS file. Now, if you have the DIAdem DataFinder running on the folder that your file is in, the DataFinder will be the first to open that file and hence generate the index file.

Files that are completely empty will be deleted on close. This requirement stems from people complaining about empty files on their disc after doing a simple open and close. Valid TDMS file created by LabVIEW, CVI or DIAdem should never be empty though. Again, if the DataFinder gets a hold of your file, that's how it will magically disappear.

Warning: If you have a TDMS and TDMS_INDEX file [same names, same folder] that are not matching, you will likely get some kind of error. You will not be able to read data from that file. Workaround is deleting the TDMS_INDEX file. We're working on preventing this kind of problem.

7) The panel close thing on the TDMS Viewer is a known problem that has been addressed. I don't have the CAR# handy. Workaround: Use the "Quit" button :headbang:

I hope I addressed all the questions you guys had in this thread, please don't hesitate to throw more at me. Thanks again for putting this together.

Herbert

[Ton Plomp]

Hello Herbert,

good answers!

An Q:

How do we (re)set the read marker?

We were tryin to read twice the same dataset, so in one loop read all data, and after that read the whole dataset again. But we got directly a 'end of file' error on the second read structure, explicitly setting the offset 0 didn't work...

The workaround was to close/open the tdms, but that's not too nice!

Ton

[PJM_labview]

QUOTE(Herbert @ Feb 21 2007, 06:22 PM)

...

6) The rules for opening a TDMS file have been subject to some confusion. It has been a key requirement that TDMS files can be transported as one file (as opposed to TDM, which always needs 2 files). Hence TDMS is implemented so the index file is optional. However, we do prefer to have the index file, because it allows for quick random access into the TDMS file without loading much information. So when we find a TDMS file without index, we will generate the index from the TDMS file. Now, if you have the DIAdem DataFinder running on the folder that your file is in, the DataFinder will be the first to open that file and hence generate the index file.

Files that are completely empty will be deleted on close. This requirement stems from people complaining about empty files on their disc after doing a simple open and close. Valid TDMS file created by LabVIEW, CVI or DIAdem should never be empty though. Again, if the DataFinder gets a hold of your file, that's how it will magically disappear.

Warning: If you have a TDMS and TDMS_INDEX file [same names, same folder] that are not matching, you will likely get some kind of error. You will not be able to read data from that file. Workaround is deleting the TDMS_INDEX file. We're working on preventing this kind of problem.

...

This is interesting, I don't not remember reading about this feature (index file is optional).

QUOTE(tcplomp @ Feb 21 2007, 09:31 PM)

Hello Herbert,

good answers!

An Q:

How do we (re)set the read marker?

We were tryin to read twice the same dataset, so in one loop read all data, and

after

that read the whole dataset again. But we got directly a 'end of file' error on the second read structure, explicitly setting the offset 0 didn't work...

The workaround was to close/open the tdms, but that's not too nice!

Ton

Yes!, I noticed that too. I was trying to access the TDMS in an intelligent global (caching the reference in a shift register) and I kept getting EOF error when I was trying to reuse the ref and when I was (or was not) at the end of the file. That error was so persistent that I could not figure out any way either to get out of the situation (offset to 0 did not work as you mentioned). I had to drop the reference caching and reopen a new one every time I was reading the data back. This is not was I was hoping to do.

PJM

[Herbert]

I would recommend iterating over a TDMS file using "List Contents" and a for loop. That obviously works multiple times in a row. See attachment 1.

The while loop / eof behavior was implemented to allow for a user experience that is somewhat consistent with the existing file I/O. We start at the most recently read group and iterate group-by-group. We do not consider offset and count in order to iterate within a group, we just go to the next group. If you want to iterate that way and you want to reset to the beginning of the file, you can simply set the group name back to the first group. There's another minor bug here in that wiring an empty string as a group name is not doing the same as not wiring anything, hence the case structure in attachment 2.

None of these approaches has a measurable performance advantage over the other.

Hope that helps,

Herbert

[Rick]

QUOTE(Herbert @ Feb 21 2007, 09:22 PM)

First of all, thanks a bunch to everybody for digging these issues up and giving all this feedback :thumbup: . I was out of office for a conference in Orange County :beer: so sorry for not answering right away ... but here we go.

1) TDMS refnums are supposed to behave like any other file refnum. The fact that they return "true" almost all the time is a bug. For further reference, the issue is filed as CAR# 46KIDFWJ and can be expected to be fixed in the next major LabVIEW release. There is no great workaround for that, the best you can do is using a function like "Get Properties" (without group or channel name wired) or "List File Contents" and see if it errors out.

2) The "Refnum to Path" issue indeed is a nice catch. Again, TDMS refnums are supposed to behave just like any other file I/O refnum. CAR# 46KII9WJ. Sorry for the inconvenience.

3) Casting the TDMS refnum into an integer and checking for not zero will not work just like that. The list of files can be sparse, so a non-zero number wouldn't necessarily mean that the refnum is referring to an open file. On top of that, the 2 issues mentioned above might get fixed in a way that breaks this workaround, so I would rather not recommend using it.

4) TDMS functions are primitives. TDMS functions and refnums are implemented on the same core level as the normal file I/O. No XNodes or something.

5) I can see how a delete function would come in handy. We had even more requests for a merge function, I think someone actually programmed it and put it up on the ni.com discussion forums (http://forums.ni.com/ni/board/message?board.id=250&message.id=27470' target="_blank">link).

6) The rules for opening a TDMS file have been subject to some confusion. It has been a key requirement that TDMS files can be transported as one file (as opposed to TDM, which always needs 2 files). Hence TDMS is implemented so the index file is optional. However, we do prefer to have the index file, because it allows for quick random access into the TDMS file without loading much information. So when we find a TDMS file without index, we will generate the index from the TDMS file. Now, if you have the DIAdem DataFinder running on the folder that your file is in, the DataFinder will be the first to open that file and hence generate the index file.

Files that are completely empty will be deleted on close. This requirement stems from people complaining about empty files on their disc after doing a simple open and close. Valid TDMS file created by LabVIEW, CVI or DIAdem should never be empty though. Again, if the DataFinder gets a hold of your file, that's how it will magically disappear.

Warning: If you have a TDMS and TDMS_INDEX file [same names, same folder] that are not matching, you will likely get some kind of error. You will not be able to read data from that file. Workaround is deleting the TDMS_INDEX file. We're working on preventing this kind of problem.

7) The panel close thing on the TDMS Viewer is a known problem that has been addressed. I don't have the CAR# handy. Workaround: Use the "Quit" button :headbang:

I hope I addressed all the questions you guys had in this thread, please don't hesitate to throw more at me. Thanks again for putting this together.

Herbert

Herbert,

Thanks for the good info and acknowledgement of outstanding bugs vs. expected behavior.

I've found another problem with the TDMS File Viewer that I'd like to throw out there:

The TDMS File Viewer doesn't load group/channel values that contain very large value length (haven't tried determining the failure limit) data arrays/waveforms. I've written a DAQ application that streams time data to disk and also makes that raw time data available to post processing routines, the output of which is appended to the same TDMS collecting the streamed raw time data. In my case, the streamed time data can easily contain well over 100,000 continuous time data points.

In trying to use the TDSM File Viewer to inspect my data file, I find that all the groups/channels entries and data is there. But, the data values are tabulated and graphed (as inspected using the tab control) ONLY for all of my post processed channel groups which contain ~20,000 data points. For the raw streamed time data channel group, the "Properties" tab lists correct information (including e.g. Property name=NI_ChannelLength with Property value=327680), but the "Values (table)" tab & "Analog values (graph)" tabs contain no data.

With some inspection, I've traced this to the "Get TDMS Properties" primitive (in the "loadAndFormatValues.vi" _{) that is returning info on the "NI_ChannelLength" property name. It seems this property is found, but that this length is returned as 0 (zero) for sufficiently large data sets. This is especially strange since the data length is correctly indicated as 327680 in the "Properties" tab.}

_{Is this a known fundamental limit of the "Get TDMS Properties" primitive?}

_Thanks,

_Rick

[Ton Plomp]

Hi Rick,

have you checked the 'settings....' buttons, default the TDMS viewer will load 10k (or 100k) points...

Ton

[Herbert]

QUOTE(Rick @ Mar 7 2007, 02:05 PM)

Is this a known fundamental limit of the "Get TDMS Properties" primitive?

No, that's a rather embarrassing bug in the TDMS File Viewer. The good news is that you can fix it really easily. It's in a subVI called loadAndFormatValues.vi. This subVI contains a loop that first gets the NI_DataType, correctly as a U16. I think I must have copy-pasted this in order to read the NI_ChannelLength right after that, without changing the data type to U64.

All you need to do is go into this subVI, right-click the integer constant that determines the data type for "Get Properties" (the one that gets NI_ChannelLength) and switch the representation to U64. Obviously, this will be fixed in the next LabVIEW release.

The reason it works on the "Properties" tab is that we get the properties for this tab as variants. These are then converted properly. The "Get Properties" we use for the dialog that configures how many values you can see in table and graph have the correct U64 constant wired.

Hope that helps,

Herbert

[Rick]

QUOTE(Herbert @ Mar 7 2007, 03:59 PM)

No, that's a rather embarrassing bug in the TDMS File Viewer. The good news is that you can fix it really easily. It's in a subVI called loadAndFormatValues.vi. This subVI contains a loop that first gets the NI_DataType, correctly as a U16. I think I must have copy-pasted this in order to read the NI_ChannelLength right after that, without changing the data type to U64.

All you need to do is go into this subVI, right-click the integer constant that determines the data type for "Get Properties" (the one that gets NI_ChannelLength) and switch the representation to U64. Obviously, this will be fixed in the next LabVIEW release.

The reason it works on the "Properties" tab is that we get the properties for this tab as variants. These are then converted properly. The "Get Properties" we use for the dialog that configures how many values you can see in table and graph have the correct U64 constant wired.

Hope that helps,

Herbert

Thanks, Herbert. Of course that did it and it was a very simple fix. Thanks, this viewer just became much more useful to me!

I figured that the use of variant was the reason that the properties tab was working, but didn't know the NI_ChannelLength value was U64.

Since you seem to be the author of this fine utility, here come a feature request:

It would be great if there was the NI-supported, intrinsic ability to view the data in an vector X vs. vector Y format, rather than waveform only. One obvious use of this would be plotting processed frequency domain data against a frequency vector. This would take the usefulness of this utility to the next level.

Thanks again for the support!

Rick