16 replies to this topic

[Wouter]

Other question... I don't really know anything about TDMS... But why save data in those files?

[hooovahh]

I'm sure others will chime in on their opinions. But some of the benefits I know of are...

Direct DMA streaming from channels to disk including channel scaling and settings (my understanding is this happens without CPU intervention).

Small file size with combination of binary and XML data.

Reading TDMS data is quite easy using the TDMS palette, and you don't need to read the whole file to get the data you need. Grab the group, and channel and display it on a graph. Data is found based on names not column or row indexes.

You can concatenate TDMS files using a binary copy function (copy /b in command line) and channel data will be appended.

Using properties of the File/Group/Channel helps with organization and header data.

Can be opened in Excel with a free add-on. (and looks decent where things like column width and font style on import, that you don't get on open in ASCII)

Can store just about any data type. http://lavag.org/top...in-a-tdms-file/

Works nativly in DIAdem for automated report generation (not much experience with this when needed I have written my own in LabVIEW)

Can be opened in Matlab (not much experience with this either but I heard there are drivers)

Comes with a half way decent file viewer, which can be modified to your needs.

I started using TDMS files for storing development and debug data. Then I started putting the actual data for what I was doing in there. Now when a test completes I can take that TDMS file, and generate a more formal looking report from it. This way there is a nice looking report with the data we care about at the end, but we still have the TDMS file which will contain all raw data. But when it comes down to it use what works for you. TDMS maybe overkill, or you might not feel comfortable using it, and that's fine. For me it works well, but it might not be right for every system.

[Wouter]

Thanks for your post, with a clear overview of the benefits.

Most of the stuff I readed @ http://zone.ni.com/d...a/tut/p/id/3727 (sorry had to read first before asking ), it then made me more curious about the disadvantages... because how NI describes it, it is the best format available which I doubt.

[Mads]

TDMS is extremely flexible. You can dump anything you want into the files, at any time - without worrying about how to find it again. The downside in my case is the speed you get when you need to read the data. Even defragmentet (a must, if you need to write in small and varying segments, otherwise the performance gets really crappy), a custom binary format will be much much faster to read.

[Darin]

HDF5 seems conspicuously absent from the discussions and comparison charts.

[Ton Plomp]

One of the mayor ups for our company is the possibility of adding meta-data (descriptions), these can be very flexible and extendible.
By configuring the DIADem data finder, you can harvest these meta-data like a database.

One of the downsides is that we had several cases of bad tdms-indexes. Those are really hard to nail down, and we never got a good explaination. It probably comes from the fact that you'll only copy the tdms file without the accompanying tdms_index file, and you can get a mix up of various data-formats, the solution is always easy: delete the tdms_index file (it will be auto-generated again upon opening the file).

There is a discussion of HDF5 vs. TDMS, from what I remember is that at the moment of development HDF5 wasn't fast enought for single-writes.
(Idea-exchange link, but there should be more)

One thing that I like of TDMS is that it limits you to groups and channels, one way or another it's always been possible to store the relevant data in that structure. But I would love to have (at least) 2-dimensional arrays.

Ton

[PaulG.]

Hooovahh's summary is better than anything I can offer. For flexibility and performance you can't beat TDMS, or at least I don't know of any other way to store huge amounts of data as efficiently and reliably. One of LabVIEW's better features. I have had no trouble with it at all.

[sachsm]

TDMS Rocks! I find it to be very fast for writes and reads. So fast that I now use it as a core data store for my DAQ applications that need to be able to acquire data and simultaneously read out and display the data. You can easily emulate the kind of real time historical data viewer that you get in the distributed system manager using this capability of TDMS.

[COsiecki]

TDMS is a fantastic format. Because I wrote almost all of the DAQ software we have, I have replaced the custom formats we used with TDMS. The ease of use through LabVIEW makes it almost as simple as adding an indicator to your front panel. It's also very powerful for adding metadata. You don't have to figure out where you are going to put some new property in your file that doesn't mess with your offsets, or if you can add another channel without needing to re-write your reader.

We have had some issues using TDMS in MATLAB. As far as I know there are is no official support for TMDS by Mathworks (they probably see NI as a competitor). One way to do it is to use the TDMS dll and do library function calls, but those can be finicky and can cause MATLAB to crash to desktop. The other problem we had with that method is that, at least when we were using it, the 64-bit version of MATLAB had no built in way to use a dll and you had to link it to an external compiler. It was not an ideal solution. In the end, we wound up using the TDMS format white paper to write a MATLAB-native reader, but there are some data types that aren't supported and we are lacking a number of nice features. It works for the way we tend to use the files we create though.

[JamesMc86]

I think all the points above are great. I always get a good reaction when showing it to customers and the excel importer can produce easier data to go though that your own text files.

One thing to just be aware of with it is fragmentation. It's a common problem that we see, esp. on embedded platforms where file size is more sensitive. There is a white paper describing the exact implementation but the short version is that each time you use TDMS Write a "chunk" of data is written to file (Im not sure if this is the proper term). If that chunk matches the previous (same channels, datatype, I think size) then it is written just as binary data and maintains the efficiencies. If it does not match then it writes a new piece of XML header data with the data, which is very inefficient. The result is if you mix the reads to a single file your XML header becomes huge, as does your file and your access becomes slow.

The solution is to try and keep to a single write function to stream to disk. If it is unavoidable then there is a TDMS defrag function which will then reduce the size again but this can take some time and CPU.

Don't misunderstand me though, this is the main caveat you could run into but overall the benefits outweigh this (and it is avoidable as long as you know it exists!) you do get extremely compact, searchable data that can be easily shared. To get similar write performance you would have to go to a binary format which means the end user will need a program capable of reading that.

Cheers,
James

[hooovahh]

One thing to just be aware of with it is fragmentation. It's a common problem that we see, esp. on embedded platforms where file size is more sensitive. There is a white paper describing the exact implementation but the short version is that each time you use TDMS Write a "chunk" of data is written to file (Im not sure if this is the proper term). If that chunk matches the previous (same channels, datatype, I think size) then it is written just as binary data and maintains the efficiencies. If it does not match then it writes a new piece of XML header data with the data, which is very inefficient. The result is if you mix the reads to a single file your XML header becomes huge, as does your file and your access becomes slow.

The solution is to try and keep to a single write function to stream to disk. If it is unavoidable then there is a TDMS defrag function which will then reduce the size again but this can take some time and CPU.

This is true but one work around is to set the buffer size of the channel before writing the first time. This will allow you to use the write function several times without it needing to actually write to disk, which means it will not need to write the XML header. The down side is the data is not actually written to disk, and if power is removed from the PC then your buffer is lost. For this reason I try to make my buffer size appropriate, so I'm not writing XML header data all the time, but I also don't have a buffer so large that I don't write all the data at once (unless that is desired).

http://zone.ni.com/r...dms_buffersize/

[ASTDan]

I would only recomend TDMS for a customer who needs the high streaming, and is very comfortable with NI tools.

I recomend for the most part saving data in simple ASCII files. The reason I like ASCII files.

1. Human Readable
2. Can be read by anything (Excel, Diadem, Matlab, any data viewing program worth its salt)
3. Been around for forever and will be around forever.
4. Non engineer follks feel comfortable working with ASCII files
5. VERY easy to use.

The knock on ASCII files is they can be slow and big. My usual answer is hard drives are cheap, and I can design the program to save data in a managable amount. That being said I have seen data requirements where TDMS would be a good solution.

The cons for TDMS

1. NI propiatary. NI can change the format when they want. We have already been through TDM, TDMS version 1, and TDMS version 2. I think the format is stable for know, however it is something to be aware of.
2. Non Human Readable
3. Outside of the NI ecosphere who has heard of TDMS?
4. Only works with NI Diadem and LabVIEW (Can download a free plug in for Excel, but that is a pain for folks. You will get a phone call everytime somebody wants to open the data)
5.Learning curve

In short my default file format is ASCII. If you need the high speed streaming, lower footprint, and are comfortable using NI tools TDMS could be a nice fit.

[Ton Plomp]

We recently sent data in TDMS format to another party (sort of competition), and we decided not to mention the dataformat, jus sent the files.
Later I spoke one of there developers, and and I figured out they were using Matlab were able to open the files without any issue.

Ton

[sachsm]

Also don't forget the TDMS Excel Plugin.

and this...

http://originlab.com...ormats&pid=1047

[JamesMc86]

Hi,

Thanks for the point about the data buffers. I had not experimented much with the buffering but it certainly does improve the situation (I will upload a comparison to the NI community shortly)

EDIT: If you want to see the effect of buffers I have now uploaded code to https://decibel.ni.c.../docs/DOC-20522

I think this discussion sums up that the whole decision comes down to the fact it is going to depend where you want the data to end up. TDMS is easiest the NI ecosystem although I have had people impressed with the way the data is pulled into excel (theres also a plugin for open office) and the C dll does allow it to be added to custom applications (including Matlab). If you need the data to work with notepad or another specific program then you have to stick to with an ASCII format or a custom binary format that the program supports.

Where I would say TDMS is a must is when your talking big sizes. With the advanced API you can actually write to disk faster than with the standard binary VIs and dealing with large files requires the random access available in the API or with DIAdem's ability with large data sets.

[sachsm]

How about this...

https://github.com/mnaberez/tdms

Wow, this just goes on and on...

http://pytdms.sourceforge.net/

[Stobber]

HDF5 seems conspicuously absent from the discussions and comparison charts.

I'm surprised nobody else noticed this. NI used to have an API for a subset of HDF5 that shipped with their Modular Instruments (Scope, FGEN, HSDIO) drivers called NI-HWS. It was pretty solid but needed some bug fixes that never got attention. The HWS format, and all HDF5 support along with it, was ultimately canned in favor of promoting TDMS as the platform-supported file format for waveform data storage. You won't hear anyone from NI marketing talking about HDF5 again.

Most of the stuff I readed @ http://zone.ni.com/d...a/tut/p/id/3727 (sorry had to read first before asking ), it then made me more curious about the disadvantages... because how NI describes it, it is the best format available which I doubt.

Again, I'm surprised nobody else has mentioned the data types that TDMS doesn't support. One I can immediately think of is multi-dimensional (>3 dims) arrays: TDMS doesn't support them, while HDF5 does.

Oh, another is that you can't put array or cluster types into TDMS properties. That can lead to some unfortunately creative restructuring of your metadata in order to facilitate the format.

Edited by Stobber, 26 June 2012 - 05:27 PM.