Jump to content


Photo
- - - - -

Why TDMS?


  • Please log in to reply
43 replies to this topic

#1 Wouter

Wouter

    Very Active

  • Members
  • PipPipPip
  • 106 posts
  • Version:LabVIEW 2011
  • Since:2006

Posted 29 December 2011 - 05:21 PM

Other question... I don't really know anything about TDMS... But why save data in those files?

#2 hooovahh

hooovahh

    Im no supervising technician Im a technical supervisor

  • Moderators
  • 1,457 posts
  • Version:LabVIEW 2013
  • Since:2004

Posted 29 December 2011 - 08:15 PM

*
POPULAR

I'm sure others will chime in on their opinions. But some of the benefits I know of are...

Direct DMA streaming from channels to disk including channel scaling and settings (my understanding is this happens without CPU intervention).

Small file size with combination of binary and XML data.

Reading TDMS data is quite easy using the TDMS palette, and you don't need to read the whole file to get the data you need. Grab the group, and channel and display it on a graph. Data is found based on names not column or row indexes.

You can concatenate TDMS files using a binary copy function (copy /b in command line) and channel data will be appended.

Using properties of the File/Group/Channel helps with organization and header data.

Can be opened in Excel with a free add-on. (and looks decent where things like column width and font style on import, that you don't get on open in ASCII)

Can store just about any data type. http://lavag.org/top...in-a-tdms-file/

Works nativly in DIAdem for automated report generation (not much experience with this when needed I have written my own in LabVIEW)

Can be opened in Matlab (not much experience with this either but I heard there are drivers)

Comes with a half way decent file viewer, which can be modified to your needs.

I started using TDMS files for storing development and debug data. Then I started putting the actual data for what I was doing in there. Now when a test completes I can take that TDMS file, and generate a more formal looking report from it. This way there is a nice looking report with the data we care about at the end, but we still have the TDMS file which will contain all raw data. But when it comes down to it use what works for you. TDMS maybe overkill, or you might not feel comfortable using it, and that's fine. For me it works well, but it might not be right for every system.
  • jcarmody and asbo like this

"Maybe Hoovah is really Crelf's alter-ego, which he uses to irk people?" - Gary Rubin
"Seemingly minor changes....can mean the difference between a working app and a quivering heap of unresponsive code." - David Boyd
"Using unofficial features of LabVIEW is cat nip laced with poison."
"3rd party VIs with block diagrams that are locked are useless. You see the problem all lies in the phrase "...if the VIs do what you want...". They typically don't and with locked diagrams you can't fix the problems." - Mike Porter
"If you really need a real-time system, then you really need a real-time system." - Hooovahh


#3 Wouter

Wouter

    Very Active

  • Members
  • PipPipPip
  • 106 posts
  • Version:LabVIEW 2011
  • Since:2006

Posted 29 December 2011 - 09:04 PM

Thanks for your post, with a clear overview of the benefits.

Most of the stuff I readed @ http://zone.ni.com/d...a/tut/p/id/3727 (sorry had to read first before asking :oops: ), it then made me more curious about the disadvantages... because how NI describes it, it is the best format available :P which I doubt.

Posted Image

#4 Mads

Mads

    Very Active

  • Members
  • PipPipPip
  • 195 posts
  • Location:Bergen, Norway
  • Version:LabVIEW 2012
  • Since:1997

Posted 29 December 2011 - 09:50 PM

TDMS is extremely flexible. You can dump anything you want into the files, at any time - without worrying about how to find it again. The downside in my case is the speed you get when you need to read the data. Even defragmentet (a must, if you need to write in small and varying segments, otherwise the performance gets really crappy), a custom binary format will be much much faster to read.
MTO

#5 Darin

Darin

    Extremely Active

  • Members
  • PipPipPipPip
  • 252 posts
  • Version:LabVIEW 2009
  • Since:1992

Posted 29 December 2011 - 09:59 PM

HDF5 seems conspicuously absent from the discussions and comparison charts.
  • Stobber likes this

#6 Ton Plomp

Ton Plomp

    How many lines per hour? Zero!

  • Members
  • PipPipPipPipPipPip
  • 2,005 posts
  • Location:New Zealand
  • Version:LabVIEW 2012
  • Since:2000

Posted 29 December 2011 - 10:10 PM

One of the mayor ups for our company is the possibility of adding meta-data (descriptions), these can be very flexible and extendible.
By configuring the DIADem data finder, you can harvest these meta-data like a database.

One of the downsides is that we had several cases of bad tdms-indexes. Those are really hard to nail down, and we never got a good explaination. It probably comes from the fact that you'll only copy the tdms file without the accompanying tdms_index file, and you can get a mix up of various data-formats, the solution is always easy: delete the tdms_index file (it will be auto-generated again upon opening the file).

There is a discussion of HDF5 vs. TDMS, from what I remember is that at the moment of development HDF5 wasn't fast enought for single-writes.
(Idea-exchange link, but there should be more)

One thing that I like of TDMS is that it limits you to groups and channels, one way or another it's always been possible to store the relevant data in that structure. But I would love to have (at least) 2-dimensional arrays.

Ton

#7 PaulG.

PaulG.

    The 500 club

  • Members
  • PipPipPipPipPip
  • 833 posts
  • Location:Greater Metro Detroit, MI, USA
  • Version:LabVIEW 2011
  • Since:1998

Posted 29 December 2011 - 10:26 PM

Hooovahh's summary is better than anything I can offer. For flexibility and performance you can't beat TDMS, or at least I don't know of any other way to store huge amounts of data as efficiently and reliably. One of LabVIEW's better features. I have had no trouble with it at all.
"'Value-added data'? What is that? Numbers fortified with art"? -Jon Stewart

#8 viSci

viSci

    Extremely Active

  • Premium Member
  • 334 posts
  • Version:LabVIEW 2013
  • Since:1998

Posted 30 December 2011 - 02:15 PM

TDMS Rocks! I find it to be very fast for writes and reads. So fast that I now use it as a core data store for my DAQ applications that need to be able to acquire data and simultaneously read out and display the data. You can easily emulate the kind of real time historical data viewer that you get in the distributed system manager using this capability of TDMS.

#9 COsiecki

COsiecki

    Active

  • Members
  • Pip
  • 19 posts
  • Location:San Antonio TX
  • Version:LabVIEW 2009
  • Since:2006

Posted 04 January 2012 - 03:46 PM

TDMS is a fantastic format. Because I wrote almost all of the DAQ software we have, I have replaced the custom formats we used with TDMS. The ease of use through LabVIEW makes it almost as simple as adding an indicator to your front panel. It's also very powerful for adding metadata. You don't have to figure out where you are going to put some new property in your file that doesn't mess with your offsets, or if you can add another channel without needing to re-write your reader.

We have had some issues using TDMS in MATLAB. As far as I know there are is no official support for TMDS by Mathworks (they probably see NI as a competitor). One way to do it is to use the TDMS dll and do library function calls, but those can be finicky and can cause MATLAB to crash to desktop. The other problem we had with that method is that, at least when we were using it, the 64-bit version of MATLAB had no built in way to use a dll and you had to link it to an external compiler. It was not an ideal solution. In the end, we wound up using the TDMS format white paper to write a MATLAB-native reader, but there are some data types that aren't supported and we are lacking a number of nice features. It works for the way we tend to use the files we create though.

#10 JamesMc86

JamesMc86

    Extremely Active

  • Members
  • PipPipPipPip
  • 253 posts
  • Location:UK
  • Version:LabVIEW 2013
  • Since:2006

Posted 14 January 2012 - 10:25 AM

I think all the points above are great. I always get a good reaction when showing it to customers and the excel importer can produce easier data to go though that your own text files.

One thing to just be aware of with it is fragmentation. It's a common problem that we see, esp. on embedded platforms where file size is more sensitive. There is a white paper describing the exact implementation but the short version is that each time you use TDMS Write a "chunk" of data is written to file (Im not sure if this is the proper term). If that chunk matches the previous (same channels, datatype, I think size) then it is written just as binary data and maintains the efficiencies. If it does not match then it writes a new piece of XML header data with the data, which is very inefficient. The result is if you mix the reads to a single file your XML header becomes huge, as does your file and your access becomes slow.

The solution is to try and keep to a single write function to stream to disk. If it is unavoidable then there is a TDMS defrag function which will then reduce the size again but this can take some time and CPU.

Don't misunderstand me though, this is the main caveat you could run into but overall the benefits outweigh this (and it is avoidable as long as you know it exists!) you do get extremely compact, searchable data that can be easily shared. To get similar write performance you would have to go to a binary format which means the end user will need a program capable of reading that.

Cheers,
James

James McNally

CLA

www.wiresmithtech.com/blog


#11 hooovahh

hooovahh

    Im no supervising technician Im a technical supervisor

  • Moderators
  • 1,457 posts
  • Version:LabVIEW 2013
  • Since:2004

Posted 27 January 2012 - 08:43 PM

One thing to just be aware of with it is fragmentation. It's a common problem that we see, esp. on embedded platforms where file size is more sensitive. There is a white paper describing the exact implementation but the short version is that each time you use TDMS Write a "chunk" of data is written to file (Im not sure if this is the proper term). If that chunk matches the previous (same channels, datatype, I think size) then it is written just as binary data and maintains the efficiencies. If it does not match then it writes a new piece of XML header data with the data, which is very inefficient. The result is if you mix the reads to a single file your XML header becomes huge, as does your file and your access becomes slow.

The solution is to try and keep to a single write function to stream to disk. If it is unavoidable then there is a TDMS defrag function which will then reduce the size again but this can take some time and CPU.

This is true but one work around is to set the buffer size of the channel before writing the first time. This will allow you to use the write function several times without it needing to actually write to disk, which means it will not need to write the XML header. The down side is the data is not actually written to disk, and if power is removed from the PC then your buffer is lost. For this reason I try to make my buffer size appropriate, so I'm not writing XML header data all the time, but I also don't have a buffer so large that I don't write all the data at once (unless that is desired).

http://zone.ni.com/r...dms_buffersize/
  • asbo likes this

"Maybe Hoovah is really Crelf's alter-ego, which he uses to irk people?" - Gary Rubin
"Seemingly minor changes....can mean the difference between a working app and a quivering heap of unresponsive code." - David Boyd
"Using unofficial features of LabVIEW is cat nip laced with poison."
"3rd party VIs with block diagrams that are locked are useless. You see the problem all lies in the phrase "...if the VIs do what you want...". They typically don't and with locked diagrams you can't fix the problems." - Mike Porter
"If you really need a real-time system, then you really need a real-time system." - Hooovahh


#12 ASTDan

ASTDan

    Extremely Active

  • Members
  • PipPipPipPip
  • 343 posts
  • Version:LabVIEW 2013
  • Since:1996

Posted 27 January 2012 - 09:46 PM

I would only recomend TDMS for a customer who needs the high streaming, and is very comfortable with NI tools.

I recomend for the most part saving data in simple ASCII files. The reason I like ASCII files.

1. Human Readable
2. Can be read by anything (Excel, Diadem, Matlab, any data viewing program worth its salt)
3. Been around for forever and will be around forever.
4. Non engineer follks feel comfortable working with ASCII files
5. VERY easy to use.

The knock on ASCII files is they can be slow and big. My usual answer is hard drives are cheap, and I can design the program to save data in a managable amount. That being said I have seen data requirements where TDMS would be a good solution.

The cons for TDMS

1. NI propiatary. NI can change the format when they want. We have already been through TDM, TDMS version 1, and TDMS version 2. I think the format is stable for know, however it is something to be aware of.
2. Non Human Readable
3. Outside of the NI ecosphere who has heard of TDMS?
4. Only works with NI Diadem and LabVIEW (Can download a free plug in for Excel, but that is a pain for folks. You will get a phone call everytime somebody wants to open the data)
5.Learning curve

In short my default file format is ASCII. If you need the high speed streaming, lower footprint, and are comfortable using NI tools TDMS could be a nice fit.
  • dannyt likes this

#13 Ton Plomp

Ton Plomp

    How many lines per hour? Zero!

  • Members
  • PipPipPipPipPipPip
  • 2,005 posts
  • Location:New Zealand
  • Version:LabVIEW 2012
  • Since:2000

Posted 28 January 2012 - 03:25 PM

We recently sent data in TDMS format to another party (sort of competition), and we decided not to mention the dataformat, jus sent the files.
Later I spoke one of there developers, and and I figured out they were using Matlab were able to open the files without any issue.

Ton

#14 viSci

viSci

    Extremely Active

  • Premium Member
  • 334 posts
  • Version:LabVIEW 2013
  • Since:1998

Posted 28 January 2012 - 04:38 PM

Also don't forget the TDMS Excel Plugin.

and this...

http://originlab.com...ormats&pid=1047

#15 JamesMc86

JamesMc86

    Extremely Active

  • Members
  • PipPipPipPip
  • 253 posts
  • Location:UK
  • Version:LabVIEW 2013
  • Since:2006

Posted 28 January 2012 - 04:39 PM

Hi,

Thanks for the point about the data buffers. I had not experimented much with the buffering but it certainly does improve the situation (I will upload a comparison to the NI community shortly)

EDIT: If you want to see the effect of buffers I have now uploaded code to https://decibel.ni.c.../docs/DOC-20522

I think this discussion sums up that the whole decision comes down to the fact it is going to depend where you want the data to end up. TDMS is easiest the NI ecosystem although I have had people impressed with the way the data is pulled into excel (theres also a plugin for open office) and the C dll does allow it to be added to custom applications (including Matlab). If you need the data to work with notepad or another specific program then you have to stick to with an ASCII format or a custom binary format that the program supports.

Where I would say TDMS is a must is when your talking big sizes. With the advanced API you can actually write to disk faster than with the standard binary VIs and dealing with large files requires the random access available in the API or with DIAdem's ability with large data sets.

James McNally

CLA

www.wiresmithtech.com/blog


#16 viSci

viSci

    Extremely Active

  • Premium Member
  • 334 posts
  • Version:LabVIEW 2013
  • Since:1998

Posted 28 January 2012 - 04:44 PM

How about this...

https://github.com/mnaberez/tdms

Wow, this just goes on and on...

http://pytdms.sourceforge.net/

#17 Stobber

Stobber

    Very Active

  • Members
  • PipPipPip
  • 119 posts
  • Location:Austin, TX USA
  • Version:LabVIEW 2011
  • Since:2005

Posted 26 June 2012 - 05:22 PM

HDF5 seems conspicuously absent from the discussions and comparison charts.

I'm surprised nobody else noticed this. NI used to have an API for a subset of HDF5 that shipped with their Modular Instruments (Scope, FGEN, HSDIO) drivers called NI-HWS. It was pretty solid but needed some bug fixes that never got attention. The HWS format, and all HDF5 support along with it, was ultimately canned in favor of promoting TDMS as the platform-supported file format for waveform data storage. You won't hear anyone from NI marketing talking about HDF5 again.

Most of the stuff I readed @ http://zone.ni.com/d...a/tut/p/id/3727 (sorry had to read first before asking :oops: ), it then made me more curious about the disadvantages... because how NI describes it, it is the best format available :P which I doubt.

Again, I'm surprised nobody else has mentioned the data types that TDMS doesn't support. One I can immediately think of is multi-dimensional (>3 dims) arrays: TDMS doesn't support them, while HDF5 does.

Oh, another is that you can't put array or cluster types into TDMS properties. That can lead to some unfortunately creative restructuring of your metadata in order to facilitate the format.

Edited by Stobber, 26 June 2012 - 05:27 PM.


#18 Zahrah Chaumun

Zahrah Chaumun

    3 more posts to go.

  • Members
  • 7 posts
  • Version:LabVIEW 2013
  • Since:2014

Posted 04 February 2014 - 07:23 PM

Hello 

I really need help concerning TDMS. I don't have any idea how to use that to make database and store information such as

student name, time of borrower equipment, which equipment are present in the laboratory. Please help

thank u.



#19 crossrulz

crossrulz

    Extremely Active

  • Members
  • PipPipPipPip
  • 317 posts
  • Location:Cincinnati, OH
  • Version:LabVIEW 2012
  • Since:2004

Posted 04 February 2014 - 08:03 PM

*
POPULAR

TDMS is setup for waveform data.  For what you are referring to, you need an actual database, not TDMS.


  • Stobber and Zahrah Chaumun like this

#20 Stobber

Stobber

    Very Active

  • Members
  • PipPipPip
  • 119 posts
  • Location:Austin, TX USA
  • Version:LabVIEW 2011
  • Since:2005

Posted 04 February 2014 - 08:08 PM

TDMS is setup for waveform data.  For what you are referring to, you need an actual database, not TDMS.

Correct. Zahrah, I recommend using http://lavag.org/fil...sqlite-labview/ instead.


  • Zahrah Chaumun likes this