What is the best file format?

[rie88]

what is the smallest file format in size??????

cause i have to do continious measurement for six month.

[Antoine Chalons]

QUOTE (rie88 @ Mar 19 2009, 06:03 AM)

what is the smallest file format in size??????

cause i have to do continious measurement for six month.

Hi,

Well.. as you question is quite general its not easy to give a good answer.

Maybe before choosing a file format you should try to estimate the amount of data you have to save during the 6 month.

What are you measuring? How many channels do you have? What's the acquisition rate?

Do you really need to generate a file? How about saving to a database? Do you need to be able to read the "old" data and keep writing data at the same time ?

You'll probably have to make some trade off, describe your application so that people here can give you useful answers.

[vugie]

QUOTE (rie88 @ Mar 19 2009, 07:03 AM)

what is the smallest file format in size??????

cause i have to do continious measurement for six month.

For such a general question I would answer: 7-zip.

Store your measurements in any way you like and automaticaly once a day or a week pack them to 7z by calling 7-zip command line executable.

[PaulG.]

QUOTE (vugie @ Mar 19 2009, 08:17 AM)

For such a general question I would answer: 7-zip.

Store your measurements in any way you like and automaticaly once a day or a week pack them to 7z by calling 7-zip command line executable.

Yes, a zip file will probably be the smallest version of a data file you can have, but I would stick with native LV zip file functions.

[vugie]

QUOTE (PaulG. @ Mar 19 2009, 02:32 PM)

Yes, a zip file will probably be the smallest version of a data file you can have, but I would stick with native LV zip file functions.

Not just zip. http://en.wikipedia.org/wiki/Comparison_of_file_archivers#Comparison_of_efficiency' rel='nofollow' target="_blank">7-zip.

[Mark Smith]

QUOTE (vugie @ Mar 19 2009, 05:38 AM)

Not just zip. http://en.wikipedia.org/wiki/Comparison_of_file_archivers#Comparison_of_efficiency' rel='nofollow' target="_blank">7-zip.

I think the chart referenced in the link above explains why more people don't use 7-zip - you get about a 24% reduction in file size compared to zip but you pay a performance penalty in time of over 300%. Also, I would expect zip is more widely supported - does the native windows zip know what to do with a 7-zip file? I don't know.

But back to the original question - we need to know more about your data to offer any useful suggestions. For instance, if you're storing waveform info scanned from a 16-bit card then you can get full fidelity by storing a waveform header with scaling info and store each data point in two bytes. If you just dumped that data as doubles, you'll use four times as much space.

Mark

[vugie]

QUOTE (mesmith @ Mar 19 2009, 02:53 PM)

I think the chart referenced in the link above explains why more people don't use 7-zip - you get about a 24% reduction in file size compared to zip but you pay a performance penalty in time of over 300%. Also, I would expect zip is more widely supported - does the native windows zip know what to do with a 7-zip file? I don't know.

[OT]

Note that these tests were performed with relatively small files (13 MB in total). In this case 300% means 40 s

These results does not scale linearly with size. I use 7-zip everyday compressing files from 20 MB to over 200 MB and I don't feel this penalty. And sending 30% smaller files over internet is 30% faster....

[/OT]

[Justin Goeres]

QUOTE (mesmith @ Mar 19 2009, 04:53 AM)

we need to know more about your data to offer any useful suggestions.

That's a key point. Just to play the contrarian, do you really need to compress the data at all? If you don't compress the data, how big will your files be after six months or a year?

Disk space is practically free. You can get a 1TB external drive for $100. Unless you're storing more than that amount of data (or need to transport it on a thumb drive or something), just buy one of those. If your data is uncompressed it's also easier to work with. You don't have to go through the gymnastics of going through a zip layer every time you want to look at it.

[hooovahh]

QUOTE (Justin Goeres @ Mar 19 2009, 09:38 AM)

Unless you're storing more than that amount of data (or need to transport it on a thumb drive or something), just buy one of those. If your data is uncompressed it's also easier to work with. You don't have to go through the gymnastics of going through a zip layer every time you want to look at it.

Agreed, I think the quick answer to the question would be save it as a binary file (assuming you don't need it to be human readable) and the file size will be pretty smal.

If this was a compression thread (which it didn't start as) I am also on the 7-zip bandwagon. I even tried making a API for LabVIEW but realized with my limited knowledge of DLL calls my only solution would be to use the command prompt to run the 7-zip command line utility. It would be a decent solution but would be operating system specific.

[PaulG.]

QUOTE (hooovahh @ Mar 19 2009, 10:16 AM)

Agreed, I think the quick answer to the question would be save it as a binary file (assuming you don't need it to be human readable) and the file size will be pretty smal.

If this was a compression thread (which it didn't start as) I am also on the 7-zip bandwagon. I even tried making a API for LabVIEW but realized with my limited knowledge of DLL calls my only solution would be to use the command prompt to run the 7-zip command line utility. It would be a decent solution but would be operating system specific.

Thread hijack. :ninja: Why would you want to mess around with command lines and a 3rd-party, open source (be afraid be very afraid) program when you can use native LV zip functions?

[hooovahh]

QUOTE (PaulG. @ Mar 19 2009, 01:37 PM)

Thread hijack. :ninja: Why would you want to mess around with command lines and a 3rd-party, open source (be afraid be very afraid) program when you can use native LV zip functions?

Why be afraid of open source? What's there to be afraid of? I don't like the built in LV zip functions, I haven't had much luck with them even on the simplest functions. Also for advanced features...there aren't any. If nothing else use the OpenG ones. But if you want high compression use 7-Zip.

[crelf]

QUOTE (PaulG. @ Mar 19 2009, 01:37 PM)

...open source (be afraid be very afraid)...

Embrace open source, or be left behind. Open source is the future.

[PaulG.]

QUOTE (hooovahh @ Mar 19 2009, 01:54 PM)

Why be afraid of open source? What's there to be afraid of? I don't like the built in LV zip functions, I haven't had much luck with them even on the simplest functions. Also for advanced features...there aren't any. If nothing else use the OpenG ones. But if you want high compression use 7-Zip.

I'm volunteering you to join the open source space shuttle control softare development team. And you get a seat on the first flight. Kick the tires and light the fires! :thumbup:

[Justin Goeres]

QUOTE (PaulG. @ Mar 19 2009, 11:22 AM)

I'm volunteering you to join the open source space shuttle control softare development team. And you get a seat on the first flight. Kick the tires and light the fires! :thumbup:

NASA already http://news.cnet.com/8301-13772_3-10097880-52.html' rel='nofollow' target="_blank">built their issue-tracking system on Bugzilla.

[LAVA 1.0 Content]

QUOTE (crelf @ Mar 19 2009, 07:03 PM)

Embrace open source, or be left behind. Open source is the future.

Then, join the Future of LabVIEW programming.

QUOTE (PaulG. @ Mar 19 2009, 08:22 PM)

I'm volunteering you to join the open source space shuttle control software development team. And you get a seat on the first flight. Kick the tires and light the fires! :thumbup:

Ditto, some things are just too important to leave to willing enthousiasts.

Ton

[crelf]

QUOTE (Ton @ Mar 19 2009, 04:07 PM)

Then, join the http://www.openg.org' rel='nofollow' target="_blank">Future of LabVIEW programming.

I'm already there.

[rie88]

QUOTE (Antoine Châlons @ Mar 19 2009, 02:54 PM)

Hi,

Well.. as you question is quite general its not easy to give a good answer.

Maybe before choosing a file format you should try to estimate the amount of data you have to save during the 6 month.

What are you measuring? How many channels do you have? What's the acquisition rate?

Do you really need to generate a file? How about saving to a database? Do you need to be able to read the "old" data and keep writing data at the same time ?

You'll probably have to make some trade off, describe your application so that people here can give you useful answers.

I'm sorry, it takes a long time before I can post again.

here i will explain about my project.

I need to do continious measuring for six month. I use DAQpad 6020e. Of course i have 16 channels, and I want to use them all.

For the acquisition rate is about 100kS/s.

Actually I want to get the waveform data, and then save it into file. I want to be able to read the old data again if there is a problem occur. So it will be large amount of data in drive. what should i do to manage the data??

thanks.........

[Antoine Chalons]

QUOTE (rie88 @ Mar 24 2009, 08:22 AM)

I'm sorry, it takes a long time before I can post again.

here i will explain about my project.

I need to do continious measuring for six month. I use DAQpad 6020e. Of course i have 16 channels, and I want to use them all.

For the acquisition rate is about 100kS/s.

Actually I want to get the waveform data, and then save it into file. I want to be able to read the old data again if there is a problem occur. So it will be large amount of data in drive. what should i do to manage the data??

thanks.........

100ks/s, 16 channels, 6 month... ufff

First of all, as I understand the datasheet 100ks/s have to be shared between the 16 channels, so in theory that a max rate of 6.25kHz per channel, and even though I didn't find it on the datasheet I assume the actual max sampling rate with 16 channels is a bit lower than 6.25kHz.

16*6.25e3*6*31*24*3600*2 ~ 3000 Go assuming you save data as U16 (2 bytes per sample), it will be twice more if you save your data as SGL (4 bytes per sample).

It might not solve everything, but for a start you can take a look at the http://forums.lavag.org/Binary-Action-Engine-file168.html' target="_blank">Binary Action Engine, you have settings for the max file size and every time the current result file reaches the max size, a new file with incremented name is created.

Then if, during the acquisition you need to read "old" data you can easily go read old files.

Let me know if this helps you

[TobyD]

QUOTE (rie88 @ Mar 24 2009, 12:22 AM)

I need to do continious measuring for six month. I use DAQpad 6020e. Of course i have 16 channels, and I want to use them all.

For the acquisition rate is about 100kS/s.

Actually I want to get the waveform data, and then save it into file. I want to be able to read the old data again if there is a problem occur. So it will be large amount of data in drive. what should i do to manage the data??

I would use tdms files in this situation. They are fast, compact, and indexed for faster viewing of incremental data. Search ni.com for tdms and you'll find quite a bit of information.

[Mark Yedinak]

I would ask the question is it really necessary to save every piece of data. You are saving the history so you can look back when and if a problem occurs. Using statistical analysis you can probably determine a sufficient sample size for your historical data that would allow you to see what you want without the need for saving every bit, or more accurately byte of data. Yes it seems like reasonable to want to be able to look at every measurement you ever took but as you see it is not always practical. I am sure you could determine a reasonable sample size that you would need to save that would provide you with a high enough confidence level that you are seeing an accurate representation of your historical data.

Think of polling. If you use a truly random and unbiased selection for your sample you can say with a specified degree (95%, 98%, 99.9%) of accuracy you have a accurate representation of your total population.

[Neville D]

Or you could save to file at a reduced sample rate.

Or use filtering to save phenomena of interest (peak values) at a reduced sample rate.

Basically you need to analyse your requirements BEFORE you start collecting Terabytes of data. If there is a "problem" how much time are you willing to spend to wade through multiple hard-drives worth of data?

Neville