Jump to content

What is the best file format?


Recommended Posts

QUOTE (rie88 @ Mar 19 2009, 06:03 AM)

what is the smallest file format in size??????

cause i have to do continious measurement for six month.

Hi,

Well.. as you question is quite general its not easy to give a good answer.

Maybe before choosing a file format you should try to estimate the amount of data you have to save during the 6 month.

What are you measuring? How many channels do you have? What's the acquisition rate?

Do you really need to generate a file? How about saving to a database? Do you need to be able to read the "old" data and keep writing data at the same time ?

You'll probably have to make some trade off, describe your application so that people here can give you useful answers.

Link to post
Share on other sites

QUOTE (rie88 @ Mar 19 2009, 07:03 AM)

what is the smallest file format in size??????

cause i have to do continious measurement for six month.

For such a general question I would answer: 7-zip.

Store your measurements in any way you like and automaticaly once a day or a week pack them to 7z by calling 7-zip command line executable.

Link to post
Share on other sites

QUOTE (vugie @ Mar 19 2009, 08:17 AM)

For such a general question I would answer: 7-zip.

Store your measurements in any way you like and automaticaly once a day or a week pack them to 7z by calling 7-zip command line executable.

Yes, a zip file will probably be the smallest version of a data file you can have, but I would stick with native LV zip file functions.

Link to post
Share on other sites

QUOTE (vugie @ Mar 19 2009, 05:38 AM)

I think the chart referenced in the link above explains why more people don't use 7-zip - you get about a 24% reduction in file size compared to zip but you pay a performance penalty in time of over 300%. Also, I would expect zip is more widely supported - does the native windows zip know what to do with a 7-zip file? I don't know.

But back to the original question - we need to know more about your data to offer any useful suggestions. For instance, if you're storing waveform info scanned from a 16-bit card then you can get full fidelity by storing a waveform header with scaling info and store each data point in two bytes. If you just dumped that data as doubles, you'll use four times as much space.

Mark

Link to post
Share on other sites

QUOTE (mesmith @ Mar 19 2009, 02:53 PM)

I think the chart referenced in the link above explains why more people don't use 7-zip - you get about a 24% reduction in file size compared to zip but you pay a performance penalty in time of over 300%. Also, I would expect zip is more widely supported - does the native windows zip know what to do with a 7-zip file? I don't know.

[OT]

Note that these tests were performed with relatively small files (13 MB in total). In this case 300% means 40 s :)

These results does not scale linearly with size. I use 7-zip everyday compressing files from 20 MB to over 200 MB and I don't feel this penalty. And sending 30% smaller files over internet is 30% faster....

[/OT]

Link to post
Share on other sites

QUOTE (mesmith @ Mar 19 2009, 04:53 AM)

we need to know more about your data to offer any useful suggestions.

That's a key point. Just to play the contrarian, do you really need to compress the data at all? If you don't compress the data, how big will your files be after six months or a year?

Disk space is practically free. You can get a 1TB external drive for $100. Unless you're storing more than that amount of data (or need to transport it on a thumb drive or something), just buy one of those. If your data is uncompressed it's also easier to work with. You don't have to go through the gymnastics of going through a zip layer every time you want to look at it.

Link to post
Share on other sites

QUOTE (Justin Goeres @ Mar 19 2009, 09:38 AM)

Unless you're storing more than that amount of data (or need to transport it on a thumb drive or something), just buy one of those. If your data is uncompressed it's also easier to work with. You don't have to go through the gymnastics of going through a zip layer every time you want to look at it.

Agreed, I think the quick answer to the question would be save it as a binary file (assuming you don't need it to be human readable) and the file size will be pretty smal.

If this was a compression thread (which it didn't start as) I am also on the 7-zip bandwagon. I even tried making a API for LabVIEW but realized with my limited knowledge of DLL calls my only solution would be to use the command prompt to run the 7-zip command line utility. It would be a decent solution but would be operating system specific.

Link to post
Share on other sites

QUOTE (hooovahh @ Mar 19 2009, 10:16 AM)

Agreed, I think the quick answer to the question would be save it as a binary file (assuming you don't need it to be human readable) and the file size will be pretty smal.

If this was a compression thread (which it didn't start as) I am also on the 7-zip bandwagon. I even tried making a API for LabVIEW but realized with my limited knowledge of DLL calls my only solution would be to use the command prompt to run the 7-zip command line utility. It would be a decent solution but would be operating system specific.

Thread hijack. :ninja: Why would you want to mess around with command lines and a 3rd-party, open source (be afraid be very afraid) program when you can use native LV zip functions?

Link to post
Share on other sites

QUOTE (PaulG. @ Mar 19 2009, 01:37 PM)

Thread hijack. :ninja: Why would you want to mess around with command lines and a 3rd-party, open source (be afraid be very afraid) program when you can use native LV zip functions?

Why be afraid of open source? What's there to be afraid of? I don't like the built in LV zip functions, I haven't had much luck with them even on the simplest functions. Also for advanced features...there aren't any. If nothing else use the OpenG ones. But if you want high compression use 7-Zip.

Link to post
Share on other sites

QUOTE (hooovahh @ Mar 19 2009, 01:54 PM)

Why be afraid of open source? What's there to be afraid of? I don't like the built in LV zip functions, I haven't had much luck with them even on the simplest functions. Also for advanced features...there aren't any. If nothing else use the OpenG ones. But if you want high compression use 7-Zip.

I'm volunteering you to join the open source space shuttle control softare development team. And you get a seat on the first flight. Kick the tires and light the fires! :thumbup:

Link to post
Share on other sites

QUOTE (crelf @ Mar 19 2009, 07:03 PM)

Then, join the Future of LabVIEW programming.

QUOTE (PaulG. @ Mar 19 2009, 08:22 PM)

I'm volunteering you to join the open source space shuttle control software development team. And you get a seat on the first flight. Kick the tires and light the fires! :thumbup:

Ditto, some things are just too important to leave to willing enthousiasts.

Ton

Link to post
Share on other sites

QUOTE (Antoine Châlons @ Mar 19 2009, 02:54 PM)

Hi,

Well.. as you question is quite general its not easy to give a good answer.

Maybe before choosing a file format you should try to estimate the amount of data you have to save during the 6 month.

What are you measuring? How many channels do you have? What's the acquisition rate?

Do you really need to generate a file? How about saving to a database? Do you need to be able to read the "old" data and keep writing data at the same time ?

You'll probably have to make some trade off, describe your application so that people here can give you useful answers.

I'm sorry, it takes a long time before I can post again.

here i will explain about my project.

I need to do continious measuring for six month. I use DAQpad 6020e. Of course i have 16 channels, and I want to use them all.

For the acquisition rate is about 100kS/s.

Actually I want to get the waveform data, and then save it into file. I want to be able to read the old data again if there is a problem occur. So it will be large amount of data in drive. what should i do to manage the data??

thanks.........

Link to post
Share on other sites
Link to post
Share on other sites

QUOTE (rie88 @ Mar 24 2009, 12:22 AM)

I need to do continious measuring for six month. I use DAQpad 6020e. Of course i have 16 channels, and I want to use them all.

For the acquisition rate is about 100kS/s.

Actually I want to get the waveform data, and then save it into file. I want to be able to read the old data again if there is a problem occur. So it will be large amount of data in drive. what should i do to manage the data??

I would use tdms files in this situation. They are fast, compact, and indexed for faster viewing of incremental data. Search ni.com for tdms and you'll find quite a bit of information.

Link to post
Share on other sites

I would ask the question is it really necessary to save every piece of data. You are saving the history so you can look back when and if a problem occurs. Using statistical analysis you can probably determine a sufficient sample size for your historical data that would allow you to see what you want without the need for saving every bit, or more accurately byte of data. Yes it seems like reasonable to want to be able to look at every measurement you ever took but as you see it is not always practical. I am sure you could determine a reasonable sample size that you would need to save that would provide you with a high enough confidence level that you are seeing an accurate representation of your historical data.

Think of polling. If you use a truly random and unbiased selection for your sample you can say with a specified degree (95%, 98%, 99.9%) of accuracy you have a accurate representation of your total population.

Link to post
Share on other sites

Or you could save to file at a reduced sample rate.

Or use filtering to save phenomena of interest (peak values) at a reduced sample rate.

Basically you need to analyse your requirements BEFORE you start collecting Terabytes of data. If there is a "problem" how much time are you willing to spend to wade through multiple hard-drives worth of data?

Neville

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.