Lossless Floating Point compression

[Donald]

Hello All,

I would like to hear your opinion how I could compress huge datasets with floating point data (320 channel EEG recordings of 512 Hz for 24 hours 7/7).

I tried a Russian audio compressor (TTA developed for russian telescope) and encoded my file as a multichannel WAV file and was surprised to see that the compressed file is only 30% of original (with Winrar it is between 50-60 of original and winzip 60-70 % of original). The problem I have with the WAV solution that I also would like to store metadata and that all channels need to have the same sampling rate.

My next trial will be AVI based storage with lossless video codec (avi is multichannel and and I guess also can handle different sample rates as this exists for audio streams) but this is a straightforward solution. I want to use codec "Lagarith' wich is loseless.

Best Regards,

Donald

[jdunham]

QUOTE (Donald @ Apr 18 2008, 03:43 AM)

I would like to hear your opinion how I could compress huge datasets with floating point data (320 channel EEG recordings of 512 Hz for 24 hours 7/7).

I tried a Russian audio compressor (TTA developed for russian telescope) and encoded my file as a multichannel WAV file and was surprised to see that the compressed file is only 30% of original (with Winrar it is between 50-60 of original and winzip 60-70 % of original). The problem I have with the WAV solution that I also would like to store metadata and that all channels need to have the same sampling rate.

Donald: Without real analysis of your data it is hard to suggest a better codec than the general-purpose ones you have tried. Different data will have better or worse compression with different codecs. If you want to approach the limits of Shannon's Law, you will have to find algorithms which can squeeze everything predictable out of your particular kind of data.

If WAV compression (it supports many codecs, but ADPCM is pretty good for audio) is working well, I would just use it. Why don't you just keep all of your channels in separate WAV files at separate data rates, keep the metadata in a text file (XML?) and dump all of that in to a ZIP file just to keep it together as a coherent package.

In our application we put metadata in a custom section directly in the WAV file, but I have been very disappointed how poorly this is handled by other software that claims to support WAVs.