Help with data file format

[AndrewH]

Apologies if this is a noobie question that has been well discussed elsewhere - however a quick search didn't appear to find any obvious hits.

The problem is as follows:

I'm writing an application that will need to be able to output a large number of data files that may then be subsequently read back in for post processing. For entirely non-technical reasons the data files will be text rather than binary. Each data file will have to contain a reasonable sized header including device ID, test conditions, user ID, test equipment ID etc etc. The info in this header will be derived from typedef clusters that contain all the relevant info within the app. My issue is - how to format the data file? Also worth mentioning that there will often be a number of items that are redundant in any one given file, although which ones will vary greatly depending on which device type we are measuring.

Initially I'd looked at trying to canibalise some of the ideas from a config [ini] type file format, where one could define sections, keys and values, but I suspect that this is not sensible given the potentially large number of files that I'll need and the way that the config files use a registry.

I'd also looked at using the flatten to XML options but this gives rise to a header that is many hundreds of lines long (not least of which because there a number of typedef enums and the XML schema lists all enum cases before listing the selection.

So - how would you chaps approach the situation? I'm really after suggestions and trying to canvas opinion from the wisdom here before ploughing headlong into writing code

thanks for looking

Andrew

[ShaunR]

Whats wrong with the "Write To Spreadsheet File" in the file functions palette? Simple 2D array where the first row is the header.

Edited September 7, 2009 by ShaunR

[Mellroth]

Hi,

If you really want to go for the ini-file format, I recommend that you check the OpenG Variant Configuration File VIs.

These VIs let you read/write clusters directly to an ini-file.

Apart from the header, what kind of data are you dealing with?

/J

[AndrewH]

Whats wrong with the "Write To Spreadsheet File" in the file functions palette? Simple 2D array where the first row is the header.

That's fine for the numeric data but the headers are a whole mix of stuff including strings (which may contain spaces), booleans, enums etx. Dumping header as the first row would make automatically decoding it a nightmare and you also get the "order matters" problem that I discuss in the next para.

Sure I could dump all as simple text lines but the issue comes 'n' months down the line when it is decided to add an additional item to a cluster. If we go for a simple text file then position within the file matters. e.g. unit serial number currently on line 7 of file now moves to line 8 because an additional entry has been placed above it. And before you ask ... its not sensible just to add new lines to the bottom of the header section as this means that related data become scattered all over the header. This is why some kind of sectioned approach (e.g. INI file or XML categories) have an appeal where I could simply add a new line within a section.

If you really want to go for the ini-file format, I recommend that you check the OpenG Variant Configuration File VIs.

These VIs let you read/write clusters directly to an ini-file.

Apart from the header, what kind of data are you dealing with?

Have tried checking out OpenG but at the moment I'm having issues with VIPM in that it complains I haven't set "localhost" as a permitted server within VI server: Exported VIs and Machine access sections. Even when I have :-(

As to data type - everything outside the header is a single precision float mostly waveforms or spectra acquired from scopes or network analysers

Thanks for your thoughts guys - please keep them coming

Andrew

[Phillip Brooks]

Thanks for your thoughts guys - please keep them coming

Jim outlined a nice technique for defining a file format in his blog "Thinking in G"; Supporting Multiple Versions of a File Format

[AndrewH]

Jim outlined a nice technique for defining a file format in his blog "Thinking in G"; Supporting Multiple Versions of a File Format

Cool - thanks Phillip - I'll check it out

[ShaunR]

Many moons ago, I had a vi that was basically a wrapper around the write to spreadsheet.vi.

It was for a data logger, and since the customer didn't really know what data he wanted saved (and kept changing his mind); I wrote a fairly simple vi that read the header (first row) and compared it to the current header to be saved.

It would look at the header and, if a column was different, would insert a new column in the existing file (i.e upgrade it). If the headers were the same, it would just append the data. Old data therefore had no entry in this new column, but the new data would, but both the old and new would still be in-line with the headers.

One caveat with this was that you could not rename headers, but that wasn't a problem in my case.

[torekp]

Moore Good Ideas has wonderful VIs called Write Anything and Read Anything. These would be perfect at least for your header data, if not for all of it. They're free, but worth more than twice the price.

Sorry if this info comes late.

[AndrewH]

Moore Good Ideas has wonderful VIs called Write Anything and Read Anything. These would be perfect at least for your header data, if not for all of it. They're free, but worth more than twice the price.

Sorry if this info comes late.

No apologies needed - will check it out and see if it merits the investment

thanks to all respondents

Andrew