Prepend a file?

[Mike Matthes]

I have an ASCII file (unfortunately) that contains header information such as test parameters, results, etc. at the top of the file. This is followed by a somewhat large tab delimited data table. This is easy to do post formatting and save the file once, but now I have the need to save the data file on the fly. The problem is the header information changes as the test finishes.

Making the end of the file appends is easy - just append to the end and keep going. Is there a way to prepend a file - ie change the length in the other direction?

I'll probably have to split off a header file and a data file, but of course I like to do things the fancy hard way if possible.

[crelf]

You could save the data on the fly to temporary file, then once the test is finished create the data file with the header an then concatenate the temporary file tot he end. Not very elegant, but it'll get you up and runinng...

[JDave]

You could save the data on the fly to temporary file, then once the test is finished create the data file with the header an then concatenate the temporary file tot he end. Not very elegant, but it'll get you up and runinng...

I think it is quite elegant. I have used this and it provides you with a lot of flexibility of naming files after the fact, optionally saving files (i.e. deleting the temporary and not creating the file w/ header), and putting in header comments that reflect what happened during a data run. If this is the environment you work under, then it works quite well.

EDIT - I forgot to add that for large files you can write a fixed length blank header as big as you will ever need. A 1k size header on a 10MB file is hardly a drop. Then you can write the true header over the blank one after the fact and resave the file.

[crelf]

I think it is quite elegant.

Yes - you're right, depending on your environment. I suppose what I was trying to get at was that it isn't very elegant to use a disk as a temporary storgae area when you could do it in memory. That said, it is certainly beneficial if your tests are long and there's a possiblility of loosing power or an error throwing out all of your interim data.

[JDave]

it is certainly beneficial if your tests are long and there's a possiblility of loosing power or an error throwing out all of your interim data.

I was going to mention that -- I'm glad you did. This could be the main reason to use a temp file, I suppose.

[jpdrolet]

I think it is quite elegant. I have used this and it provides you with a lot of flexibility of naming files after the fact, optionally saving files (i.e. deleting the temporary and not creating the file w/ header), and putting in header comments that reflect what happened during a data run. If this is the environment you work under, then it works quite well.

EDIT - I forgot to add that for large files you can write a fixed length blank header as big as you will ever need. A 1k size header on a 10MB file is hardly a drop. Then you can write the true header over the blank one after the fact and resave the file.

I agree with that. Reserve a fixed length header in the file. Files are random access, you can rewrite the header every time is is needed.

[Mike Matthes]

I agree with that. Reserve a fixed length header in the file. Files are random access, you can rewrite the header every time is is needed.

I know I can "concatenate" by opening the 2nd file, reading it, and appending it to the first. I could do this in chunks to reduce memory use, in the case that the file is huge. This will take time (one of the problems I'm trying to solve).

The "reserved" space option is also one to consider, and probably the most efficient for processing time. Unfortunately, that could leave a some dead space. I will likely have to live with that and use this solution.

So far I haven't heard an equivelant solution to "appending" a file, where you do not have to read in file chunks to memory. A file on disk is just a bunch of fragments. It seems like windows could add a new fragment and change the file table on disk appropriately. I'm 99.9% sure Labview doesn't support this. I'm wondering if somebody has tackled this some fancy way.

Thanks for the suggestions - they are most likely the easy options available at this time.

[Mike Ashe]

So far I haven't heard an equivelant solution to "appending" a file, where you do not have to read in file chunks to memory. A file on disk is just a bunch of fragments. It seems like windows could add a new fragment and change the file table on disk appropriately. I'm 99.9% sure Labview doesn't support this. I'm wondering if somebody has tackled this some fancy way.

Once upon a time, back in BioBench development we did the header as a "trailer". The last 4 bytes of the file gave the header size, which we would read off the _end_ of the file by rewinding the file index pointer that far. That way we had an arbitrary length "header" on an arbitrary length file. The header gave descriptions and indexes back into the file for channels, data, etc.

When we started to append new data, we would first read the "trailer" into memory, then append new data starting where the old header-trailer started. When we finished with data we would measure the size of the new header-trailer, write the trailer-header, then write the size.

Only downside is if you have a power loss during your data run.

YMMV

[JDave]

The "reserved" space option is also one to consider, and probably the most efficient for processing time. Unfortunately, that could leave a some dead space. I will likely have to live with that and use this solution.

If the files really are huge, then dead space is not much of an issue. It can't affect your overall file size measurably. I know people don't like wasted space, but if you design the file structure that way the "dead space" converts to "reserved space".

[Rolf Kalbermatter]

So far I haven't heard an equivelant solution to "appending" a file, where you do not have to read in file chunks to memory. A file on disk is just a bunch of fragments. It seems like windows could add a new fragment and change the file table on disk appropriately. I'm 99.9% sure Labview doesn't support this. I'm wondering if somebody has tackled this some fancy way.

LabVIEW does not support this because there is no possibility to do something like this from Win32 API or any of the other OS user applications APIs LabVIEW runs on AFAIK. You would have to go directly to the filesystem driver API and that is messy and prone to errors not to mention that many filesystems couldn't possibly support such a mode at all.

Rolf Kalbermatter