Log File Structure

[rharmon@sandia.gov]

I have an error log and a couple other logs associated with a program I've written. My question is this, when you read a log file, do you expect the newest information at the beginning of the file or the end.

I think I prefer it at the beginning of the file... That said I worry about large log files. The only way I figured out how to accomplish this was to read the entire file, add my new entry at the beginning and then re-write the entire file. Seems like a lot of overhead just for a style preference. 

Is there a better way to accomplish writing new entries at the beginning? Or does everyone prefer new log entries at the end of the file?

thanks for you comments and ideas...

Bob Harmon

[JKSH]

50 minutes ago, rharmon@sandia.gov said:

The only way I figured out how to accomplish this was to read the entire file, add my new entry at the beginning and then re-write the entire file. Seems like a lot of overhead just for a style preference.

That is the only way to add data to the start of a file. This is due to the way filesystems are designed: A file can be easily extended beyond its current end point, but its start point can't be moved.

 

50 minutes ago, rharmon@sandia.gov said:

Is there a better way to accomplish writing new entries at the beginning?

Not to the file itself.

However, rather than adding to the start of your file, you could write a simple log viewer app that reads the file and displays the entries on screen in reverse order.

 

 

50 minutes ago, rharmon@sandia.gov said:

My question is this, when you read a log file, do you expect the newest information at the beginning of the file or the end.

I think I prefer it at the beginning of the file...

Personally, I'm so used to logs having newer entries at the end that I don't expect it the other way. I guess I prefer this out of habit and due to the efficiency of appending data to the end.

Having said that, I understand the convenience of having the latest data being visible as soon as the file is opened.

Edited January 19, 2021 by JKSH

[Darin]

Log files are why Linux has the 'tail' function and Powershell has the -Tail option for the Get-Content command.  For a straight log file it would never occur to me to put new entries in the beginning, just use tail.

The other alternative is to log to a database instead of a simple file, but in my view that often adds a lot of moving parts for no real benefit.

[dhakkan]

Assuming these log files are to be consumed for application support, here are a couple of more possibilities for viewing logs saved in chronological order.

Use 'tac' (reverse of cat) from the terminal. This differs from 'tail' command in that tac presents lines in the reverse order, whereas tail presents lines in the same order as saved in the file, but from an offset relative to the End-Of-File marker. tac is built-in for linux terminals. I don't know of Windows command or powershell built-in equivalents - I couldn't find anything from a cursory search. GNU utilities can be downloaded for Windows for the same tac tool.

If you use Notepad++ (Windows only), this link has a built-in approach requiring a few steps.

EDIT 1: Reg. 'large' log files, log rotation may help. Linux has 'logrotate' available. For Windows, I just implemented a rudimentary function in LabVIEW. File size is checked periodically. When a predefined threshold size is exceeded, the existing file is flushed, closed and moved to an incrementally numbered file based on already existing numbered files in the same folder.

Edited January 19, 2021 by dhakkan
Addressing another point

[rharmon@sandia.gov]

You guys always have the best information/ideas... Thank you all...

Since I really like the new entries at the top of the log file, and my major worry is that the file gets too big over time and causes the the log write to consume too much time I really like dhakkan's approach of checking the file size periodically and flushing the file and saving the data to numbered files.

[Neil Pate]

18 hours ago, rharmon@sandia.gov said:

You guys always have the best information/ideas... Thank you all...

Since I really like the new entries at the top of the log file, and my major worry is that the file gets too big over time and causes the the log write to consume too much time I really like dhakkan's approach of checking the file size periodically and flushing the file and saving the data to numbered files.

I totally get you wanting to display newer entries first, but surely this is just a presentation issue and should not be solved at the file level? I really think you are going against the stream here by wanting newer entries first in the actual file on disk. It is almost free to append a bit of text to the end of a file, but constantly rewriting it to prepend seems like a lot of trouble to go to.

Rotating the log file is a good idea regardless though.

Notepad++ has a "watch" feature that autoreloads the file that is open. It is not without its warts though as I think the Notepad++ window needs focus.

Edited January 20, 2021 by Neil Pate