process logfiles takes too long

[RagingGoblin]

I want to process large logfiles. The lines inside the logfile are like these:

A 26072006 142451.376 0728 IncomingFile IFSERV 00000000 DETECT DVB

A 26072006 142501.344 0728 IncomingFile IFSERV 00000000 DETECT H-000-MSG1__-MSG1________-HRV______-000016___-200607261415-C_

A 26072006 142501.344 0728 IncomingFile IFSERV 00000000 DETECT DVB

A 26072006 142511.407 0728 IncomingFile IFSERV 00000000 DETECT H-000-MSG1__-MSG1________-HRV______-000017___-200607261415-C_

A 26072006 142511.407 0728 IncomingFile IFSERV 00000000 DETECT DVB

A 26072006 142521.376 0728 IncomingFile IFSERV 00000000 DETECT H-000-MSG1__-MSG1________-IR_016___-000006___-200607261415-C_

As you can see first there is a timestamp which I want to read in order to process the stuff. I wrote a vi which reads a logfile line by line, because the files are too big to read entirely at once (20 MB). If I run this vi the proces takes 20 minutes!!! Thats way too long. Does anyone know a way to speed this up?

see untitled.vi

Download File:post-4151-1153928564.vi

Download File:post-4151-1153928590.vi

[Chris Davis]

I want to process large logfiles. The lines inside the logfile are like these:

A 26072006 142451.376 0728 IncomingFile IFSERV 00000000 DETECT DVB

A 26072006 142501.344 0728 IncomingFile IFSERV 00000000 DETECT H-000-MSG1__-MSG1________-HRV______-000016___-200607261415-C_

A 26072006 142501.344 0728 IncomingFile IFSERV 00000000 DETECT DVB

A 26072006 142511.407 0728 IncomingFile IFSERV 00000000 DETECT H-000-MSG1__-MSG1________-HRV______-000017___-200607261415-C_

A 26072006 142511.407 0728 IncomingFile IFSERV 00000000 DETECT DVB

A 26072006 142521.376 0728 IncomingFile IFSERV 00000000 DETECT H-000-MSG1__-MSG1________-IR_016___-000006___-200607261415-C_

As you can see first there is a timestamp which I want to read in order to process the stuff. I wrote a vi which reads a logfile line by line, because the files are too big to read entirely at once (20 MB). If I run this vi the proces takes 20 minutes!!! Thats way too long. Does anyone know a way to speed this up?

see untitled.vi

You don't have to have the open file and close file outside the while loop, the "read line" vi will open and close the file by itself. You might also setup the read line vi to read 1000 lines of the log file and process those, then read 1000 more and so on, to see if that speeds up your code.

Of course you don't mention what speed computer you are trying to process these log files on, and that would make a difference if you are running labview on a resource starved machine.

[eaolson]

I want to process large logfiles. The lines inside the logfile are like these:

A 26072006 142451.376 0728 IncomingFile IFSERV 00000000 DETECT DVB

You might want to look into the Scan From String function rather than making all those string subsets and formatting them into numbers manually. It would also make your diagram a lot simpler. For the "26072006 142451.376" string above, a format string of "%2d%2d%4d %2d%2d%2d%.3f" would parse it into the appropriate numbers from which you could then build a date/time cluster.

[Ton Plomp]

Hallo Ealson,

you could even format the string right into a timestamp:

Good luck,

Ton

PS 20 MB shouldn't be problem if you program properly.

[eaolson]

you could even format the string right into a timestamp:

I didn't realize you could do that. It doesn't work for me in 7.1.1 for the full string, but does in 8.0.1.

[Tom Limerkens]

What you do best, is keep the file open,

and just do a read and parse line by line.

When you enable the 'line mode' in the read file, it reads until the EOL character.

This way the reading goes a lot faster, because the OS doesn't need to close/open the file,

and search line position for every line read.

Combine this with the scan from string from posts above, and your parsing will go a lot faster.

Tom

(In attachment the changed 'Reader')

Download File:post-1135-1153986216.vi

[RagingGoblin]

Thanks for all solutions, but but I read the file from a network place !!! :headbang: This was causing the delay :headbang:

Anyway allso tried the scan from file function, at least that looks much better.

I allways work from network since a heavy fire in a building of our department caused many projects to loose all their stuff. I shoud have thought of this.

Now with all these timers I could really see what is the fastest solution.

With the scan from file function, I get a delay ??? First it took 9000 ms, now it takes 9300 ms.

Tom nice solution: It takes now only 1000 ms.