How to increase performance on lage configuration INIs?

[flowschi]

Hello,

I am using the variantconfig.llb for saving/loading configuration files.
Unfortunately these configuration-files got very huge: 10.000 lines or even more, having hundreds of sections.

Loading these files takes 15-30 seconds. The workflow:

- getting all sections with "Get Section Names.vi" (is very fast)
- a for-loop is using "Read Section Cluster_ogtk.vi" on each section Name (using Profiler, I can see that this uses the most amount of time). Depending on the section name, I am selecting the cluster type.

 

Do you have any hints, how I can speed up the process?

Thanks!

Edited May 18, 2016 by flowschi

[Phillip Brooks]

It depends on the structure of your data. The biggest hit reported by most users is when reading an array of clusters.

http://forums.ni.com/t5/LabVIEW/Saving-Clusters-to-a-Config-File-Using-Variants/td-p/1586500

There is a VI collection called Read/Write Anything from Moore Good Ideas that might help.

http://www.mooregoodideas.com/readwrite-anything-vis/

WIth such a large data file, you may want to consider switching to a different format, like XML or maybe even a database.

[ShaunR]

• Buy an SSD?
• Only load the sections that are not the same as the LabVIEW in-memory defaults?
• Only load what you need when you need it (just in time config)?
• Split out into multiple files? (diminishing returns)
• Refactor to to use smaller configurations?

Lets face it. 15-30 seconds to load 10,000 line inifiles is pretty damned good for a convenience function where you are probably stuffing complex clusters into it. What is your target time?

[drjdpowell]

OpenG was written before proper recursion existed in LabVIEW, and uses (slow) VI-Server calls to dive into clusters.  MGI is more recent and uses proper recursive calls, so should be significantly faster.  

Personally, I use JSON for configuration files.

[Yair]

Like I said in the thread Philip linked to (there should be some actual numbers in one of my older threads on LAVA, but I don't feel like looking it up), in my testing I saw that the main slowdown in the OpenG VIs came from the NI VIs and switching to the MGI VIs had a performance difference of ~10-20X.

The basic functional difference between the two libraries is that the NI INI VIs let the OpenG library handle existing files and modify items within them, whereas the MGI VIs simply overwrite the entire file. At least, that was the case ~8-9 years ago, when I did this. Since then many things have changed (like the internal structure of the NI INI VIs).

[hooovahh]

Yup another vote for MGI.  I believe it does read/write the whole file so some trickery might be needed to make it as modular as the OpenG calls.

The reason it is slow is because the OpenG functions call the NI INI functions under the hood.  And each line in the text file is a call to the NI function, where the MGI write their own INI parsing code, and reading and writing is just one call to a file I/O primitive.  Basically if you have a file that is 5KB in size or so either will work fine, but if you get clusters of arrays of clusters and your output file is larger than that, then you'll end up with slow downs.  Another option I used in the past was to flatten large structures into binary blobs.  This would mean the OpenG INI writing would only need one call because that binary section would be just one line.  Of course the down side is that data type won't be human readable in the file, and changes to the data type will cause errors, where the INI parsing does a decent job at preserving it.

[smithd]

If I remember correctly the NI ini code reads the whole file when you open it, but it keeps all the values in a single array (no var attribute lookup) so if you do random access its real slow. I've seen similar performance issues with openG, mgi and the json libraries (on different scales, of course), and I've also seen it with the labview xml and json functions. I personally think the only real right answer is:

Quote

Lets face it. 15-30 seconds to load 10,000 line inifiles is pretty damned good for a convenience function where you are probably stuffing complex clusters into it.

 

 

Edited May 20, 2016 by smithd

[drjdpowell]

20 hours ago, smithd said:

I personally think the only real right answer is:

That 15-30 sec figure, at 1.5 to 3 ms per line, seems an order of magnitude too slow to me.