Jump to content

How to increase performance on lage configuration INIs?


Recommended Posts

Hello,

I am using the variantconfig.llb for saving/loading configuration files.
Unfortunately these configuration-files got very huge: 10.000 lines or even more, having hundreds of sections.

Loading these files takes 15-30 seconds. The workflow:

- getting all sections with "Get Section Names.vi" (is very fast)
- a for-loop is using "Read Section Cluster_ogtk.vi" on each section Name (using Profiler, I can see that this uses the most amount of time). Depending on the section name, I am selecting the cluster type.

 

Do you have any hints, how I can speed up the process?

Thanks!

profile.PNG

Edited by flowschi
Link to comment

It depends on the structure of your data. The biggest hit reported by most users is when reading an array of clusters.

http://forums.ni.com/t5/LabVIEW/Saving-Clusters-to-a-Config-File-Using-Variants/td-p/1586500

There is a VI collection called Read/Write Anything from Moore Good Ideas that might help.

http://www.mooregoodideas.com/readwrite-anything-vis/

WIth such a large data file, you may want to consider switching to a different format, like XML or maybe even a database.

  • Like 1
Link to comment
  • Buy an SSD?
  • Only load the sections that are not the same as the LabVIEW in-memory defaults?
  • Only load what you need when you need it (just in time config)?
  • Split out into multiple files? (diminishing returns)
  • Refactor to to use smaller configurations?

Lets face it. 15-30 seconds to load 10,000 line inifiles is pretty damned good for a convenience function where you are probably stuffing complex clusters into it. What is your target time?

  • Like 1
Link to comment

Like I said in the thread Philip linked to (there should be some actual numbers in one of my older threads on LAVA, but I don't feel like looking it up), in my testing I saw that the main slowdown in the OpenG VIs came from the NI VIs and switching to the MGI VIs had a performance difference of ~10-20X.

The basic functional difference between the two libraries is that the NI INI VIs let the OpenG library handle existing files and modify items within them, whereas the MGI VIs simply overwrite the entire file. At least, that was the case ~8-9 years ago, when I did this. Since then many things have changed (like the internal structure of the NI INI VIs).

Link to comment

Yup another vote for MGI.  I believe it does read/write the whole file so some trickery might be needed to make it as modular as the OpenG calls.

The reason it is slow is because the OpenG functions call the NI INI functions under the hood.  And each line in the text file is a call to the NI function, where the MGI write their own INI parsing code, and reading and writing is just one call to a file I/O primitive.  Basically if you have a file that is 5KB in size or so either will work fine, but if you get clusters of arrays of clusters and your output file is larger than that, then you'll end up with slow downs.  Another option I used in the past was to flatten large structures into binary blobs.  This would mean the OpenG INI writing would only need one call because that binary section would be just one line.  Of course the down side is that data type won't be human readable in the file, and changes to the data type will cause errors, where the INI parsing does a decent job at preserving it.

Link to comment

If I remember correctly the NI ini code reads the whole file when you open it, but it keeps all the values in a single array (no var attribute lookup) so if you do random access its real slow. I've seen similar performance issues with openG, mgi and the json libraries (on different scales, of course), and I've also seen it with the labview xml and json functions. I personally think the only real right answer is:

Quote

Lets face it. 15-30 seconds to load 10,000 line inifiles is pretty damned good for a convenience function where you are probably stuffing complex clusters into it.

 

 

Edited by smithd
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.