Labview Read INI File Slow

[yiweihua2002]

Dear ALL: I have a VI use OpenG Read INI Cluster subvi to read a complex cluster data structure called step config data,and when I Run this VI,It sometimes read INI File quickly,and sometimes read INI File very slowly.If I want to speed up the read INI File speed,I must quit labview and restart it,and when I quit labview it will reports error just look like Labview Error Message.JPG

[crelf]

QUOTE(yiweihua2002 @ Jan 22 2008, 08:20 PM)

I have a VI use OpenG Read INI Cluster subvi to read a complex cluster data structure called step config data...

Just by looking at the right scroll bar on your front panel, I'm assuming that's one huge cluster - can you please post your code (the actual VI, not just the screen shot) for us to have a look at?

[yiweihua2002]

QUOTE(crelf @ Jan 23 2008, 09:57 AM)

Just by looking at the right scroll bar on your front panel, I'm assuming that's one huge cluster - can you please post your code (the actual VI, not just the screen shot) for us to have a look at?

QUOTE(crelf @ Jan 23 2008, 09:57 AM)

Just by looking at the right scroll bar on your front panel, I'm assuming that's one huge cluster - can you please post your code (the actual VI, not just the screen shot) for us to have a look at?

[Justin Goeres]

I've noticed, too, that for large files the OpenG Variant Config File VIs are slow. I have a system for setting up GUI interlocks (enable/disable on various trigger conditions) that uses an INI file to store the interlock configurations. The number of lines in the file is slightly larger than (number of GUI slave controls) x (number of interlock signals). In one particular case, this is about 350x20 = 7000 entries in the INI File, and that takes a long time to load (2-4 minutes, as I recall). The file itself was something over 100KB if memory serves.

I was able to mitigate the delay by stripping the "default" values out of the INI file. For instance, for every boolean array element that was FALSE, I removed that line from the file (using the File I/O palette and some regular expressions). In my case (because my data was mostly booleans, and mostly FALSE), this reduced the file size by something like 75%, and reduced the load time by about half. Something like that might work for you if some of your cluster data is just a default value.

However, I've never gotten LabVIEW to crash while reading an INI file. That's a new one to me.

EDIT: Just to be clear, this isn't a dig at the OpenG Variant Config File VIs. I love them . There's just a point of diminishing returns when you try to use them to store large amounts of data.

[LAVA 1.0 Content]

I noticed in your orignal screen shot that you are using something called "encrypted open config data.vi". I recall this VI name from an example in "A Software Engineering Approach to LabVIEW".

Try removing the encrypted read and write functions and use native LabVIEW open/read/write/close functions. If the speed is still to slow, consider using a binary file format rather than using an encrypted INI.

[Neville D]

QUOTE(yiweihua2002 @ Jan 22 2008, 05:20 PM)

Dear ALL: I have a VI use OpenG Read INI Cluster subvi to read a complex cluster data structure called step config data,and when I Run this VI,It sometimes read INI File quickly,and sometimes read INI File very slowly.If I want to speed up the read INI File speed,I must quit labview and restart it,and when I quit labview it will reports error just look like Labview Error Message.JPG

I have heard that the reason the ini VI's are slow is because the native LV VI's themselves are a bit ancient and clumsy. You can try these http://www.mooregoodideas.com/goodLabViewStuff.htm' target="_blank">VI's from David Moore that are supposed to be much faster since they replace the native VI's.

I haven't used them before.

Neville.

[yiweihua2002]

QUOTE(Justin Goeres @ Jan 24 2008, 12:19 AM)

I've noticed, too, that for large files the OpenG Variant Config File VIs are slow under certain circumstances. I have a system for setting up GUI interlocks (enable/disable on various trigger conditions) that uses an INI file to store the interlock configurations. The number of lines in the file is slightly larger than (number of GUI slave controls) x (number of interlock signals). In one particular case, this is about 350x20 = 7000 entries in the INI File, and that takes a long time to load (2-4 minutes, as I recall). The file itself was something over 100KB if memory serves.

I was able to mitigate the delay by stripping the "default" values out of the INI file. For instance, for every boolean array element that was FALSE, I removed that line from the file (using the File I/O palette and some regular expressions). In my case (because my data was mostly booleans, and mostly FALSE), this reduced the file size by something like 75%, and reduced the load time by about half. Something like that might work for you if some of your cluster data is just a default value.

However, I've never gotten LabVIEW to crash while reading an INI file. That's a new one to me.

EDIT: Just to be clear, this isn't a dig at the OpenG Variant Config File VIs. I love them . There's just a point of diminishing returns when you try to use them to store large amounts of data.

Thank you Justin Goeres:

I have post my question in ni.com,and in which I got the answer to let me set File»VI Properties

and selecting Execution in the VI Properties dialog box,set the preferred execution system to user interface.

[Tim_S]

QUOTE(Neville D @ Jan 23 2008, 02:01 PM)

I have heard that the reason the ini VI's are slow is because the native LV VI's themselves are a bit ancient and clumsy. You can try these http://www.mooregoodideas.com/goodLabViewStuff.htm' target="_blank">VI's from David Moore that are supposed to be much faster since they replace the native VI's.

I haven't used them before.

Neville.

The LabVIEW VIs for INI files are definitely slow. It's not bad on small files, but very noticable with large files. My initial workaround for reading in the INI file was to read in the entire file as a string and then process the lines. I don't remember the time difference at the moment, but it seems like it was an order of magnitude change.

I've not used the David Moore VIs. Can anyone relate how they compare?

Tim

[Neville D]

QUOTE(Tim_S @ Jan 25 2008, 03:57 AM)

I've not used the David Moore VIs. Can anyone relate how they compare?

Tim

See this http://forums.lavag.org/New-Freeware-from-Moore-Good-Ideas-t8756.html' target="_blank">link its supposed to be 500x faster.

I would have loved to try it out, but I have moved away from ini's and only store config data in an xml format.

Neville.

[Yair]

From the image posted to the NI thread, it looks like the reason for the speed is that there is an array of clusters and the code for analyzing the cluster is called repeatedly in a loop.

The MGI VIs will probably not help the original poster, since they are 8.x only.

[Mads]

It's not just the native configuration file VIs that cause this, the openG code seems to be slow on its own.

I especially noticed this now that I'm using them on a RIO controller...just reading in a section containing a cluster of 3 small 1D arrays (10 elements in each) takes 2,5 seconds with the OpenG read section VI. I typically have to read more than 10 of these sections so that takes a whopping 25 seconds or more. Because of compatibility issues with existing software it's not an option go use a different file format so I'm stuck with the configuration files.

I've just skimmed the code and I saw that the decoding of the arrays use a recursive call, is that what makes it so slow? I'll have a closer look at it myself when I've got the time, but has anyone looked at this already and found any optimizations?

[Yuri33]

Have you looked at the Profiler? What subVI is taking the most time, and how many calls are made to that subVI?

[Yair]

Since you're running on a cRIO, the issue might be the overhead of dealing with variants, but I saw that using the OpenG VIs (on a PC) without the NI VIs caused huge performance improvements. If you want, you can modify that or the MGI VIs to use the OpenG format, but my quick look into it showed that it might require a couple of days of work to get right with all the differences.

[Mads]

The MGI VIs are in fact just as slow as the OpenG Config file VIs on my PAC units. The OpenG VIs have their main weakness in the use of a recursive call. The time consuming part in the MGI Read VI is the "Get Type Info.vi"...which has a locked diagram so I am unable to investigate it any further...

QUOTE (Yen @ May 27 2008, 07:05 PM)

Since you're running on a cRIO, the issue might be the overhead of dealing with variants, but I http://forums.lavag.org/-t10449.html&view=findpost&p=44633' target="_blank">saw that using the OpenG VIs (on a PC) without the NI VIs caused huge performance improvements. If you want, you can modify that or the MGI VIs to use the OpenG format, but my quick look into it showed that it might require a couple of days of work to get right with all the differences.

[Yair]

I did actually make the change I described here and I posted it to the other thread, but as mentioned there, it's not production level code. I didn't run it on a cFP, but on the PC it was faster. Either code has to do the recursion of the data structure, but the MGI code has the advantage that it doesn't have to manage the values, only to build them into a string.

[jdunham]

QUOTE (Mads @ May 27 2008, 03:20 AM)

I've just skimmed the code and I saw that the decoding of the arrays use a recursive call, is that what makes it so slow? I'll have a closer look at it myself when I've got the time, but has anyone looked at this already and found any optimizations?

What usually takes the time is the recursive parser. For the same data type, parsing the call on each read and write is a big waste of time. I have some ideas about caching the decomposed type structure, but that wouldn't speed up the first call. It would be interesting to combine this approach with faster INI-file i/o routines.

[JodyK]

I just found an application that strongly benefits from the MGI INI tools, but I'm nervous about the fact that the block diagrams are locked on two of the VIs. With a BSD license, doesn't the code have to be open source?

Also, any word on inclusion of the MGI INI tools into OpenG? OpenG is approved for use in a lot of my projects, whereas using code from another source is a little more questionable and would require approval.

Maybe I'll just have to reverse engineer the locked VIs... or better, improve the performance of the OpenG INI tool.

-Jody

[jpdrolet]

QUOTE (JodyK @ May 19 2009, 07:15 PM)

...or better, improve the performance of the OpenG INI tool.

-Jody

OpenG INI tools are wrapped around vi.lib's Config File VIs. The latter are the real problem for performance on large INI files. Optimizing OpenG INI VIs (like replacing recursion with a stack) would only marginally improve the performance on large INI files. Just try to replace Config Data Registry.vi (the core of Config File VIs) with a do nothing VI and you'll see that the remaining code (OpenG) takes only a smal fraction of the execution time.

I have not used MGI VIs but did take a look at the code. When operating on a file, it only parses the specified section of entries. There is a feature to pay attention for: when writing in a section, the whole section is erased and rewritten so previous existing keys that are not specifically rewritten in the current operation are lost. That allows great speed improvements and the only inconvenience is that you have to structure your data so that you don't write different pieces of data in the same section in multiple write operations.

[asbo]

With BSD licenses, only the copyright notice must stay intact - the code can be distributed in either source or compiled form at the will of the developer.

[JodyK]

QUOTE (jpdrolet @ May 20 2009, 08:41 AM)

OpenG INI tools are wrapped around vi.lib's Config File VIs. The latter are the real problem for performance on large INI files. Optimizing OpenG INI VIs (like replacing recursion with a stack) would only marginally improve the performance on large INI files. Just try to replace Config Data Registry.vi (the core of Config File VIs) with a do nothing VI and you'll see that the remaining code (OpenG) takes only a smal fraction of the execution time.

Yeah, I came to the same conclusion regarding the Config File VIs. Luckily, they are not locked so they could be replaced with an OpenG version. I'm not saying it would be easy, but it would definitely be doable.

Interesting note about the BSD license. I'm so used to the sourceforge and linux worlds where anything under BSD is always open source (or at least I can't think of any exceptions right now). How does the BSD license handle forks and derived works? In other words, if the locked MGI VIs were replaced how would that affect the project? Could it then be integrated into the OpenG toolkits or would MGI still have creative ownership over it?

-Jody

[jpdrolet]

QUOTE (JodyK @ May 20 2009, 11:37 AM)

Yeah, I came to the same conclusion regarding the Config File VIs. Luckily, they are not locked so they could be replaced with an OpenG version. I'm not saying it would be easy, but it would definitely be doable.

Interesting note about the BSD license. I'm so used to the sourceforge and linux worlds where anything under BSD is always open source (or at least I can't think of any exceptions right now). How does the BSD license handle forks and derived works? In other words, if the locked MGI VIs were replaced how would that affect the project? Could it then be integrated into the OpenG toolkits or would MGI still have creative ownership over it?

-Jody

Indeed BSD is an Open Source licence. Redistribution of derived work doesn't require to include the source but the licensed code should definitely be available.

[Yair]

I didn't check, but I'm guessing the locked VIs you're refering to are actually from NI, not MGI. You can easily check by seeing if they're in the vi.lib folder.

As such, they should be in any version of LabVIEW for the forseeable future and you don't need to redistribute them yourself.

I would advise against spending time on trying to refactor NI's config VIs. For one thing, it's not impossible that they will be changed anyway in an upcoming version of LabVIEW. For another, even if they're reworked, they're going to be relatively slow because they have to do lookups, adds and replaces.

I've found that if you need the speed the tradeoff of the section being overwritten is well worth it.

[JodyK]

QUOTE (Yair @ May 20 2009, 01:12 PM)

I didn't check, but I'm guessing the locked VIs you're refering to are actually from NI, not MGI. You can easily check by seeing if they're in the vi.lib folder.

As such, they should be in any version of LabVIEW for the forseeable future and you don't need to redistribute them yourself.

I would advise against spending time on trying to refactor NI's config VIs. For one thing, it's not impossible that they will be changed anyway in an upcoming version of LabVIEW. For another, even if they're reworked, they're going to be relatively slow because they have to do lookups, adds and replaces.

I've found that if you need the speed the tradeoff of the section being overwritten is well worth it.

The VIs I am referring to are "Get Type Info.vi" and "Get Cluster Info.vi" and both are definitely part of the MGI set. Looking at the VI hierarchy it actually appears that these VIs may just be some kind of wrappers around libraries anyways.

The real issue has nothing to do with them being in future versions or any need to redistribute them. I mean, under BSD, they are out there and therefore can't be removed from public domain. The problem is my clients don't want closed source code from from outside the development environment. Can't say I blame them. If I sent you a piece of code I wrote with a locked block diagram would you want to stick it in your projects without being able to ever change it or know what it's actually doing?

Besides, I'm not sure I would trust vi.lib code to be available in its current incarnation for the forseeable future... NI has challenged that assumption before (think reporting toolkit, etc)

I agree that refactoring the NI VIs is probably not a good use of time. Though in essence this is kind of what the MGI guys did - reproduce an existing functionality with better performance.

-Jody

[Yair]

QUOTE (JodyK @ May 20 2009, 10:00 PM)

The VIs I am referring to are "Get Type Info.vi" and "Get Cluster Info.vi" and both are definitely part of the MGI set.

I don't seem to have them in my clean LabVIEW, so I guess you're right. If you want a leg up, you can have a look at the conversion I posted http://forums.lavag.org/-t10449.html&view=findpost&p=44633#entry44633' target="_blank">here.

[JodyK]

QUOTE (Yair @ May 21 2009, 01:08 PM)

I don't seem to have them in my clean LabVIEW, so I guess you're right. If you want a leg up, you can have a look at the conversion I posted http://forums.lavag.org/-t10449.html&view=findpost&p=44633#entry44633' target="_blank">here.

I've actually been emailing with the MGI guys and it looks as though the "Get Cluster Info.vi" is actually an NI VI (my mistake). However the "Get Type Info.vi" is definitely locked. Luckily, it's really just a re-engineered version of "vi.lib/Utility/VariantDataType/GetTypeInfo.vi".

If you replace the MGI version with the NI version, everything works fine, but it does run slower (300 msec vs 100 msec in my app, which beats OpenG's 3300 msec). There are OpenG VIs which basically handle the same functionality, but the enum for types doesn't line up with the NI one. The MGI guys also said that the OpenG VIs didn't work quite right with waveforms and maybe one other data type. It looks like you ran into a lot of these issues in your other thread and went through the same process of replacing with OpenG VIs to get everything to work in LV 7.

I'm also talking to the MGI guys about modifying this to use the OpenG VIs and then integrating it with the rest of the OpenG framework. Not sure they'll go for it, but I do think it would be useful.

Who knows, maybe LV 2k9 will fix all the INI performance issues and make this a moot point...

-Jody