Saving and loading multiple versions of a cluster

[Yair]

I have a project which uses quite a large cluster with some large subclusters in arrays. These need to be saved occasionally and loaded at startup. Not very nice, but it's not my design and I wasn't really allowed to change it.

Initially, I used the OpenG variant config VIs, which worked great, as always. At some point, however, they started taking 100% CPU and running slowly. I think it was after passing a certain critical mass in complexity, but I'm not sure. Upgrading to the latest version seemed to solve the problem, but it came back after a couple of days.

Before I do any drastic changes, I was wondering what ideas people had for alternatives.

Here are my thoughts:

• Can't use LVOOP because it's 7.0.
  
• Flattening the cluster to a string will work, but the problem is that I couldn't think of a way of upgrading from an older version without doing an explicit unbundle and bundle for every element from the old version to the new one, because the clusters have different types. Since the cluster is big, that's kind of nasty.
  
• Getting into the OpenG and NI VIs to optimize them. A lot of complex work. Not something I want to do currently.
  
• Changing the cluster content to be smaller. For example, placing controls of the same type into an array and indexing the values out of it. Then, I can keep using the OpenG VIs. This will require some work and make the code not as nice, but it currently looks like the best solution.
  

Any other ideas?

[Tomi Maila]

What about saving it as an XML? JKI could have a perfect solution for this.

[PaulG.]

We have some apps running in the field with HUGE clusters that use the OpenG variant config functions and to my knowledge they have been running fine for quite some time. Granted, we are running 8.5 with 8.5 OpenG. That might have something to do with it, but it's all I can think of at the moment.

[Yair]

QUOTE (Tomi Maila @ Mar 27 2008, 10:54 PM)

What about saving it as an XML? JKI could have a perfect solution for this.

EasyXML is only supported for 7.1 and higher. I suppose I could try it just to check, but I seem to remember from my testing that a lot of the problem lied in the variant VIs, which are also shared by that toolkit, if memory serves.

QUOTE (PaulG. @ Mar 27 2008, 10:55 PM)

We have some apps running in the field with HUGE clusters that use the OpenG variant config functions and to my knowledge they have been running fine for quite some time.

Mine aren't huge. Combined, they probably have around 100-150 elements, some of which are in arrays, so the VIs have to iterate over them. LabVIEW 8.x does have better performance for variants, so I suppose this could be an issue either with the variant performance of LabVIEW or a specific thing which the VIs don't like in my clusters.

[Yair]

Another idea that I thought about now is modifying the OpenG VI so that I do the header analysis part of the variant parsing before building the EXE and save it as a constant, but that has its own disadvantages and I'm not sure how much improvement it will get me.

[Michael Aivaliotis]

QUOTE (Yen @ Mar 28 2008, 03:06 AM)

Another idea that I thought about now is modifying the OpenG VI so that I do the header analysis part of the variant parsing before building the EXE and save it as a constant, but that has its own disadvantages and I'm not sure how much improvement it will get me.

I'm not convinced that you have a problem. You say that it used to work and now it doesn't. Why? You say you upgraded to the latest variant tools and it worked. Then after a while it didn't.

Could you post the data structure so we can try it out and benchmark it?

[Yair]

I think this has to do with how much data is actually in the structure. I will try to get a situation where I can reliably reproduce this and then investigate by removing stuff. If I still don't find anything, I'll post something then.

[Yair]

OK, turns out most of the problem is in NI's VIs, which I knew were quite inefficient, but I didn't realize how much. Making this change so that NI's VIs didn't actually do anything changed the result considerably:

At this point, since I recently installed 8.5.1 and saw that the MGI VIs had a huge performance margin, I decided to look into them to see why they couldn't be backsaved (since that's what I remembered about them). I knew they used some of the variant VIs which were added in 8.x, but I thought I might be able to work around that. So I started by replacing all the 8.x variant VIs with the OpenG variant VIs. It took some adapting, but I eventually got a version which produced the same result with no 8.x code. I then backsaved it to 7.0 and hacked around the differences until the code again produced the same file.

Currently, I have a set of VIs which works in LV 7.0 and which produces what is essentially an MGI INI file at about 5% of the time the variant config VIs take. You can open and run it with the sample cluster to see the performance differences. As you can see, the cluster has a total of ~180 elements and since about half of them are in an array, in this case this translated to something around 1500 elements.

Some important points about this:

1. It works, but it's a hack. The data types in the 8.x and OpenG list don't match and I only moved stuff around enough for it to work. I also had to change some stuff because flattening to a string has more options in 8.x.
   
2. The MGI code uses variant attributes (all strings) to hold the lines. When working on this, I changed this to a simple LV2 global because I feared the variant might have performance issues. That means that using this with more than a single INI file can cause bugs. This needs either to be reverted to a variant or to use some other mechanism to hold a unique list for each file.
   
3. It requires some OpenG VIs to run.
   

I think that this code can be used as a basis for refactoring the variant config VIs. Currently, it uses the MGI format, but I assume changing it to work with the OpenG format should be relatively straightforward. I don't think that there is any legal issue, since the MGI VIs are freeware, but asking MGI for permission is probably not a bad idea. Unfortunately, I don't know if I can currently spare the time to work on this properly. I'll probably modify it enough to get it working reliably, but I won't have time to do the whole thing.