Slow Index Array of Classes

[GregFreeman]

I have an array of classes, let's call the object TestPass, of size 1 (but it is an array because it can scale out to multiple test passes). In this class, there is one other nested class which is not too complex, then various numeric and string fields to hold some private data. There is also an array of clusters. In this cluster there is a string, two XY pair clusters, and an integer. Not very confusing.

This array of clusters gets fairly large, however, upwards of 80-100k elements. What I am finding is when I index the array of pass classes it is crazy slow. On the order of 30 ms. Doesn't seem like much, but we are indexing the array in our method to "Get Current Pass" which is used in various places throughout our code. This is adding potentially hours to our test time over the 80k devices we are testing. 

So, I started digging. When I flatten the class to a string and get the length, it's 3 mb. But, when I run the function with the profiler is is allocating close to 20 mb of memory!

My gut feel was that the string is causing the issues. So I removed the string from the cluster and the index time went to 0 ms. 

Luckily we can normalize a bit and pull the strings out of the cluster since a lot of them are duplicates. But it makes our data model a bit uglier. 

Has anyone seen these kind of performance issues before? I saw them in 2013 and 2017.

Edited October 16, 2017 by GregFreeman

[GregFreeman]

I normalized my data but was still seeing awful use of memory. Upwards of 3 gb, and I would randomly get a copy that would give me an out of memory error.

So, i went into my project settings, unmarked everything that had compiled code separated. Cleared compiled object cache. Did a save all. My memory usage has dropped from 3gb with tons of seemingly unnecessary copies to 1 gb just by doing this. I have on and off seen some very bizarre issues with classes and separate source from compiled, and even with that setting I still get lots of dirty dots anyways which isn't buying me much. I think I'll be staying away from it in the future.

Edited October 16, 2017 by GregFreeman

[crossrulz]

You could also try clearing the class mutation history and see what that does.

[Neil Pate]

I definitely second the idea of nuking the mutation history (the most unwanted feature in LabVIEW I have ever come across). 

[hooovahh]

7 hours ago, Neil Pate said:

the most unwanted feature in LabVIEW I have ever come across

Sorry I couldn't hear you over the sound of my ZOOOOOOM!

Okay maybe zoom wasn't my number 1 request, but it likely was the number one request by over all.  Maybe XNodes would be my most requested features.  VIMs gets 80% there.

[ShaunR]

7 hours ago, Neil Pate said:

I definitely second the idea of nuking the mutation history (the most unwanted feature in LabVIEW I have ever come across). 

It's not a feature. It is an internal class mechanic that you were never supposed to realise was there.

[Stagg54]

When you are trying to hide something, making it smaller is generally preferable...  Making it gigantic almost ensures it will be found...

[GregFreeman]

Alright, I rolled back to a "bad version", I grabbed this snippet off the idea exchange and I'm going to run it on all my classes. I'll see what happens...

[GregFreeman]

Well...didn't fix it per se. But, we did a build WITHOUT normalizing the string array (i.e. no code changes) and it's using drastically less memory in the EXE than the dev environment....We're talking like 600 mb of memory usage instead of 2.5 gb.

My guess now is having debugging on in some of these VIs is causing issues in the dev environment. Probably copies everywhere. Either way, normalizing things was a much more memory efficient way of doing this and is a needed improvement. Rather than have 24 classes each with 80k strings, many of which are duplicates, we have 24 classes each with about 2k strings, and we have 80k integers that point to an index from that string array.

As much as I'd love to dig into the LabVIEW memory manager to truly understand what's happening in the dev environment (not), I am just going to put this in the "no longer a problem" column and move on.