Memory impacts of Build array growing in a while loop

[Arka Dipta Das]

Building an array with a while loop is never a good idea but I somehow am guilty of that practice and now it has come back to bite me.

I started using such a method after I read in a forum discussion that is a simple way to track changes due to a looped structure and that the "build array" block itself optimizes the memory so that the Vi does not get bogged down when run for long periods.

My state of my code says otherwise and therefore I want to know for certain how building such an array is handled in LV.

I have attached a sample VI to demonstrate what I mean. 

 

Growing build Array.vi

[hooovahh]

The answer like most things is it depends.  Are the values from one iteration loop, into the next needed for some comparison check?  Like are you going to be checking to see if a value in the array already exists from a previous iteration?  If not I would suggest wiring the output of the loop and change the tunnel to be indexing.  This has the added benefit of having the conditional input as an option which can choose which elements to have be passed out or not.

Another method that might be suggested is to preallocate the array with the initialize array function, then perform replace array element inside the loop.

The reason why building in a loop like this is discouraged, is because beginners of LabVIEW often leave this array building unchecked in size.  For simple tests that don't run long this might be just fine.  Not optimal but will work fine.  The larger problem is when this type of technique is used in a program that will run for months or years non-stop.  Memory will grow forever, and it will crash.

There is a performance hit from having to change the array size every loop which in this example will be pretty small.  If you are okay with this, and if you have a way to limit the array size some how, then I'd be fine with this type of code.  This might be something like check the array size each loop, and if it is greater than some value, get an array subset.  This can be a type of circular buffer.

[Arka Dipta Das]

I use this method to do a couple of things:-

1. to create a stream of historical data points (say values generated in the last 1000 iterations of the loop) and then find the highest value in that array. Data points older than 1000 iterations are not required and I would want them to get overwritten. Does this code (Fig.1) work as an alternative of what you mentioned

Quote

Another method that might be suggested is to preallocate the array with the initialize array function, then perform replace array element inside the loop.

about pre-allocation of the size of the memory that the Build Array block will use?

Since I am using a while loop here, i can not achieve this with a indexing tunnel. If (Fig.1) does not achieve this, will (Fig.2) be a good substitute?

(Fig.1):-                  (Fig.2):-  

2. to monitor the loop execution times in real-time rather than use the Profile>Performance and Memory tool and getting only snapshots. (Fig.3)

(Fig.3):- 

 

3. to check the response of hardware vs. each value generated by the For loops. This I do by having a XY-plot within the loop. (Fig.4). This is used in a For-Loop.

 

(FIg.4):- 

[hooovahh]

Your Fig 1 still has a couple issues.  First the memory issue.  You build the array, then put it into a shift register that continues to build.  You need to perform the array subset, and then put that into the shift register.  Because adding elements at the start of an array were less efficient than at the end (not sure if LabVIEW improved) I often put elements at the end, then perform an Array Size, and if it is greater than some value, I use the Delete From Array, providing the index to delete as 0.  Just a pointer that probably doesn't matter.  The other issue with Figure 1 is you start with 1000 elements, then add elements one at a time.  I suspect you either want to start with an empty array, or use the replace array subset but then things like pointers and shifting might be an issue.  If you are just trying to keep it simple, I'd suggest this.  Also if you are preallocating the array with a bunch of data, doing an Array Min/Max is going to use that preallocated data too which probably isn't what you want.  If you go with a preallocated method you can still use array min max, but you will have to keep track of how full it is with real data, and then do array subset before Min/Max.  This seems easier and honestly isn't that bad.

Also you should be posting VI Snippets, these allow for posting pictures of your code as PNGs, that can then be dragged into the block diagram.  Then others can get your code more easily.

[crossrulz]

50 minutes ago, Arka Dipta Das said:

1. to create a stream of historical data points (say values generated in the last 1000 iterations of the loop) and then find the highest value in that array. Data points older than 1000 iterations are not required and I would want them to get overwritten.

Just use the Array Max & Min PtByPt.vi.  It does all of this for you.  There is also the Data Queue PtByPt.vi that you might find useful.

[Arka Dipta Das]

27 minutes ago, hooovahh said:

Your Fig 1 still has a couple issues.  First the memory issue.  You build the array, then put it into a shift register that continues to build.  You need to perform the array subset, and then put that into the shift register.  Because adding elements at the start of an array were less efficient than at the end (not sure if LabVIEW improved) I often put elements at the end, then perform an Array Size, and if it is greater than some value, I use the Delete From Array, providing the index to delete as 0.  Just a pointer that probably doesn't matter.  The other issue with Figure 1 is you start with 1000 elements, then add elements one at a time.  I suspect you either want to start with an empty array, or use the replace array subset but then things like pointers and shifting might be an issue.  If you are just trying to keep it simple, I'd suggest this.  Also if you are preallocating the array with a bunch of data, doing an Array Min/Max is going to use that preallocated data too which probably isn't what you want.  If you go with a preallocated method you can still use array min max, but you will have to keep track of how full it is with real data, and then do array subset before Min/Max.  This seems easier and honestly isn't that bad.

Also you should be posting VI Snippets, these allow for posting pictures of your code as PNGs, that can then be dragged into the block diagram.  Then others can get your code more easily.

this implementation seems interesting. Thank you for the suggestion.

I realize now the mistake of initializing with a 1000 element array. 

I am not sure if you mean "initializing the array" when you say "preallocating the array". I was talking about memory allocation for the array rather then the initial values of the array.

VI Snippets is something I did not know about. Thank you for bringing it to my notice.

[Arka Dipta Das]

20 minutes ago, Arka Dipta Das said:

VI Snippets is something I did not know about. Thank you for bringing it to my notice.

They work when I save them and drag them back locally but the snippet you shared is not working. I have tried saving your snippet and then dragging it but still did not work... 

Edited July 22, 2020 by Arka Dipta Das

[hooovahh]

Sorry about the snippet thing, LAVA sometimes strips out the meta data for it.  In any case Crossrulz is right there is a function for this already.