Invalid Dynamic Shared Variable Performance Issue

[Omar Mussa]

Recently experienced a performance bug that I traced to a bug in my code related to shared variables that I just want to share to hopefully help others avoid this error.  (This was running in LV2014 SP1 on Windows)

 

I am using a dynamic shared variable "Read Variable Function.vi" but not opening the connection prior to reading the data.  I accidentally had passed in an invalid PSP variable path to the "shared variable refnum in" input.  This VI happened to be in a loop that was updating fairly frequently (every 50 ms).  Well ... it turns out that because I wasn't opening the connection first and passing in an actual refnum, every time it hit this node, it retried opening a connection to a variable that didn't exist which eventually timed out - this was a very shared variable/(scan engine?) intense operation and effectively brought all of my other shared variable operations to a halt - performance went from awesome to a total drag and my application was almost unusable.

 

Short term fix - I changed my configuration to point to an existing and valid PSP variable and all of my performance went back to normal.  Hope this helps someone else avoid my "mistake" although I'd also call this a LV bug in a sense that it shouldn't drag an application's performance down in such a severe way if a variable goes from existing to non-existent.

Edited August 11, 2015 by Omar Mussa

[viSci]

In this case would the read variable with timeout help? Or should the warning be to always try opening the connection first?

[Omar Mussa]

Great point - I'm not sure whether 'read var with timeout' would help or not - in principle it should be better but I'm not sure if the timeout is respected if the variable does not exist (as in you pass in an invalid PSP variable that is not known to the scan engine).  I'll test it once I have time to do that.

[smithd]

Help says "Caution  If you use this function to access a shared variable without first opening a connection to the variable, LabVIEW automatically opens a connection to the variable. However, this implicit open operation can add jitter to the application. Therefore, National Instruments recommends that you open all variable connections with the Open Variable Connection function before accessing the variables."

 

Generally I'd be sure to open all connections first and never use a read directly on an unconnected string. To me what you described is expected and desired behavior -- if the implicit connect failed it should retry, and your code should have sufficient error handling to deal with that situation.

 

Something else that may help is to use double libraries. That is your cRIO has a library with all the variables it needs hosted locally. The PC has a library with all its variables locally. Then when you're ready to test them together you bind the appropriate variables. Then no variable is ever missing.

[Omar Mussa]

Generally I'd be sure to open all connections first and never use a read directly on an unconnected string. To me what you described is expected and desired behavior -- if the implicit connect failed it should retry, and your code should have sufficient error handling to deal with that situation.

 

 

Fair enough - in our case all variables 'should' always exist and this issue was the result of some bad configuration data that got merged from a dev branch.  I think it makes sense that the variable should try to open a connection, I was *surprised* that this meant every other parallel shared variable operation was blocked waiting for the open to timeout.

[smithd]

Ah yeah I see, the blocking behavior is annoying. Thats always been a bit of a downer for the SVE, the fact that its mostly single-threaded.

 

Wouldn't help in this case, but there is a method for opening the variable connection in the background. This essentially sends a message to the SVE requesting that the connection get opened, and then you can use a property node on the last refnum in your set of connections to determine if the connections have finished opening. This is significantly faster than waiting for each connection to open individually.

[Omar Mussa]

I finally had some time to get back to this issue.  In my view it seems that the PSP/Shared Var Engine seems to work best when you are sure that the target and variable will exist for the lifetime of your application.  Reconnecting is definitely 'expensive' as is having non-existent variables (which force engine to try to reconnect).

 

I tried a few of these methods out today to see what would happen and I'm not sure what exactly to make of it but I figured I'd share for those interested.  All of the tests I tried were running a Windows 7 machine connected to a cRIO 9067 that was on my desk - so network traffic/infrastructure were not a factor.

 

First, there is a major difference in performance (execution time) between an invalid variable on a valid PSP engine (target exists, variable does not) vs a variable on an invalid PSP engine (target does not exist so all variables also do not exist) - meaning that if the target does not exist at all then ALL of the connect methods will take significantly longer than usual (for example, "Open Variable Connection in Background" still takes 5000 ms if the PSP var engine is not found, compared with <50 ms if the target is found but the variable does not exist).

 

Here are the cases I tested:

"Read Variable.vi" - the default way I had tested initially.  If target does not exist, times out after ~5000 ms with error -1950679035 - subsequent calls timeout at ~0 ms for a few seconds and then var attempts to reconnect automatically and takes ~5000 ms per call - does this about 5x per variable.  If the target exists but variable is not deployed then the read call errors immediately, unfortunately with same error code/message (-1950679035).

 

Here's the graph of timing for this case:

 

 

"Read Variable with Timeout.vi" - Has exactly the same behavior as "Read Variable.vi" in terms of this test (same errors, same timing).

 

"Open Connection in Background" - The first time I tried this I thought - great I'll just call this once and then wait for the connection to get to a connected state.  Seems like the best option but unfortunately, if the initial call throws the error -1950679035 then the var connect does not continue in background so you will never get a connected status.  In this case the error description is more helpful - the error description is "LabVIEW:  (Hex 0x8BBB0005) Unable to locate the shared variable in the Shared Variable Engine (SVE). Deployment of this shared variable may have failed, the SVE has not started, or the SVE is too busy to respond to this request.".  So if you get this error, you need to retry the open connection in background, and you end up with the exact same timing profile as the "Read Variable.vi".

 

So basically it seems to me that my PSP application needs to either - always have the variables available - especially at connect time OR be able to cull out bad targets and not try to connect while the target is down OR be performance insensitive so that it doesn't really matter if the target you're connected goes down or not.

 

I've attached the VI I used for timing tests.  Basically I would swap between invalid targets and invalid variable names to do the tests.

 

Test PSP Read Time.vi

[Jordan Kuehn]

@Omar Mussa Thank you for this detailed analysis. I've found that my testing in 2021 on LV 2020 almost exactly replicates your results.

 

I found some code on the darkside from @smithd that doesn't fix this issue, but I then repurposed into a check before the read that prevents the system from repeatedly attempting to reconnect and then time out. It's not ideal in my use case, but is better than blocking timeouts from a few dozen individual reads in my application that bring it to its knees.

 

 

I've found that this will reconnect if the system and SVE is present during the initialization when I use the "Open Connection in Background" (which is as you observed a red herring). And if the system and SVE is not present I do not incur these 5s timeouts and just get the error from the type cast that I can use to output NaN or whatnot.

 

Perhaps others have found other workarounds. This is quite frustrating to me as the PSP variables are the recommended approach for using ethernet cRIO expansion chassis and I cannot guarantee their presence on the network 100% of the time and cannot have its absence subsequently kill the application.

[viSci]

Been there done that with NSV's.  For cRIO connectivity I now use the RTI DDS toolkit or the messenger library which are both free and excellent.

[Jordan Kuehn]

10 minutes ago, viSci said:

Been there done that with NSV's.  For cRIO connectivity I now use the RTI DDS toolkit or the messenger library which are both free and excellent.

I have spent a lot of time with the RTI DDS toolkit and it is good. I’m using MQTT for a lot of similar use cases. I believe we chatted some time ago about this a little. 
 

However, I don’t think this or the messenger library are feasible with the expansion chassis (9147 in my case) unless I were to implement the communication protocol in their FPGA. I would be happy to be wrong here though.