Stale VI Server Connections

[John Lokanis]

I am running into an issue where my VI Server connection goes stale after a few hours.  Looking for a fast way to detect this and recover.

Currently here is what I am doing:

• On first call, open the application reference and then open the VI reference.  Cache both of these.  Use the VI reference to call the remote VI.
• On subsequent calls, test the cached references to verify they are still valid and then call the remote VI.

What appears to be happening is the references still appear to be valid in the caller but the connection is broken so the remote call fails.  Then I detect this and reopen the app and VI refs and can again call the remote VI.

The issues are:

• The failing remote call takes a long time to timeout and I do not see how to control this timeout.
• The test of the refs does not actually test to see of the network connection is still good.

The result is it takes a long time to recover and this is upsetting the user since it appears the system is locked up.

What I need is a way to control the timeout of the 'call by reference' node or a way to quickly test the references to verify the network connection is still good before I attempt the remote call.

Any ideas?

thanks,

-John

[hooovahh]

Well that sounds terrible, but also something I can't say I've seen.  Do you have a simple VI that exhibits the issue that can be ran to reproduce the issue?  What OS and LabVIEW versions?

[John Lokanis]

I could make a simple VI, but there is not much to it.  Open app reference followed by open VI reference followed by call by reference.  Save the app and vi refs for later.  Then wait awhile and try to call the remote VI again with the cached refs.

I am sure that IT is to blame for my open connection going bad, but I can't really control them and just need a way to recover quickly.

I don't want to incur the cost of opening the app and vi refs for every call.  Looking for an alternate solution.

 

[ShaunR]

You can get these symptoms if there is a proxy between you and the target. What tends to happen is the connection between you and the proxy is kept alive but the connection to the endpoint from the proxy closes due to inactivity. This means your connection looks good but the tunnel fails when you try to use it. The easiset solution is to heartbeat the connection.

Edited August 11, 2016 by ShaunR

[John Lokanis]

No proxy but there may a firewall.  Need to talk to the IT guys.  Keep alives would work but the system is not designed for that and it would add to traffic.  Especially when across large distances with limited pipes.

I wish there was some docs about how long the call by ref timeout is and how to control it.

Another option is to store a time-stamp with the refs and refresh them if enough time has elapsed.  Wish there was an easier solution.

[John Lokanis]

Confirmed that the firewall kills stale connections after 1 hour.  I'll add a timestamp to refresh the connection if it is over 55 mins old.

 

Still want to know why the timeout for call by reference is so long.  It appears to be 60 seconds and there is no way to change it.

[ShaunR]

9 hours ago, John Lokanis said:

Confirmed that the firewall kills stale connections after 1 hour.  I'll add a timestamp to refresh the connection if it is over 55 mins old.

 

Still want to know why the timeout for call by reference is so long.  It appears to be 60 seconds and there is no way to change it.

If IT have said that a keep-alive will work, then that is the proper solution and you should be trouble free from then on. Checking every 55 minutes because they said the idle timeout is one hour is just asking for  it to fall over in 6 months when someone in IT decides 45 minutes would be better

Edited August 12, 2016 by ShaunR

[Rolf Kalbermatter]

16 hours ago, John Lokanis said:

Confirmed that the firewall kills stale connections after 1 hour.  I'll add a timestamp to refresh the connection if it is over 55 mins old.

 

Still want to know why the timeout for call by reference is so long.  It appears to be 60 seconds and there is no way to change it.

Well that's the problem of using layered software. Standard socket operations are normally synchronous and don't have a timeout parameter themselves. In order to program asynchronously, one has to first set the socket into O_NONBLOCK  and then use  select() or  poll()  before attempting to do a read() or write(). And the timeout for most synchronous socket operations is an option that can be changed for each socket through the setsockopt() call.

VI Server might use internally asynchronous sockets but it is designed as a synchonous interface to the user. The reason is obviously ease of use, since asynchonous interfaces are usually pretty hard to use and debug. And unfortunately nobody thought ever about adding a possibility to add a timeout property to the VI server refnum class. But that's understandable since the VI server refnum is not generally a network connection, it just can be routed over a TCP or ActiveX connection, but just as much be a LabVIEW process internal protocol layer. A timeout property would only be meaningful to the TCP protocol layer but wouldn't apply to the other two classes.

It's the classical problem of abstraction where you can't and don't want to expose all the lower level details to the higher level interfaces, especially when they are not universal for all the lower level implementations.

Edited August 12, 2016 by rolfk

[mje]

I've had to tweak some ping/timeout settings for a build manager I wrote. My VI Server refnums were going stale because the foreign application instances would be so busy making an exe they wouldn't reliably service VI Server requests for long periods of time. I'm pretty sure I ended up putting a timeout on the order of an hour or two, so I think it is possible to tweak with this stuff. I can't remember how, if you can't find anything I can try to dust the code off early next week.