Detecting network drop-out from cRIO

[PaulL]

We have a requirement that any controller running on a CompactRIO should go to a fault state, stopping all motion in the process, if there is a loss of network communication (which could happen if someone disconnected the network cable, the network went down, etc.).

We meet this requirement at present by monitoring the error code from a Read Variable with Timeout function we use. In the event of a network loss there this function reports an error (-1950679035) and we safe the system and go to a fault state as required.

The Read Variable with Timeout function does not report error -1950679035 immediately after someone disconnects the network cable, however, in tests. It takes some finite (at least 8 s) and not configurable time. The customer has asked us to detect the network loss in a shorter (preferably configurable) time. Does anyone know how to do this?

Yesterday I found that I was able to use a ping approach successfully between two Windows machines (although I think the final implementation details would be a little more involved than what I did in my test). The specific approach doesn't work on VxWorks, I think, although I have seen a reference to a pinglib, but I don't know off-hand how I would call such a function. I have seen a couple threads (and a related idea on ni.com/ideas) but I don't have an answer yet. Maybe there is a completely different approach that will work better.

I thought I'd post here and see what suggestions I might get.

Thanks!

[Tim_S]

We've used TCP connection instead of variables; this allowed us to configure a timeout and similarly check the error code.

[ned]

It might be too extreme for your case, but I believe the cRIO has a "Halt system if TCP/IP Fails" option that you can set in MAX.

[PaulL]

We've used TCP connection instead of variables; this allowed us to configure a timeout and similarly check the error code.

So you already have a port to check (the one you are already using for communication)? Is that correct?

[ShaunR]

Like Tim_S. I assume this is an NSV issue since I've never seen 8 second delays with TCPIP. I havn't timed it, but it usually breaks the connection as soon as the LED goes off on the port although WiFI can be problematic for this sort of thing.

You could open another connection with straight TCPIP just to monitor to see if it disapears since a lost wire will affect all connections on that wire/card. If you go that route, then it's sometimes a good idea to have it in a separate exe running in the background so that it also crowbars your main application (taskkill /f kinda thing).

[asbo]

It might be too extreme for your case, but I believe the cRIO has a "Halt system if TCP/IP Fails" option that you can set in MAX.

My impression was that setting only did anything if DHCP failed, not necessary any TCP/IP connection.

[PaulL]

Well, what I said about the timeout not being configurable may not be correct. I found this http://zone.ni.com/reference/en-XX/help/370051K-01/cvi/cvinetvartips/. It is for LabWindows but I did find the relevant entry ([LOGOSXT]

Heartbeat_AbsenceDetectCount) in ri-rt.ini on the CompactRIO. (We remembered seeing something like this, but then we couldn't find it, and here it is again. Maybe we installed the right software or we got it when we moved to 2011?) I am going to run a test....

[PaulL]

Later...

Well, it's definitely not working as I had hoped. I still don't get the error for about 20s after I pull the plug, even if I use 1 for this value (and for a value I added, Heartbeat_SendTimeoutSecs). I contacted NI Support to see if they can clarify what these parameters actually do....

Paul

[Tim_S]

So you already have a port to check (the one you are already using for communication)? Is that correct?

Yes, the communication port also checks the connection. The loop is always trying to read from the connection, so the other end dropping the connection (by hook or crook) trips an error immediately from the TCP Read.

Tim

[PaulL]

Yes, the communication port also checks the connection. The loop is always trying to read from the connection, so the other end dropping the connection (by hook or crook) trips an error immediately from the TCP Read.

So I think you are doing essentially what we are doing with the Read Variable with Timeout function. It looks like that might not be sufficient, so we may need to use a TCP function just for this check, which we can do but it is more involved for us. (If we could just configure the SV timeout we wouldn't have to write any additional code.) I'm wondering: Is there a port or service that we can anticipate will be open and can monitor with TCP/IP so that we don't have to open an additional one just for this purpose?

[Tim_S]

I'm wondering: Is there a port or service that we can anticipate will be open and can monitor with TCP/IP so that we don't have to open an additional one just for this purpose?

I'm using a port that I've acquired and aliased specifically for communication, so I don't have the issue of trying to find a port available. There is a list of standard TCP ports located at http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers. I don't expect a cRIO has all of these. Echo (port 7) would be my first pick, should the cRIO have it, as connection health (versus connection presence) could be checked by sending a message.

Tim

[PaulL]

It might be too extreme for your case, but I believe the cRIO has a "Halt system if TCP/IP Fails" option that you can set in MAX.

Well, we would like to detect the problem and then go to the fault state gracefully within our application. As I said, we do that now, the detection time is just too long.

I'm using a port that I've acquired and aliased specifically for communication, so I don't have the issue of trying to find a port available. There is a list of standard TCP ports located at http://en.wikipedia....DP_port_numbers. I don't expect a cRIO has all of these. Echo (port 7) would be my first pick, should the cRIO have it, as connection health (versus connection presence) could be checked by sending a message.

Tim

Hmm... Thanks for the link. So I can detect a problem by trying to connect to the cRIO (the near end of the wire) or do I want to listen to a port on the remote machine (Windows in this case)? It looks like this is something to explore....

[PaulL]

Later... well, no luck connecting to port 7 (or anything else yet). I'd really like to avoid opening a new port on the remote machine just to do this....

[Tim_S]

Later... well, no luck connecting to port 7 (or anything else yet). I'd really like to avoid opening a new port on the remote machine just to do this....

Chances are that the cRIO doesn't have an echo service running, thus a connection to port 7 would fail. NI should have a list of the services it does have running that you wouldn't have to put any code on the cRIO to accommodate. You might be stuck writing a listener, etc., on the cRIO.

Tim

[WMassey]

I recently looked this up and saw at the time that it was built-in functionality. This link says "All CompactRIO, FieldPoint and Compact FieldPoint controllers have built-in network monitors. If you enable the network watchdog and the controller loses communication with all hosts or clients over the network, controller sets the output channels to predefined values corresponding to the watchdog state."

There are also these if you you want to do it yourself:

Uni-Directional Network Watchdog Reference Example

Fail-Safe Control Reference Design for CompactRIO

[PaulL]

Thanks for the links!

The solution we found that worked best for our purposes was the following:

Our "data listener" (kind of a message handler) on the desktop periodically sends a heartbeat command to the RT controller using the same mechanism as other commands. (There is only one shared variable since we are using the Command Pattern. In other words, the data listener just periodically writes a heartbeat object.)

We wire the timeout on the "Read Variable with Timeout" function (in the controller, now) to be whatever we want the heartbeat timeout to be. If there is a breakdown in communication the timeout will occur, and the controller responds appropriately.

This solution has turned out to work quite well and allowed us to achieve what we wanted just by adding a small piece to the existing communication scheme.