Client-Server network message transport options

[John Lokanis]

I have run into a bit of a problem with how I am trying to do networked messaging between a client and a server application.  I am considering trying Network Streams as a solution but wanted to see if anyone has had success (or failure) with this solution or has a better idea.

 

First, a little background on the application:

In my system, there are N servers running on the network.  Each of these is running on it's own VM with a static IP and unique machine name.

There are also N clients.  These are running on physical machines, VMs and a Windows RDS (Remote Desktop Server).

The messages are all abstract class objects.  Both client and server have a copy of the abstract classes so they can send/receive any of the messages.

My current solution is to use VI Server to push messages from one application to the other.  There is no polling.

Clients and servers can go offline at anytime without warning and the application at the other end of the connection must deal with the disconnect gracefully.

In normal operation, the client will establish connection to a group of servers (controlled by the user).  The client provides it's machine name and VI Server port.  Each server then connects back to the client.  All messages then pass via these connections.

Therefore, a client is uniquely identified to the server by its network machine name.

 

Now the problem.  A Windows RDS session does not have a unique machine name.  Instead it uses the machine name of the RDS.  So, if two users run the client application in two separate RDS sessions at the same time, when they connect to the servers, they will look like the same machine.  There is no way to uniquely identify them and route messages to the correct one.

 

So, I have started looking at network streams as a possible solution.  They looked interesting because you could add a unique identifier to the URL so you could have more than one endpoint on a single machine.  This looked like it would solve my problem because I could combine the machine name and user name to make my client endpoint unique.  But here is the process I would need to follow:

• Server creates an endpoint to listen for client connections.
• Client gets the list of server names (from a central DB) and then tries to connect to each server's endpoint.
• If successful, the client creates a unique reader endpoint for that server and then passes the server the information about how to connect to this endpoint.
• When the server connects to this endpoint, it also creates a unique reader for the client and sends that info.
• The client connects to the server using this unique endpoint and we now have two unique network streams setup between client and server, one in each direction.
• The client then disconnects from the first server endpoint since it now has its own unique connection.
• Both ends will need to have some sort of loop that waits for messages (flattened class objects) and then puts them into the proper local queue for execution within their application.
• Both will also need to be able to send a shutdown message to each of these loops when exiting to close the connection.

My concerns are how to have the server listen for connections with its generic endpoint to multiple clients at once.  There is no guarantee that only one client will try to connect at a time.  And once the client is fully connected and releases the generic connection, will the server be able to listen for more connections or will that endpoint be broken?  How do I reinitialize it?

 

This is all so much easier with VI Server that I might have to just give up on the RDS solution altogether.  But I want to give this my best shot before I do that.

 

Thanks for any tips or ideas or pointing out any pitfalls I missed.

 

-John

[drjdpowell]

Not to distract from the question but I’m interested in why you aren’t using TCP directly.  I haven’t had a multi-machine project in ages, but TCP Listeners and bi-directional TCP Connections seem intended for exactly this kind of problem.

[John Lokanis]

Not sure how to meet all the requirements with the TCP primitives.

1. No Polling. Seems like there will always be a need to poll the connection.  If nothing else then to be able to respond to internal shutdown messages.

2. Robust connections.  How do you re-establish the connection from the server side due to your flaky network dropping it if the client does not have a unique machine name and port to connect to?

 

It seems like I would run into the same issues and would have to design a very complex solution from the ground up.  

[ShaunR]

I have run into a bit of a problem with how I am trying to do networked messaging between a client and a server application.  I am considering trying Network Streams as a solution but wanted to see if anyone has had success (or failure) with this solution or has a better idea.

 

First, a little background on the application:

In my system, there are N servers running on the network.  Each of these is running on it's own VM with a static IP and unique machine name.

There are also N clients.  These are running on physical machines, VMs and a Windows RDS (Remote Desktop Server).

The messages are all abstract class objects.  Both client and server have a copy of the abstract classes so they can send/receive any of the messages.

My current solution is to use VI Server to push messages from one application to the other.  There is no polling.

Clients and servers can go offline at anytime without warning and the application at the other end of the connection must deal with the disconnect gracefully.

In normal operation, the client will establish connection to a group of servers (controlled by the user).  The client provides it's machine name and VI Server port.  Each server then connects back to the client.  All messages then pass via these connections.

Therefore, a client is uniquely identified to the server by its network machine name.

 

Now the problem.  A Windows RDS session does not have a unique machine name.  Instead it uses the machine name of the RDS.  So, if two users run the client application in two separate RDS sessions at the same time, when they connect to the servers, they will look like the same machine.  There is no way to uniquely identify them and route messages to the correct one.

 

So, I have started looking at network streams as a possible solution.  They looked interesting because you could add a unique identifier to the URL so you could have more than one endpoint on a single machine.  This looked like it would solve my problem because I could combine the machine name and user name to make my client endpoint unique.  But here is the process I would need to follow:

• Server creates an endpoint to listen for client connections.
• Client gets the list of server names (from a central DB) and then tries to connect to each server's endpoint.
• If successful, the client creates a unique reader endpoint for that server and then passes the server the information about how to connect to this endpoint.
• When the server connects to this endpoint, it also creates a unique reader for the client and sends that info.
• The client connects to the server using this unique endpoint and we now have two unique network streams setup between client and server, one in each direction.
• The client then disconnects from the first server endpoint since it now has its own unique connection.
• Both ends will need to have some sort of loop that waits for messages (flattened class objects) and then puts them into the proper local queue for execution within their application.
• Both will also need to be able to send a shutdown message to each of these loops when exiting to close the connection.

My concerns are how to have the server listen for connections with its generic endpoint to multiple clients at once.  There is no guarantee that only one client will try to connect at a time.  And once the client is fully connected and releases the generic connection, will the server be able to listen for more connections or will that endpoint be broken?  How do I reinitialize it?

 

This is all so much easier with VI Server that I might have to just give up on the RDS solution altogether.  But I want to give this my best shot before I do that.

 

Thanks for any tips or ideas or pointing out any pitfalls I missed.

 

-John

 

There is an API for interfacing to RDS. As it is a session based system you would need to get the session information and use that to create a unique ID to route your data.

 

Your channel setup process is an almost exact description of the Dispatcher handshake. I notice you have only specified a single connection to a client so  I think for network streams the end points are dedicated to either writing or reading so it would be uni-directional only.

 

You are also missing the "dealer" in your description that needs to copy the data to each end-point if there are multiple clients to a single service. That may or may not be a requirement in your case but most of the time it is needed and you might need to consider control contention if multiple clients are to ultimately all have bi-directional or reverse control channels.

Edited November 18, 2015 by ShaunR

[John Lokanis]

I checked out those MS links.  I am not sure how to use that information since what I am looking for is a way to connect to each session individually using network protocols (IP and Port) if I use VI Server or TCP.  But with Network Streams I can create a unique name.  I suppose I could use the sessionID as that unique name, but that would simply replace the machinename/username unique identifier.

As for the bi-directionality, the proposal I put forth would establish two streams between the server and client, one for command/data in each direction.

Regarding the dealer concept, I already handle that in the server code via a subscription mechanism.  This is abstracted from the transport mechanism.

 

I think the problem that Network Streams has is there is no way around the 1:1 connection issue so I would need some alternate system to establish the connections.  Perhaps a mix of VI Server and Network Streams might work but would be rather complex.

[drjdpowell]

1. No Polling. Seems like there will always be a need to poll the connection.  If nothing else then to be able to respond to internal shutdown messages.

2. Robust connections.  How do you re-establish the connection from the server side due to your flaky network dropping it if the client does not have a unique machine name and port to connect to?

 

It seems like I would run into the same issues and would have to design a very complex solution from the ground up.  

You can cross (1) off at least, as there is no need to poll a TCP connection.   Just have one loop sit waiting for incoming messages, with a second loop to do sending in parallel.  TCP is bi-directional and the two directions don’t interfere with each other, just like a pair of Network Streams.  But being bi-directional, you don’t need to establish a second connection from server back to client, and thus don’t need to know the client name.  If the client can reach the server you are done.  And the TCP Listener is exactly what you asked for as far as a server handling incoming connections.   To me, TCP seems much simpler than what you described.

 

Re (2), I don’t have experience in unreliable networks.  But I have always been confused by Network Streams, as TCP is already a robust network protocol that resends lost packets; what is Network Streams able to do that TCP can’t?   

 

— James

[drjdpowell]

Remember this conversation?  Those examples I gave use a TCP client-server.   They should work with multiple copies of the client.

[MarkCG]

Agree drjdpowell. I went to the trouble of figuring out to create having N instances of an executable opening a pair of network streams to remote targets, handling disconnects gracefully on both ends, and figuring out all the error codes associated with them. In hindsight I should have used the simple tcp messaging library.

Supposedly network streams are goodn or high data throughput of one data type but how often does anyone really need that?

[JKSH]

Agree drjdpowell. I went to the trouble of figuring out to create having N instances of an executable opening a pair of network streams to remote targets, handling disconnects gracefully on both ends, and figuring out all the error codes associated with them. In hindsight I should have used the simple tcp messaging library.

Supposedly network streams are goodn or high data throughput of one data type but how often does anyone really need that?

 

Network streams are built on top of TCP. It provides a simpler interface than TCP for streaming data point-to-point, at the cost of losing the ability to listen for multiple connections. It's nice to be able to dump an array of doubles in one end and get the same array out the other end, without having to search for data boundaries and reinterpret bytes.

 

We are currently developing a system that controls and monitors a pilot plant, using CompactRIOs and CompactDAQs connected to a central PC server. There are a large number of I/O points, a number of which need kHz sample rates. The cRIOs and cDAQs use network streams to transfer the waveform data to the server for logging.

 

Also, our customer wants to allow an arbitrary number of client PCs to view the logged data. The server accepts requests from a client via raw TCP, taking advantage of the TCP Listener function. Once a request is received, the server sends a unique ID to the client to establish a network stream for transferring the waveform data.

[MarkCG]

Network streams are built on top of TCP. It provides a simpler interface than TCP for streaming data point-to-point, at the cost of losing the ability to listen for multiple connections. It's nice to be able to dump an array of doubles in one end and get the same array out the other end, without having to search for data boundaries and reinterpret bytes.

 

We are currently developing a system that controls and monitors a pilot plant, using CompactRIOs and CompactDAQs connected to a central PC server. There are a large number of I/O points, a number of which need kHz sample rates. The cRIOs and cDAQs use network streams to transfer the waveform data to the server for logging.

 

Also, our customer wants to allow an arbitrary number of client PCs to view the logged data. The server accepts requests from a client via raw TCP, taking advantage of the TCP Listener function. Once a request is received, the server sends a unique ID to the client to establish a network stream for transferring the waveform data.

 

 

yes that sounds like a good application for them and what they are designed for. For generic command and event data, where the data type can be anything, at a rate of at most 10s of Hz, I would say just use TCP via the Simple TCP Messaging library. 

[drjdpowell]

It's nice to be able to dump an array of doubles in one end and get the same array out the other end, without having to search for data boundaries and reinterpret bytes.

 

For anyone listening, here is how to send messages (or any other data type) via a TCP connection.  Basically, one just puts a message length at front and reads that first.   TCP handles making sure you never miss a byte, so there is no need to identify data boundaries.

[OlivierL]

We have used NS on a project recently with a cRIO and we found a few caveats that we were not expecting:

1. The NS engine uses ~10% of each core on the cRIO (9067) as soon as a connection is established (we had three streams)
2. There is also an impact on the host CPU, even if very little amount of data is transferred
3. All NS are unidirectional and limited to one connection (based on your description, you are better off with TCP)
4. The NS properly register that the end point is disconnected ONLY if the connection ends gracefully (completing executing on the Host for example.) If you disconnect the network cable, neither the Host nor the cRIO registered the problem automatically and the stream no longer worked when the cable was reconnected. You would need to build that yourself...
5. Memory management in the NS is less than ideal. One of our NS was transferring arrays and we had set the depth of the NS to 100k elements. LV thought it was a good idea to go through each of the 100k in the buffer even is there was ever only 1 or 2 arrays in the buffer. The result was that both the cRIO and the PC were trying to allocate >1GB of memory space that it didn't really need. Sure, bad programming but you would hope for better reaction than what appeared, for the longest time, to be a memory leak. If your messages are long and you create a large buffer, expect to run into strange problems down the road for obscure reasons...

I would suggest you go straight to the TCP method. It will require a little bit more work upfront but you will have more flexibility, less overhead on the CPU and the ability to handle multiple connections easily.

 

NS is great at delivering a simple option to quickly get started with and streaming large amount of data in simple situation but doesn't offer much IMHO benefit over TCP when scaling up.

[JKSH]

For anyone listening, here is how to send messages (or any other data type) via a TCP connection.  Basically, one just puts a message length at front and reads that first.   TCP handles making sure you never miss a byte, so there is no need to identify data boundaries.

 

Clever. The key ingredient I missed was the ability to make the TCP Read node block until the specified number of bytes arrives -- this definitely simplifies things.

[ShaunR]

Clever. The key ingredient I missed was the ability to make the TCP Read node block until the specified number of bytes arrives -- this definitely simplifies things.

This is demonstrated in the shipped TCPI/IP examples with LabVIEW. You are effectively creating a simple proprietary protocol with a single header field of "Data Length". Transport extends this further by adding a timestamp, encryption and compression flags to the header.

Edited November 19, 2015 by ShaunR