Problems with losing data using Shared Variables and Datasockets

[ibbuntu]

Hi,

Firstly sorry for the length of this, I have tried to be as succinct as possible. I would appreciate any suggestions anyone might have.

I have been writing code for a while now to use for collecting data in an upcoming experiment. At the core of the code I am using shared variables to pass messages between the different computers on the network. The messages are small, they are a cluster of 4 strings: From, To, Type, Message. The Type field is the type of message being sent e.g. Register, Status, Enable and the Message field contains any extra information. To pass the messages I have a loop which reads the shared variable and checks to see if it has changed since the last time we read it (using the timestamp), it checks to see whether it is relevant (i.e. whether the To: field corresponds to itself) and as a further check which I added later it checks to see if the message is blank. If it sees a new message it adds it to a queue and writes a blank message to the shared variable.

This is the server side, on the client side a message is sent by first checking to see if the message stored in the shared variable is blank, and then writes a message to the shared variable.

This system works well when there is only one object sending and receiving messages from the server, and if it is running on the same computer. However the object is on a different computer, or there are many objects communicating messages are somehow lost and the system grinds to a halt. This can occur if a client is waiting for some information from the server (say a path for it to save its data to), and the message that the server sent was not received, the code will simply sit in an infinite loop waiting for the data.

I originally coded this using plain shared variables, with one for messages coming into the server and one for messages going out of the server. This worked ok, up until I had about 3 things communicating with the server. I decided therefore I would try using one shared variable per object instead, so that I couldn't have the case that two objects were trying to write to the shared variable at the same time. For reasons I won't go into this meant I had to change from using shared variable reads and writes to using the datasocket vis to read and write data. Upon doing this the number of lost messages has increased and the program performs even worse than it did before.

If you got this far, thanks for reading this. I'm really at a loss for what to do, so any suggestions are welcome.

Thanks,

Tom

[Eugen Graf]

Hello Tom, please try this design pattern, I think it will solve your issues.

http://forums.lavag.org/Publish-Subscribe-file122.html

With this pattern you can communicate between different loops and PCs.

P.S. I think any type of variables (locals, globals and shared) takes some traffic. If you don`t want to check if a variable is blank or not, so use VIs from the synchronisation palette. E.g. Queue or notifier, if you want to mix user input with programm data, so use user events.

Reagrds, Eugen

[ibbuntu]

QUOTE (Eugen Graf @ Mar 20 2008, 11:41 PM)

Hello Tom, please try this design pattern, I think it will solve your issues.

http://forums.lavag.org/Publish-Subscribe-file122.html

With this pattern you can communicate between different loops and PCs.

P.S. I think any type of variables (locals, globals and shared) takes some traffic. If you don`t want to check if a variable is blank or not, so use VIs from the synchronisation palette. E.g. Queue or notifier, if you want to mix user input with programm data, so use user events.

Reagrds, Eugen

Thanks for that, this does look like exactly what I am trying to do. I will have a look and see whether I can incorporate this into my code. I guess I am suffering from this problem with my implementation (from the wikipedia page):

QUOTE

As a first example, many pub/sub systems will try to deliver messages for a little while, but then give up. If an application actually needs a stronger guarantee (such as: messages will always be delivered or, if delivery cannot be confirmed, the publisher will be informed), the pub/sub system probably won't have a way to provide that property.

In my application I do require the guarantee that all messages will always be delivered, as my system requires the messages to undergo transitions in state. Such as going from a state of waiting for path information to a state of capturing data and saving to the provided path.

Tom

[wevanarsdale]

QUOTE (ibbuntu @ Mar 20 2008, 04:18 PM)

...For reasons I won't go into this meant I had to change from using shared variable reads and writes to using the datasocket vis to read and write data. Upon doing this the number of lost messages has increased and the program performs even worse than it did before...

I use LabVIEW 7.1 datasockets to broadcast successfully at 600 bytes/sec on up to 9 different sockets. You might try setting the DataSocket Write input ms timeout (0) to -1.

[Eugen Graf]

QUOTE (ibbuntu @ Mar 21 2008, 09:29 AM)

Thanks for that, this does look like exactly what I am trying to do. I will have a look and see whether I can incorporate this into my code. I guess I am suffering from this problem with my implementation (from the wikipedia page):

In my application I do require the guarantee that all messages will always be delivered, as my system requires the messages to undergo transitions in state. Such as going from a state of waiting for path information to a state of capturing data and saving to the provided path.

Tom

Yes, I agree, the dispatcher tryes to send a message to a client, and if it doesn't work, than gives up. But that it only a implementation problem. I am working now on the handshake between dispatcher and clients, that should give more safety in the communication. So you can detect if a client is alive or not. If not you can fire a error message in your programm.

Bot othersides if you want, you can try to send your message again and again from dispatcher to client. In fact I never got problems in communication in my design pattern. Only if a client is not alive or the (TCP/IP) connection is broken.

I don't think the implementation of this pattern in your code is a big issue.

E.g. I installed a dispatcher executable on a web server, and rewritten (in 5 minutes) my TCP-Client-Example to the string messages, so I got a working web-chat. That works very stable. If anybody want to test it, so mail me.

Regards, Eugen

[viSci]

Have you tried Buffered Network Shared Variables? I have used them in applications were multiple clients communicate with a single server. Also you can look at the error

out of the Shared Variable node to see if there are any messages pending. NI has put alot of enginnering into the performance of Network Shared Variables so they should be up to the task you have in mind.

Mike Sachs

Intelligent Systems

[ibbuntu]

Thanks for all your replies. I have been busy at work on my code, as I only have 2 weeks left before the experiment starts (I started writing the code in October, so it's not last minute, it's just taken me a long time to get to grips with writing such a large program in LabView using many varied techniques). As I didn't have time to understand and implement your code Eugen, I decided to modify my code. I now have implemented code to prevent message loss. When I send a message, I check to see if a "received" message has been written to the variable, and if it hasn't then I send the message again (after a 100ms delay).

Now I don't lose any messages and the code works the way it should, apart from the fact that it runs slowly. The loop which sends the messages has to run many times before a message is sent by the client and a "received" message is received from the server. This is variable, sometimes it will happen first time, and sometimes it can take up to 100 times to successfully send the message. I can't cope with such long delays in my code. Any ideas?

Thanks,

Tom

[ibbuntu]

QUOTE (ibbuntu @ Mar 27 2008, 10:33 AM)

Now I don't lose any messages and the code works the way it should, apart from the fact that it runs slowly. The loop which sends the messages has to run many times before a message is sent by the client and a "received" message is received from the server. This is variable, sometimes it will happen first time, and sometimes it can take up to 100 times to successfully send the message. I can't cope with such long delays in my code. Any ideas?

Thanks,

Tom

Ok it appears I had some messages clashing. I have now re-written the code to use the dstp protocol. The code now crashes LabView, but when I run the debugger to find out where it crashes, LabView doesn't crash. What could be causing this? I would provide more information, but I don't know where to start.

(Getting very frustrated now).

Tom

[ibbuntu]

QUOTE (ibbuntu @ Mar 27 2008, 07:02 PM)

Ok it appears I had some messages clashing. I have now re-written the code to use the dstp protocol. The code now crashes LabView, but when I run the debugger to find out where it crashes, LabView doesn't crash. What could be causing this? I would provide more information, but I don't know where to start.

(Getting very frustrated now).

Tom

I went back to using Shared Variables and LabView has stopped crashing. My code now works. Thanks to all who replied.