Sending sensor data from a TCP socket in python(RPI) to LABVIEW

[Diegodemf]

Hello, I'm trying to send data from my RPI b+ to labview and store the information in a spreadsheet, right now the data I'm using is just a sawtooth generated in the python code but eventually it will be sensor data adquired via spi, the data comes at =~ 6-7khz but the time it takes to send the data to labview is much greater, I fear the circular buffer I intended to use to store the data before sending it via tcp will get filled pretty fast. I'm not very experienced in either python nor Labview, I was wondering if i could get some advice in the optimization of the code.

I attached the VI of the client and a VI that graphs the data on the spreadsheet, also the python code.

Thanks in advance, sorry for the mess.

Waveform Grapher.vi

Client.vi

serverSerial.py

[Francois Normandin]

Hi Diego, 

to speed things up, you should consider separating the TCP read  and the file writing in two separate loops. For example, if you make two parallel loops: one that reads and adds data to a queue, and a second which reads the queue and writes to file, then you can read multiple datapoints at once and write less frequently to the file.

Investigate the use of a Producer-Consumer architecture.

I took the liberty of reorganizing your example to demonstrate the concept. I'd probably approach your task where your Python code is the client that publishes data and LabVIEW would act as a server (instead of the other way around), but I wanted to keep with the spirit of your example.

 

 

Client.vi

[Diegodemf]

I like your idea, and will probably try to adopt it, but I didn't get much of an improvement in terms of performance, I have tried just with the tcp read part and the time is similar. I think my option is increasing the size of the packet sent each time, but with 6000 bytes the inmediate setting its not working correctly, it sends "?" several times between reads, the speed is a little better:

Using standard and 30ms timeout I get less errors but the speed is much worse:

Also with this solution i would need to rearrange the array efficently, and for some reason the unflatten from json throws me no data( neither in this new setup nor in the previous one, I have had to parse manually)

[ShaunR]

Prepend the packet size to each message in your python script then you can use the following method which is the most efficient  Watch out for the endianness of the size bytes. LabVIEW is big endian (or network byte order). Once you do this you can get fancy later by adding timestamps and whatnot, if you want, since you effectively just using a custom packet header..

Next. find out about the Nagle Algorythm settings on the RPi. I think on Windows the default window is 8K bytes but I have no idea for RPi what it is set to. This will limit transmitting to be either sent immediately (if greater than the buffer) or after N millisecs if less. (Again, windows is default to 250ms). Decide if you need to turn it off to achieve what you need. Note that this only affects the server side.

You haven't said how you are connecting (wired ethernet or Wifi). Wired ethernet is far superior for high speed acquisition than Wifi. wired ethernet will  easilyget you up to 80% of the the bandwidth available - Wifi you'll be lucky to get 50%.

[smithd]

I'm super confused...it sounds like you just want to stream data, right?

So latency isn't important -- just throughput. So turning off nagle won't help.

However you're introducing a latency issue by having LabVIEW request data by sending a string with "?" in it?

The stated goal is 7 kHz * 50 bytes, which is 2.8 Mbit/s -- hardly anything. But when you introduce round trip communication into the streaming pathway, you're introducing slow performance.

Can you clarify if there is an actual  need for this back and forth?

 

How important is reliability? If latency is unimportant and reliability is very important, I'd suggest writing your data to disk on the raspberry and then ftp/scp/webdav-ing it off.  If reliability is not important, and this is on a local network, consider UDP.

Edited December 3, 2018 by smithd

[Diegodemf]

Hello, thanks for the answers.
Right now the connection is made via wired ethernet (LAN) but ideally it would work with wifi as well

Yes, I want to stream data, having a bit of initial latency is okay, as long the throughput is continous afterwards, the application has to work on real time. Via SPI we will get 7 measurements, the application will be for power measurement,  via SPI I get 7 words of 16 bits each, thats the info i want to forward, using numbers of that size in the example the function socket.send returns 60 bytes send( i assume this is without the headers) . You're right I do not need to send data from Labview to the RPI, i can try to make the server in Labview and the client, but i think i will still need to gather a good amount of measurements to send it in a big packet to improve the speed.

The most important thing is reliability, I have thought of ftp/scp but i haven't found a way to connect those services directly with labview and automate the process of reading on the PC and erasing the file on the rpi, as the space is limited on it, the data would need to be separated in several files and erase the earlier ones as it writes the new ones.

Any suggestion? Thanks in advance.

[smithd]

 

14 hours ago, Diegodemf said:

i think i will still need to gather a good amount of measurements to send it in a big packet to improve the speed.

However this will increase latency. That is the tradeoff. To be reasonably efficient with the network you'd need to transfer bursts of ~20 samples over wired ethernet. 

14 hours ago, Diegodemf said:

the application has to work on real time.

How real-time is real time? What does the labview code do with the information?

14 hours ago, Diegodemf said:

The most important thing is reliability

Then my answer would be that you should write to a file, but as above it depends on exactly how important it is. What happens if you lose a sample? What happens if the network disconnects for 30 seconds?

14 hours ago, Diegodemf said:

I have thought of ftp/scp but i haven't found a way to connect those services directly with labview and automate the process of reading on the PC and erasing the file on the rpi

Yes, you would have to split the file up yourself. The strategy I follow is to have a directory /data with a file /data/temp.ext and a subdirectory /data/done which contains data1.ext, data2.ext, etc. I then use the built-in labview FTP (I believe the connectivity palette but I'm not sure) functions to fetch files out of the /data/done folder. That way I don't accidentally pull off a half-finished file. This appears to be automated in python: https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler and https://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/

If you don't like FTP for some reason, someone put together a winscp wrapper library: https://lavag.org/topic/20474-free-labview-winscp-library/?tab=comments#comment-124582

 

 

[Diegodemf]

11 hours ago, smithd said:

 

However this will increase latency. That is the tradeoff. To be reasonably efficient with the network you'd need to transfer bursts of ~20 samples over wired ethernet. 

How real-time is real time? What does the labview code do with the information?

Then my answer would be that you should write to a file, but as above it depends on exactly how important it is. What happens if you lose a sample? What happens if the network disconnects for 30 seconds?

Yes, you would have to split the file up yourself. The strategy I follow is to have a directory /data with a file /data/temp.ext and a subdirectory /data/done which contains data1.ext, data2.ext, etc. I then use the built-in labview FTP (I believe the connectivity palette but I'm not sure) functions to fetch files out of the /data/done folder. That way I don't accidentally pull off a half-finished file. This appears to be automated in python: https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler and https://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/

If you don't like FTP for some reason, someone put together a winscp wrapper library: https://lavag.org/topic/20474-free-labview-winscp-library/?tab=comments#comment-124582

 

 

After the acquisition of the data the goal is to calculate and graph Electric power with the Voltages and currents and several factors related to it as phases, harmonics, power factor,max values, sudden changes,etc.

As of now i'm only graphing  with the VI listed above, I'm doing this right now reading the file written in the PC, but this is something I need to work at.

The device should be acquiring data for around 2 hours, losing 1 or 2 samples in between should not be a big issue but losing 30 secs of data might be important, the information should be as reliable as possible, in that way, writting to a file seems to be the more reliable option to store the data with the least amount of loss.

I'll take a look into the FTP matter and get back at you, thanks for the help