Jump to content

Sending sensor data from a TCP socket in python(RPI) to LABVIEW


Diegodemf

Recommended Posts

Hello, I'm trying to send data from my RPI b+ to labview and store the information in a spreadsheet, right now the data I'm using is just a sawtooth generated in the python code but eventually it will be sensor data adquired via spi, the data comes at =~ 6-7khz but the time it takes to send the data to labview is much greater, I fear the circular buffer I intended to use to store the data before sending it via tcp will get filled pretty fast. I'm not very experienced in either python nor Labview, I was wondering if i could get some advice in the optimization of the code.

image.png.37a88c58f2590ccb515199adbeafbb41.png

I attached the VI of the client and a VI that graphs the data on the spreadsheet, also the python code.

Thanks in advance, sorry for the mess.

Waveform Grapher.vi

Client.vi

serverSerial.py

Link to comment

Hi Diego, 

to speed things up, you should consider separating the TCP read  and the file writing in two separate loops. For example, if you make two parallel loops: one that reads and adds data to a queue, and a second which reads the queue and writes to file, then you can read multiple datapoints at once and write less frequently to the file.

Investigate the use of a Producer-Consumer architecture.

I took the liberty of reorganizing your example to demonstrate the concept. I'd probably approach your task where your Python code is the client that publishes data and LabVIEW would act as a server (instead of the other way around), but I wanted to keep with the spirit of your example.

image.thumb.png.e47e7d3ae765e7c2240d8bb30b37f57c.png

 

 

Client.vi

  • Like 1
Link to comment

I like your idea, and will probably try to adopt it, but I didn't get much of an improvement in terms of performance, I have tried just with the tcp read part and the time is similar. I think my option is increasing the size of the packet sent each time, but with 6000 bytes the inmediate setting its not working correctly, it sends "?" several times between reads, the speed is a little better:

image.thumb.png.bc14050e12e2ac4ed326424068753491.png

Using standard and 30ms timeout I get less errors but the speed is much worse:

image.thumb.png.5931d3753d4eb2982857d22724fed937.png

Also with this solution i would need to rearrange the array efficently, and for some reason the unflatten from json throws me no data( neither in this new setup nor in the previous one, I have had to parse manually)

Link to comment

Prepend the packet size to each message in your python script then you can use the following method which is the most efficient  Watch out for the endianness of the size bytes. LabVIEW is big endian (or network byte order). Once you do this you can get fancy later by adding timestamps and whatnot, if you want, since you effectively just using a custom packet header..

Untitled.png.0efcf234de4746147e971622a20f3f3b.png

Next. find out about the Nagle Algorythm settings on the RPi. I think on Windows the default window is 8K bytes but I have no idea for RPi what it is set to. This will limit transmitting to be either sent immediately (if greater than the buffer) or after N millisecs if less. (Again, windows is default to 250ms). Decide if you need to turn it off to achieve what you need. Note that this only affects the server side.

You haven't said how you are connecting (wired ethernet or Wifi). Wired ethernet is far superior for high speed acquisition than Wifi. wired ethernet will  easilyget you up to 80% of the the bandwidth available - Wifi you'll be lucky to get 50%.

Link to comment

I'm super confused...it sounds like you just want to stream data, right?

So latency isn't important -- just throughput. So turning off nagle won't help.

However you're introducing a latency issue by having LabVIEW request data by sending a string with "?" in it?

The stated goal is 7 kHz * 50 bytes, which is 2.8 Mbit/s -- hardly anything. But when you introduce round trip communication into the streaming pathway, you're introducing slow performance.

Can you clarify if there is an actual  need for this back and forth?

 

How important is reliability? If latency is unimportant and reliability is very important, I'd suggest writing your data to disk on the raspberry and then ftp/scp/webdav-ing it off.  If reliability is not important, and this is on a local network, consider UDP.

Edited by smithd
Link to comment

Hello, thanks for the answers.
Right now the connection is made via wired ethernet (LAN) but ideally it would work with wifi as well

Yes, I want to stream data, having a bit of initial latency is okay, as long the throughput is continous afterwards, the application has to work on real time. Via SPI we will get 7 measurements, the application will be for power measurement,  via SPI I get 7 words of 16 bits each, thats the info i want to forward, using numbers of that size in the example the function socket.send returns 60 bytes send( i assume this is without the headers) . You're right I do not need to send data from Labview to the RPI, i can try to make the server in Labview and the client, but i think i will still need to gather a good amount of measurements to send it in a big packet to improve the speed.

The most important thing is reliability, I have thought of ftp/scp but i haven't found a way to connect those services directly with labview and automate the process of reading on the PC and erasing the file on the rpi, as the space is limited on it, the data would need to be separated in several files and erase the earlier ones as it writes the new ones.

Any suggestion? Thanks in advance.

Link to comment

 

14 hours ago, Diegodemf said:

i think i will still need to gather a good amount of measurements to send it in a big packet to improve the speed.

However this will increase latency. That is the tradeoff. To be reasonably efficient with the network you'd need to transfer bursts of ~20 samples over wired ethernet. 

14 hours ago, Diegodemf said:

the application has to work on real time.

How real-time is real time? What does the labview code do with the information?

14 hours ago, Diegodemf said:

The most important thing is reliability

Then my answer would be that you should write to a file, but as above it depends on exactly how important it is. What happens if you lose a sample? What happens if the network disconnects for 30 seconds?

14 hours ago, Diegodemf said:

I have thought of ftp/scp but i haven't found a way to connect those services directly with labview and automate the process of reading on the PC and erasing the file on the rpi

Yes, you would have to split the file up yourself. The strategy I follow is to have a directory /data with a file /data/temp.ext and a subdirectory /data/done which contains data1.ext, data2.ext, etc. I then use the built-in labview FTP (I believe the connectivity palette but I'm not sure) functions to fetch files out of the /data/done folder. That way I don't accidentally pull off a half-finished file. This appears to be automated in python: https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler and https://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/

If you don't like FTP for some reason, someone put together a winscp wrapper library: https://lavag.org/topic/20474-free-labview-winscp-library/?tab=comments#comment-124582

 

 

Link to comment
11 hours ago, smithd said:

 

However this will increase latency. That is the tradeoff. To be reasonably efficient with the network you'd need to transfer bursts of ~20 samples over wired ethernet. 

How real-time is real time? What does the labview code do with the information?

Then my answer would be that you should write to a file, but as above it depends on exactly how important it is. What happens if you lose a sample? What happens if the network disconnects for 30 seconds?

Yes, you would have to split the file up yourself. The strategy I follow is to have a directory /data with a file /data/temp.ext and a subdirectory /data/done which contains data1.ext, data2.ext, etc. I then use the built-in labview FTP (I believe the connectivity palette but I'm not sure) functions to fetch files out of the /data/done folder. That way I don't accidentally pull off a half-finished file. This appears to be automated in python: https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler and https://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/

If you don't like FTP for some reason, someone put together a winscp wrapper library: https://lavag.org/topic/20474-free-labview-winscp-library/?tab=comments#comment-124582

 

 

After the acquisition of the data the goal is to calculate and graph Electric power with the Voltages and currents and several factors related to it as phases, harmonics, power factor,max values, sudden changes,etc.

As of now i'm only graphing  with the VI listed above, I'm doing this right now reading the file written in the PC, but this is something I need to work at.

The device should be acquiring data for around 2 hours, losing 1 or 2 samples in between should not be a big issue but losing 30 secs of data might be important, the information should be as reliable as possible, in that way, writting to a file seems to be the more reliable option to store the data with the least amount of loss.

I'll take a look into the FTP matter and get back at you, thanks for the help

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Content

    • By mhsjx
      Hi,
      I'm a beginner in labview, and now test cRIO about two weeks. I still can not solve the problem. I attach my test project for explanation.
      I want to realize that , for example, with time sequence t1, t2, t3, t4,  DO outputs T, F, T, F, AO1 outputs A1, A2, A3, A4, AO2 outputs B1, B2, B3, B4, and the delay of AO1 and AO2 should as small as possible(AO1 and AO2 may comes from difference modules).
      I search in Google, NI forum, and decide to use for loop and loop timer in FPGA.
      The reason as follow:
      1. To realize the specific time interval, I can use Wait and Loop timer. But in "FPGA 0--Test DO.vi", it can't not realize specific time interval by several us's error(maybe large). And to complete once of while loop, it needs 134us. I can't explain that it can realize time interval below 134us, even I acturally realize a delay of 10us, but the input is not acturally 10us, so it's not accurate. 
      And by NI example, I use the Loop timer.
      2. In "FPGA 1--Test DO and AO.vi", I find that the loop timer helps me to realize accurate time interval, however, it ignore the first time interval. Such as, t1, t2, t3, t4, with disired output A1, A2, A3, A4. It goes A1(t2), A2(t3), A3(t4), A4(t1). And in "FPGA 2--Test DO and AO.vi", it has same problem. DO0 and AO1 goes A1(t2), A2(t3), A3(t4), A4(t1). And AO0 is always ahead of DO of t1. 
       
      The people of NI forum advice that I should put AO0 and AO1 into one FPGA/IO node and use SCTL. But up to now, I don't find any example about it(Google or NI forum, maybe it's primary).  Mainly that AO0 and AO1 must go with different timeline, the dimension of input array is different.  Can anyone offer advice for me?
      Thanks
      Test.7z
    • By kpaladiya
      I would like to build a model using image data and NI-cRIO-9063 and NI 9264 for voltage control.
      for image, I made a script in python using OpenCV libraries that detecting some points . For voltage control, I use cRIO-9063 with NI 9264 voltage controller.
      My question is, I am new in LabVIEW and I don't have any idea how can I make a loop for voltage control in python. Is there any library available in python that directly connect cRIO and NI 9264 devices? if not then how can I combine my image data(which is in python) with cRIO device? I need argent help.
    • By Makrem Amara
      Hi there,
      I am working on a machine vision project with LabVIEW.
      The camera will locate some parts and send their coordinates via TCP/IP 
      and I created a client also with LabVIEW to display these coordinates here is how the communication going.
      First, if the camera detects something then msg will be sent to the client to inform him.
      then if msg was received correctly client responds with another msg to request the coordinates. 
      at last, the server sends the coordinates to the client. 
      here I faced some problems
      1- the msg sent are with variable length ("x=0,y=0,Rz=0"==> "x=225,y=255,Rz=5" ==> "x=225,y=255,Rz=90"  length vary between 16 and 22 ) with the constant "byte to read " it will not display the full msg.
      2-the client works fine but at a certain time, it shows errors like ("LabVIEW: (Hex 0x80) Open connection limit exceeded";;;;;; "LabVIEW: (Hex 0x42) The network connection was closed by the peer. If you are using the Open VI Reference function on a remote VI Server connection, verify that the machine is allowed access by selecting Tools>>Options>>VI Server on the server side"
       
       





    • By drjdpowell
      I am just starting on trying to be able to use Python code from a LabVIEW application (mostly for some image analysis stuff).  This is for a large project where some programmers are more comfortable developing in Python than LabVIEW.  I have not done any Python before, and their seem to be a bewildering array of options; many IDE's, Libraries, and Python-LabVIEW connectors.  
      So I was wondering if people who have been using Python with LabVIEW can give their experiences and describe what set of technologies they use.
    • By Yaw Mensah
      I have installed Labview 2020 on Dedian Buster using the rpm to deb conversion method via alien. Due to Architecture mismatch i deleted the *i386.rpm files before conversion.
      My Problem is that after creating a project at "Build Specification"-> "rigth click" i am only able to select "Source Distribution". Application does not show up as an option. 
      I will be grateful for any suggestions.
      Thank you in advance.
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.