Jump to content

file transfer using UDP protocol


Recommended Posts

Hi everybody

I hope someone can help me. I'm trying to build a client-server file transfer application. The client reads a file stored on a given working station(PC) and sends it to a server installed on another working station for further usage. I am using UDP as the transport protocol. The problem is that my application works perfectly with files whose size is in the range of kilobytes, however when I try to send files whose seize is in the range of megabytes, the application crashes down and the attached error message shows up

I understood that I have to increase the size of both the UDP sending and receiver buffers, but I don't know how to do so. please if you have any idea on how to do so, just let me know

Best regardspost-26861-0-34497200-1334856682.png

Edited by moralyan
Link to post

UDP does not guarantee ordering, non-duplication or even delivery of "chunks", so you would need to make sure each write contains enough information to completely identify which part of which file it is. That way you could reassemble them on the other side. Even then you may not get them all, so probably want a way to ask for some part of the file to be resent.

These are things that TCP would take care of for you. Do you really need to use UDP?

  • Like 2
Link to post

Here's a KB article on setting the read buffer. We had to do this because the unit we are testing is sending at Gb speeds and we would loose the packets in Windows. It only speaks of the receive buffer, but I imagine the send buffer is very similar if even needed.

Lost UDP Packets at Fast Transfer Rates

The TFTP example looks promising as well.

Link to post

Can you send your data in chunks? You really don't want to set your datagram size to megabytes.

honestly i have never read about chunks, could you give me some details please

UDP does not guarantee ordering, non-duplication or even delivery of "chunks", so you would need to make sure each write contains enough information to completely identify which part of which file it is. That way you could reassemble them on the other side. Even then you may not get them all, so probably want a way to ask for some part of the file to be resent.

These are things that TCP would take care of for you. Do you really need to use UDP?

i have built successfully an aplication that transfer files using TCP, however it is really slow, this is why i need to use a faster protocol like the UDP

There's a TFTP example on the NI website: https://decibel.ni.c...t/docs/DOC-8301

No need to reinvent the wheel.

ok you can ask my teacher about that, he insist on building a new application!!

Here's a KB article on setting the read buffer. We had to do this because the unit we are testing is sending at Gb speeds and we would loose the packets in Windows. It only speaks of the receive buffer, but I imagine the send buffer is very similar if even needed.

Lost UDP Packets at Fast Transfer Rates

The TFTP example looks promising as well.

thanks for the link, i have already tried it and actually it can't give a transfer rate more than 2Mb/s, which is far from my goals!!

Link to post

honestly i have never read about chunks, could you give me some details please

i have built successfully an aplication that transfer files using TCP, however it is really slow, this is why i need to use a faster protocol like the UDP

ok you can ask my teacher about that, he insist on building a new application!!

thanks for the link, i have already tried it and actually it can't give a transfer rate more than 2Mb/s, which is far from my goals!!

Sorry I didn't realize this is a homework assignment. In that case, you can learn about TFTP and implement your own solution by reading RFC 783 and RFC 1350. This will also give you an understanding of what chunks are.

Good Luck!

~Dan

  • Like 1
Link to post

i have built successfully an aplication that transfer files using TCP, however it is really slow, this is why i need to use a faster protocol like the UDP

...

thanks for the link, i have already tried it and actually it can't give a transfer rate more than 2Mb/s, which is far from my goals!!

It's incorrect to say that UDP is a "faster" protocol. It is more lightweight than TCP, but that doesn't inherently make it faster - especially in the use case you have, where you'll basically need to handle everything that TCP does automatically (and at a lower level).

Speeds far in excess of 2Mbps are feasible using TCP, so you probably need to take a look at your implementation and maybe talk to your teacher about what you can do to optimize it.

  • Like 2
Link to post

Sorry I didn't realize this is a homework assignment. In that case, you can learn about TFTP and implement your own solution by reading RFC 783 and RFC 1350. This will also give you an understanding of what chunks are.

Good Luck!

~Dan

thanks a lot for the help, i'll read what you mentioned even thogh i still think the solution resides in increasing the size of buffers!!

It's incorrect to say that UDP is a "faster" protocol. It is more lightweight than TCP, but that doesn't inherently make it faster - especially in the use case you have, where you'll basically need to handle everything that TCP does automatically (and at a lower level).

Speeds far in excess of 2Mbps are feasible using TCP, so you probably need to take a look at your implementation and maybe talk to your teacher about what you can do to optimize it.

i think you are right, and i'll go back to TCP, but i still want to increase the buffers size, how can i do so??

Link to post

Under windows, the default udp datagram size is 1500 bytes (i.e the default MTU). The maximum it can be is 65535 bytes (you can set it to more than this, but it will still be limited to 65k ). This is because the UDP header field is a U16 and cannot represent more than 65k.

UDP, User Datagram Protocol

Edited by ShaunR
  • Like 2
Link to post

Under windows, the default udp datagram size is 1500 bytes (i.e the default MTU). The maximum it can be is 65535 bytes (you can set it to more than this, but it will still be limited to 65k ). This is because the UDP header field is a U16 and cannot represent more than 65k.

UDP, User Datagram Protocol

so you wanna say that there is no possibility to go beyond transfer rates of 65Kb/s using UDP!!

Link to post

so you wanna say that there is no possibility to go beyond transfer rates of 65Kb/s using UDP!!

No. I'm saying the payload cannot exceed 65k. If you can send 1 payload (of 65k) every 10ms then that would be 6.5 MB/sec

Edited by ShaunR
  • Like 2
Link to post

No. I'm saying the payload cannot exceed 65k. If you can send 1 payload (of 65k) every 10ms then that would be 6.5 MB/sec

Brother, this is exactly what i wanna do, I have a 600Mb TDMS file and i need to transfer it as fast as possible. now i understand that the maximum data packet size is 65Kb, which implies that i must devide my 600Mb file into packets of 65Kb each and then send those packets using UDP. My question is: how can i devide my file into packets??

Link to post

Brother, this is exactly what i wanna do, I have a 600Mb TDMS file and i need to transfer it as fast as possible. now i understand that the maximum data packet size is 65Kb, which implies that i must devide my 600Mb file into packets of 65Kb each and then send those packets using UDP. My question is: how can i devide my file into packets??

Just read 65k bytes (or 1500 bytes is better for the reasons Rolf outlined) from the file at a time and write it using the UDP write. However. As GregR stated. It is not guaranteed to be recieved in the correct order, or indeed a chunk to be received at all! So that's not a lot of good for files, but usually acceptable for audio or video where a lost frame or two doesn't matter too much.

The easy solution, as others are saying, is to use TCPIP where all the ordering and receiving reassembly is handled for you (as well as other things UDP doesn't do out-of-the-box such as congestion control, re-transmission of lost packets etc). If you really must use UDP then you will have to code all that yourself (unless missing or incorrectly ordered file chunks is acceptable-for TDMS it wouldn't be). There are a few examples of reliable UDP (RDP). Noteably UDT and NORM. I am not aware of anyone doing this in Labview however, since it is just easier to use TCPIP.

Edited by ShaunR
  • Like 2
Link to post

Just read 65k bytes (or 1500 bytes is better for the reasons Rolf outlined) from the file at a time and write it using the UDP write. However. As GregR stated. It is not guaranteed to be recieved in the correct order, or indeed a chunk to be received at all! So that's not a lot of good for files, but usually acceptable for audio or video where a lost frame or two doesn't matter too much.

The easy solution, as others are saying, is to use TCPIP where all the ordering and receiving reassembly is handled for you (as well as other things UDP doesn't do out-of-the-box such as congestion control, re-transmission of lost packets etc). If you really must use UDP then you will have to code all that yourself (unless missing or incorrectly ordered file chunks is acceptable-for TDMS it wouldn't be). There are a few examples of reliable UDP (RDP). Noteably UDT and NORM. I am not aware of anyone doing this in Labview however, since it is just easier to use TCPIP.

I do appreciate, and i am with you for the choice of TCP/IP instead o UDP, but unfortunatly, my teacher doesn't do so. He is insisting on UDP!!. Anyway, i have tried what you have suggested, but i got the same errror, and i think it is logical, if the buffer size is only 65Kb or less, how can it handle 600Mb!! what i want to know is what is the code i should use to devide my file into packets of 65Kb, because wiring the 600Mb file directly to the UDP Read VI didn't work. or shall i use a while loop?

Link to post

shall i use a while loop?

Yes.

This is a similar VI that it used in the "OPP Push File" in the LAVA CR.

It opens a file and reads "chunks" that it then sends (in this case) via bluetooth. You need to do the same except with UDP rather than bluetooth..

Edited by ShaunR
  • Like 1
Link to post

Yes.

This is a similar VI that it used in the "OPP Push File" in the LAVA CR.

It opens a file and reads "chunks" that it then sends (in this case) via bluetooth. You need to do the same except with UDP rather than bluetooth..

thanks ShaunR, i will try this, but could you provide me with more details about "chunks", what are they exactely??

Link to post

It means a piece. A bit. A portion.......of the file.

For GOD sake, this is my VI, please help me, i know TCP is better, but i just wanna see it working with UDP, so what shall i change to not see this error message anymore :(

post-26861-0-39564500-1335127492_thumb.p

Link to post

You are loading the ENTIRE FILE then trying to transmit (all 600MB presumably) in one go. To remove the error you need to send a maximum of 1500 bytes of the file A BIT AT A TIME by reading the first 1500 bytes, write that to the UDP port then reading the next 1500 bytes and write that .... and so on - a method of which I demonstrated to you in the image posted for bluetooth earlier (the code for which you can download from the Code Repository).

Short of writing it for you (and it is not my home-work), there is not a lot else I can tell you. Perhaps someone else can explain it better than I :rolleyes:

Edited by ShaunR
  • Like 1
Link to post

For GOD sake, this is my VI, please help me, i know TCP is better, but i just wanna see it working with UDP, so what shall i change to not see this error message anymore :(

You'll find that getting angry at people trying to help you here won't get you far, especially if it is a homework assignment and the goal is for you to figure it out on your own. I agree with Phillip, the information here + some Googling will get you where you need to go.

Link to post

You are loading the ENTIRE FILE then trying to transmit (all 600MB presumably) in one go. To remove the error you need to send a maximum of 1500 bytes of the file A BIT AT A TIME by reading the first 1500 bytes, write that to the UDP port then reading the next 1500 bytes and write that .... and so on - a method of which I demonstrated to you in the image posted for bluetooth earlier (the code for which you can download from the Code Repository).

Short of writing it for you (and it is not my home-work), there is not a lot else I can tell you. Perhaps someone else can explain it better than I :rolleyes:

There is nothing wrong with your explaining skills! There are however people who do not want things to be explained to them but instead things being done for them. Or English is not the native language but the posts look pretty ok to me in grammar, so why someone with such a grasp of the language never has heard about "chunks" is beyond me.

Also while reading a file in chunks and sending it over UDP will probably work in a local network, I'm pretty sure the resulting file on the receiver end will be damaged after being transmitted like that over a more complex network infrastructure. That is unless one adds also packet sequencing code to the transmission (and in doing so more or less reimplements the TCP protocol on top of UDP).

  • Like 1
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Content

    • By iayestaran
      Hi everyone,
      I am perceiving some strange behavior in my UDP connections. Every time I re-start my cRIO 9063, the UDP Write block outputs ERROR 66 (LabVIEW:  The network connection was closed by the peer. If you are using the Open VI Reference function on a remote VI Server connection, verify that the machine is allowed access by selecting Tools>>Options>>VI Server on the server side.) and it does not transmit. 
      The most bizarre thing is that if I simply stop the execution and re-launch it again (given that I am executing the code of the cRIO in Debug mode from a Windows laptop), the UDP connection behaves perfectly well. The only time it fails is the first time I launch the program after I reset the cRIO. For me, it looks like some service is not properly initialized when the cRIO is started.
      I attach a picture with the UDP connections of my LabVIEW RT program. The 'Comms RX' VI simply calls the 'UDP Read' function, and the 'Comms TX' calls the 'UDP Write'. The 'Open UDP Conn' also simply calls 'UDP Open'. 
      Can anyone help me?

    • By Calorified
      Hi there,

       

      I have RIO in a Windows VirtualBox inside a Ubunbtu Host OS.

       

      I am sending data from a C++ program in the Ubuntu Host system to labview within the Guest OS.

      I can receive the data on labview installed on the windows guest os. Below is the png of the Windows working program.

       



       

      But when I tried to send the data to myRIO, I was getting a udp read only error 42.

       

      Somewhere on the NI forums, someone suggested the net address of the "UDP Multicast Read-Only" vi  be wired to the address of the RIO which I have done below

       



      .

       

      Now, the code runs on myRIO but I can't receive any data on RIO. The multicast address I am sending to from Boost Asio C++ is 235.255.0.1 on port 30001.

       

      The RIO has a public ip of 172.22.11.2 and I set up a static ip address for it in NI MAX as the address of the UDP Multicast : "235.255.0.1.

       

       

       

      At this moment, I do not see what I am missing. All firewalls have been disabled and I have set the permissions for RIO through the Windows security page.

       

      Any help would be appreciated.

      Thank you!

    • By Calorified
      I have a UDP connection setup on a Windows 7 machine that runs an IMAQ code. The machine can send and receive udp connection so long as data being sent it is on the same machine. When I send the code from a different process (running on Windows 7), I am able to read the data on the RIO udp receiver VI (running on the Windows Host) as well.
      So I changed the sending program to a labview code and the udp receiver won't pick up the data.
      Could there be something I am missing?
      Attached are the PNG files (lava1 is the talker (same as the Kinect Code VI) while lava2 is the receiver).
      Any clues would be appreciated.


      Kinect Code.vi
    • By ShaunR
      Name: Transport.lvlib
      Submitter: ShaunR
      Submitted: 27 Aug 2011
      Category: Remote Control, Monitoring and the Internet
      LabVIEW Version: 2009License Type: Other (included with download)



      Transport.lvlib is a LabView API to simplify and accelerate networked communication development.
      It simplifies development by abstracting TCPIP, UDP and Bluetooth and TLS interfaces
      into a single polymorphic vi which is a thin wrapper around the conventional
      open, read, write, close and listener VIs for all the network interfaces.
       
      Features:
      Supports TCP/IP, Bluetooth, UDP (p2p, broadcast and multicast) and TLS.
      Supports symetric encryption (blowfish).
      Supports compression (zlib).
       
      INSTALLATION:
      Run the supplied installer and follow the instructions.

      Click here to download this file
    • By hilbert00
      Dear LabVIEW friends, 
       
        I'm running into a problem distribution my application. This is opening a UDP connection registering a service as explained in the documentation. It is perfectly working when running within the development system and also as a standalone application on the development computer. 
       
      The problem is with the installer and the distribution package. When I install the application of a different computer, it fails with error 63 at the Open UDP. Reading the available documentation it looks like it is a Service Locator fault. So I checked and indeed the NI Service Locator is not running on that machine and it is not even distributed with the installer. 
       
      I've tried to go through the list of additional installers in the Installer build properties but couldn't find anything useful. 
       
      I'm sure I'm doing badly something wrong, can you help me in sorting this out?
       
      Cheers,
      toto
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.