UDP Read of Variable Size Packets

[JimboH]

Hi all,

I had a couple of questions related to reading udp packets in Labview. I am receiving 100 - 200 Mbps of variably sized packets. The first byte in the packet specifies its size in bytes*4.

1) When I perform a read, I have no idea how much data to request, so I set an arbitrary size. It seems in doing this that I am getting multiple packets together and that I subsequently need to parse the data into packets after reading. Doing so seems to slow down the machine a bit and I'm nervous I'm dropping packets (see next point). I also tried reading only 1 byte, then reading X number of bytes based on that 1 byte, but that threw errors (I think it was error 113 about more data being available than I read).

2) How do I tell if I am overflowing the socket buffer? I am getting weird behavior that seems to indicate dropped packets occasionally but no error is coming out of the udp read function.

Thanks,

Jim

[ned]

Reading first byte to determine the number of bytes to read, then doing a second read of that number of bytes is the correct way to do this. Could you check what error you're receiving, and attach the code that demonstrates it? There's no reason that should return an error, and code 113 does not match your description of it.

[asbo]

I suspect you might be getting a *warning* that there is more data than you read, but an error does not make sense, as ned said.

[JamesMc86]

And to add to that, based on the description you should just ignore it as your software is designed for this. You can check for this error code and clear the error cluster if it appears to ignore it.

VISA throws something similar as well.

(null)

[JimboH]

Hi all,

Thanks so much for the help. I had some errors in my implemenation (U8 vs I32) and bytes vs quadlets (bytes*4).

After fixing them the problem remains. I wasn't sure of the error off hand, but I confirmed it is 113. Unfortunately ignoring it seems not to work as there is simply no data returned. My guess is this is a Labview implementation that if this error occurs it chooses to retain the data for a subsequent read rather than require concatenation.

This error occurs on the first read. I thought it might be a startup issue but on subsequent read loops I still get nothing.

Thoughts? Thanks again for the help.

Jim

[asbo]

Are you routing the error 113 back into the read node? The majority of LabVIEW nodes do not perform any significant function if there's an incoming error.

[JimboH]

I am not. There is no error input to the first udp read. The second udp read has an input but I clear the error if it is 113. However, I never get to the second udp read because the first udp read is always empty and I have a case structure around the second one in which it doesn't run if the first one returns no data.

[asbo]

I'm not sure why that's not working for you. You should probably post your VI, just for clarity.

An alternative, though not as "correct" of a solution, would be to have a parallel loop constantly reading as much data as possible (if you can get that to work) and then dumping it onto a queue which is processed by another loop. The consumer loop would have to be able to figure out where packet boundaries are and I don't know how tenuous that would be for your application. If you wanted to add complexity, you could package the UDP data with a boolean that lets the producer hint at whether it's a packet boundary or not.

[TomOrr0W]

I looked at the documentation for the UDP read VI, and I believe you are getting error 113 and no data on that first read because the one byte you attempt to read is part of a larger packet. From the documentation of the Max Size control of that VI:

max size is the maximum number of bytes to read. The default is 548. (Windows) If you wire a value other than 548 to this input, Windows might return an error because the function cannot read fewer bytes than are in a packet.

[JimboH]

Way to read the documenation! Thanks.

I'm not sure how to improve the reading I have. I've tried two options:

1) Processing an entire "read" and concatenating a partial packet (if it exists) to the next read

2) Implementing a circular buffer.

The first was a lot simpler but performance was not great. The second was much more difficult and I didn't see much performance gain (if any). I think it is likely due to memory copying that isn't immediately clear to me. Ideally I would only have one fixed space in memory that I am ever working with for all packets, in addition to the memory generated for each packet that I pull out of the buffer.

I was sort of hoping this was a solved problem with a accepted way of handling it.

Finally, any ideas why I am not getting buffer overflow errors? Does Labview report this?

[TomOrr0W]

I think you are not getting overflow errors because the operating system drops packets silently once the buffer is full. I have no idea why you are getting partial packets - the documentation seems to imply that LabVIEW only reads complete packets (I may be wrong on this). I think asbo's solution is probably the best approach to take with the issue of not reading fast enough.

[ned]

My apologies for misleading you, I suggested the standard approach for TCP and missed the note in the documentation that makes it unsuitable for UDP.

If you post your circular buffer approach I'm sure someone will be happy to suggest how it can be made more efficient.

[JimboH]

Thanks all for the feedback.

I haven't changed the default buffer size yet so I'll give that a try:

http://digital.ni.com/public.nsf/allkb/D5AC7E8AE545322D8625730100604F2D

I'm also doing minimal processing (read a couple of bytes and conditionally throw onto a queue) in addition to the packet splitting, so I hadn't split it up yet into separate read and split loops but I'll try that next.

After some brief web searching it wasn't immediately obvious if there was some system way to detect buffer overflow. I'll probably just do some testing and convince myself that at high rates exceeding my typical usage the system seems stable.

The partial packets may be indicative of a buffer overflow. I'm going to work on getting the system to be more reliable then see if the problem still exists.

[ShaunR]

UDP was quite extensively covered recently here.

You can also download transport.lvlib which handles variable length UDP datagrams.

Edited May 5, 2012 by ShaunR