Troubleshooting Latency On The Serial Bus

[mje]

Are there any fundamental limitations to the speed of dealing with NI-VISA in relation to serial communication?

Basically I need to continuously handshake off on a series of devices on an RS-422 bus. Each exchange looks something like this:

I send off a string down the bus and wait for a response. My command is a fixed size of 12 bytes and the responses can be of varying sizes from 3 to maybe 20 bytes. I'm operating at 38400 baud and seeing a cycle time of about 30 ms for this exchange. This definitely won't do for the number of devices and how fast we want to run our control loop.

I suspect that most of this latency is from the device that I'm talking to. Do any of you have ideas on how I'd be able to test the response time of the device to confirm this isn't a latency associated with VISA?

I imagine (hope) the overhead on the NI-VISA calls are low enough that I should be able to get close to the maximum data transmission rates on this timescale? If I have 38400 bps and 10 bits per byte (1 start + 1 stop bit) I ought to be able to get close to 3840 bytes/sec I hope?

[asbo]

If you add a 30ms delay between read and write and then time across the VISA read, you should be able to measure the latency of the read call itself, as all the data will already be at the port.

Alternatively, use the read node in a loop, reading only one byte per iteration. This should allow you to see how long it takes to read each byte.

[todd]

If there a "latency" setting somewhere - either in a PN on that VISA reference, or in Device Manager - whose value can be decreased?

[mje]

If you add a 30ms delay between read and write and then time across the VISA read, you should be able to measure the latency of the read call itself, as all the data will already be at the port.

Interesting idea. So I've set things up with a fixed outbound packet size of 13 bytes and a reply of 4 bytes. I can add a delay of up to 5 ms between send/receive primitives without noticing any change in the rate at which data frames arrive. This makes sense since it should take ~4-5 ms to move the 17 bytes. Delays longer than 5 ms add linearly to the observed time between data frames. To me this means my device is responding very quickly.

Hmm, needs more exploring...

Todd, I'm not aware of any such setting.

[asbo]

Interesting idea. So I've set things up with a fixed outbound packet size of 13 bytes and a reply of 4 bytes. I can add a delay of up to 5 ms between send/receive primitives without noticing any change in the rate at which data frames arrive. This makes sense since it should take ~4-5 ms to move the 17 bytes. Delays longer than 5 ms add linearly to the observed time between data frames. To me this means my device is responding very quickly.

Cool, that's kind of what I was expecting to see (though the 30ms I mentioned obviously includes the write duration as well, oops). The obvious approach is to increase the devices' baud rate, but if it were so easy, I suspect you would not be posting.

To follow up on what Todd said, some (most? all? few?) COM ports have some hardware/buffer settings you can dink with. Check out the attached, maybe your COM port is mal-adjusted.

[mje]

Haha, cool. More numbers. So with a single read and single write, I get 30 ms frame time. Subtract out the 5ms transit time over the serial bus and that leaves 12.5 ms per VISA operation.

Since my test is fixed at 4 bytes received, I broke the read operation down to four single byte reads, for a total of five VISA operations per frame. Wouldn't you know, that leads to a frame time of 70 ms. Knock 5 ms off that for the communication time leaving 65 ms, 65/5 operations = 13 ms. Seems to be constant behavior of 12-13 ms per VISA operation.

Those settings don't seem to make a difference on my system. Moving the sliders, unchecking the FIFO option doesn't affect my observed latency. I think my control loop has just hit a brick wall...

[ShaunR]

Haha, cool. More numbers. So with a single read and single write, I get 30 ms frame time. Subtract out the 5ms transit time over the serial bus and that leaves 12.5 ms per VISA operation.

Since my test is fixed at 4 bytes received, I broke the read operation down to four single byte reads, for a total of five VISA operations per frame. Wouldn't you know, that leads to a frame time of 70 ms. Knock 5 ms off that for the communication time leaving 65 ms, 65/5 operations = 13 ms. Seems to be constant behavior of 12-13 ms per VISA operation.

Right click on the read and write VIs, set them to "synchronous" and you might be able to get that down a bit.

[mje]

Negative, Ghost Rider. No observable change. I'm starting to explore options about busting out the Win32 API...

[asbo]

What API calls do you think are relevant? Rather than messing with that, is it possible to put your devices on separate buses and use individual COM ports? That would give you some parallelism which may combat the constant time delay.

The unfortunate fact is that even if you could bump up to 150k baud or higher, you're still only picking at < 20% of your overall iteration duration.

[todd]

Oops, forgot the picture.

This is for a virtual com port, though. It's in the settings because of that particular VCP's DLL.

Dinking was definitely the idea.

[mje]

I've never done serial programming in Win32, but I know you can access COM ports using the basic CreateFile/ReadFile/WriteFile/etc functions. Though this assumes synchronous access which might interfere with my ability to log data. Also looking into async (overlapped) operations. I found http://msdn.microsoft.com/en-us/library/ms810467.aspx, which seems to be a pretty good (though very old) article. No idea though if all that work will get me anything.

You're probably right though. I can live with 30 ms if its not additive per device. Maybe I'll break apart my bus, and just put everything on a dedicated on RS-232 line. Need to hunt down a USB-serial converter, not enough ports to do that at the moment.

Oh yeah, and 38400 bps is the limit for these devices when in party mode on an RS-422 bus. If I put each on a dedicated line I can go faster, but no real need for that if its the VISA calls that are occupying the overhead.

[asbo]

Oops, forgot the picture.

This is for a virtual com port, though. It's in the settings because of that particular VCP's DLL.

Dinking was definitely the idea.

Ahh, I recognize that panel. I have a FTDI RS232-3V3 adapter and I was a little slack-jawed at how much stuff you can tweak, especially compared to my built-in one.

[todd]

I forgot another point: I've used Windows-native com ports at 921600bps (with a full pipe). VISA r/w did not slow me down.

FTDI running at multiple Mbps slowed me down until the latency was adjusted.

[Rolf Kalbermatter]

I forgot another point: I've used Windows-native com ports at 921600bps (with a full pipe). VISA r/w did not slow me down.

FTDI running at multiple Mbps slowed me down until the latency was adjusted.

Todd, if you had read those 921600 bps with one VISA Read per received character it sure enough would have slowed you down

There is definitely an overhead with VISA calls, as they go through several software layers of abstraction so minimizing the number of VISA calls definitely is a good idea. I have to say that 12ms per VISA Read does sound a little excessive though. I'm sure I have seen less on computers that were a lot less powerful than nowadays average low cost computer.

[vugie]

I've never done serial programming in Win32, but I know you can access COM ports using the basic CreateFile/ReadFile/WriteFile/etc functions. ..

Here you may find lvserial library for serial communication. I used it once in application with high performance requirements. Additional advantage is that you don't have to include NI-VISA installer...

[todd]

if you had read those 921600 bps with one VISA Read per received character it sure enough would have slowed you down

I used a separate loop to query "Bytes at Port" then read that many Bytes and put them one by one on a queue. (Termination character did not apply to that binary, streamed protocol.)

With the Windows com port at 921600, groups of between 500 Bytes and 4k Bytes are available as close together as 300ms.

With a VCP DLL at 921600, groups of between 100 Bytes and 3k Bytes are available as close together as 5ms.

So my guess is that the latency settings in the VCP allow for a "long" timeout (255ms before returning: "you expected n Bytes, but they did not all arrive"), and a short timeout of 2ms - which works well for the way I was using it (always reading available Bytes).

Edited May 29, 2012 by todd

[Phillip Brooks]

If your number of ports is more than eight, maybe you could use a terminal server instead. Our ESS setup uses two 16 port terminal servers to attach up to 32 UUTs via RS-232.

We have a mix of code that uses NI-VISA configured for a raw socket and/or use the Internet Toolkit Telnet VIs to the ports. We run from four to 16 hours using TestStand to execute command line tests on the UUTs. Our biggest problem to date has been the reliability/quality of the UUT diags...

Most terminal servers can be configured for 232/422/485 on a per-port basis. I haven't used any of the latest, but some include COM redirect drivers much like the USB-Serial devices to make interfacing easier...

Edited May 29, 2012 by Phillip Brooks

[mje]

Update:I still have no idea why the VISA calls are so slow. As rolfk implied, I expect some amount of slow down due to the various abstraction layers, but to be diplomatic: something in the millisecond timescale is a little unexpected.

I had a few problems with the software from the link vugie shared, so I rolled my own proof with the Windows API.

Using Win32 takes my "frame" duration down to 10 ms. A "frame" involves:

• Writing bytes to the port. This is a simple WriteFile call.
  
• Reading a "line" from the port. This involves looping on ReadFile and concatenating the returned bytes into a buffer until I successfully find my end of line character. The port is configured such that read calls return immediately if there is any data waiting, or to wait for 1 ms for new data if nothing is there.
  

For this test, my write operation is a constant 13 bytes, read operation is a constant 5 bytes. Transit time should be around 5 ms at 38400 baud for the raw data. I have yet to roll this into my main application, so I can't comment on how well it will scale when I start communicating with the four devices on our bus and have variable sized replies (up to perhaps 20 bytes), but I expect I'll be fine since I'm initially targeting for a 10 Hz control loop.

Thanks everyone for the advice,

-m

[Mads]

Based on the scan rate target you mention it seems like you have more than thirty devices on the same port, is that the case? We normally only use 8 devices on each port as we typically get into electrical and timing issues at higher numbers. Port servers are a great solution to get multiple ports, but the virtual port drivers that come with them are *always* crappy (Moxa, Advantech, Westermo...etc) so get one that you can set in TCP/UDP server / raw TCP/IP mode (we mostly use 16 port devices from Moxa).

I wrote a test that simulates both a master and a slave (running either as two separate produce/consume loops, or as a single sequence), and used a set of virtual ports (Eltima) to check how they performed in the scenario you describe. I got down to 25 ms at 38400 baud.

With the virtual ports I can turn off strict baud rate simulation though, that got the cycle time down to 0.6 ms(!). That seems to indicate that the situation is a bit complex. It would be nice to know what happens under the hood that explains the larger than expected (due to actual transfer time) difference.

Edited May 30, 2012 by Mads

[mje]

We only have four devices connected to the bus which are involved in continuous data exchange, and an optional fifth device which is for the most part silent.

Over the last few days I've deployed the Win32 solution into my main application and it works beautifully. For what it's worth, I'll attach the library I made as a wrapper to the Win32 API. Consider it public domain, anyone can do what they will with it.

LabVIEW.Win32Serial LV11SP1.zip

Caveats to those who are downloading it:

• It does exactly what I need it to do, it might not do what you need it to do. That is it's by no means complete.
  
• It operates synchronously.
  
• It's set up so read operations operate by reading a "line" at a time. That is waiting for a line termination sequence. If you want to use other read modes, you'll need to modify the library.
  
• Behavior is undefined if you do not set a timeout.
  
• Timeouts don't quite work the same as they do in LabVIEW. As-is, if you use the Set Timeout method, it will configure the port such that Read Line will timeout if no data is received for the specified duration. That is Read Line can block indefinitely if data slowly trickles in at a rate that's faster than the specified timeout...
  

Have fun with it if you'd like.

-m

[asbo]

Thanks for posting this up. I don't need it at the moment, but it sounds like it'd be interesting to play with. It would probably be small potatoes to add a supervisory bit of code to make the timeout behave in LabVIEW terms.

[mje]

FYI, I've tracked down a bug in the code I posted earlier. All was working well for me until I tried to use it off a fresh boot. I think it's related to the DCB structure that is get/set as part of the open operation.

Anyways, symptoms for me are if I open the port before hand in hyperterminal, close it, then use the library, everything works. However if I use my library right after a boot before any other application has "properly" used the port, then I'm out of luck. I've been playing with the DCB cluster trying to make things work by removing padding bytes, properly setting the DCBLength field, etc, but no luck yet. I know a lot of these settings (if not all) persist on the COM port outside the scope of any application, which is why proper practice is to get the port state before you modify it, and restore it when you're done.

I'll post more info if I track the solution down. For now our procedure is just to make sure you initialize the port with hyperterminal before you use our application.

[asbo]

Does it have to be HyperTerminal? What about just one iteration of using VISA calls?

[shoneill]

Have you tried adding a "flush serial buffer" primitive with your code to speed things up?

 

There are some UART chips which behave different to others (Especially when dealing with USB-RS232 converters).

 

I've had success with that in the past.

[mje]

Indeed, that was one of the first things I tried to resolve the issue. Flushing had no measurable effect as far as I recall. The hardware for this project has since been disassembled so no further testing can be done.