Timeout of -1 Queue Performance on RT

[QueueYueue]

I'm having some weird behavior on RT on my softmotion application. Softmotion is throwing errors like it's not getting enough CPU to perform the needed tasks, but my CPU usage on the RIO is only in the 50% range.

 

I have several(6-10) concurrent tasks going in my RT code, most of them are idle however, sitting on a dequeue with a -1 for the timeout. My communication manager loop is waiting for a message from the PC, and when it does it will enqueue a message to on of the concurrent tasks. The task will do a little casting as needed, then start executing a task that can take anywhere from 5 sec to 5 min. Some of these tasks involve time critical sub components. None of the time critical parts run for longer than 30 seconds.

 

So as someone who's more experienced with desktop programming, all those dequeues look like 0% processor usage to me. And judging by the distributed system manager, the CRIO seems to agree. When there's nothing going on, my CPU usage drops down to the 15%-20% range. But I'm still getting weird softmotion errors (like -77055).

 

So my question is: Does a dequeue with a -1 timeout tie up the CPU in RT, even though the usage doesn't seem to show it?

[Neil Pate]

Dequeue with -1 timeout should use virtually zero of your CPU resources. You can rely on this mechanism on RT the same as your do on the PC. 

 

RT-FIFOs do have the option where read and writes are polling (using CPU) or blocking (CPU sleeping), but as you state you are using queues and not RT-FIFOs then this does not apply to your application.

 

Is it possible that perhaps something else is going on that periodically causes a big CPU spike? If it is a quick event you may not notice it in your CPU profiling code, but some other bit of code may be affected.

 

The other culprit could be timed loops. Are you using any?

[QueueYueue]

Thanks for the response.

 

Yes I'm using timed loops. Most of the time there is only 1 timed looped running, it is a loop that is responsible for broadcasting some communication values up to the PC. It's set with a low priority and it's period is 200 ms. I've recently gone back and switched my low priorty timed loops out to while loops with waits, and it doesn't seem to be helping at all (although i've only run it once or twice). Related followup question: If a timed loop only has a little bit of work to do in a longer period (2ms of work to do in a 10ms period we'll say) is this affecting my CPU more than I would think?

 

I've done some CPU profiling with RT trace toolkit and there doesn't seem to be any spikes. I usually have a pretty healthy amount of time in the "VxWorks Null thread"

 

Im currently investigating the possibility of a bad 9512 module (admittedly it's a longshot). I have 5 axis that I need to control. 4 of them are on an ethernet rio and all work fine, the one axis that is plugged into the CRIO is the one giving me problems.

[Mellroth]

Thanks for the response.

 

Yes I'm using timed loops. Most of the time there is only 1 timed looped running, it is a loop that is responsible for broadcasting some communication values up to the PC. It's set with a low priority and it's period is 200 ms. I've recently gone back and switched my low priorty timed loops out to while loops with waits, and it doesn't seem to be helping at all (although i've only run it once or twice). Related followup question: If a timed loop only has a little bit of work to do in a longer period (2ms of work to do in a 10ms period we'll say) is this affecting my CPU more than I would think?

 

I've done some CPU profiling with RT trace toolkit and there doesn't seem to be any spikes. I usually have a pretty healthy amount of time in the "VxWorks Null thread"

 

Im currently investigating the possibility of a bad 9512 module (admittedly it's a longshot). I have 5 axis that I need to control. 4 of them are on an ethernet rio and all work fine, the one axis that is plugged into the CRIO is the one giving me problems.

 

What cRIO are you using?

I've seen very bad performance with the older range of cRIOs (VxWorks), and was told that that was because they had a very limited number of threads.

The point was that anything that had a sleep function, like TimedLoop, Dequeue, Enqueue etc. caused the performance to go down, and caused a lot of jittering

 

Our solution was to move to a more poll-based design rather than interrupt driven.

 

/J

[ned]

I think the older cRIOs ran PharLap ETS, and it's the newer ones that run VxWorks? But maybe I have it backwards. A bit off-topic, but I had a possibly-related issue with code on a newer sbRIO (running VxWorks, if I remember correctly) where it crashed or wouldn't run an executable that used more than one execution subsystem. I was in a rush to get it working and so didn't have time to investigate or report the problem, I just set everything to the same execution subsystem and it ran fine.

[QueueYueue]

Thread stuff turned out to be a red herring. the cause of my problem as reseting the CRIO's clock while softmotion was running.

[Jordan Kuehn]

Do you mean resetting the system clock on the cRIO?

[QueueYueue]

Do you mean resetting the system clock on the cRIO?

Yeah, I was trying to sync the CRIOs time with the connecting PC's time, so I could use "Get Date/Time in Seconds" on the CRIO.

 

Sync'ed time was a "nice to have but not necessary" feature, so I just scrapped it.

[Neil Pate]

I recently had some nasty issues with doing something very similar. I have reasonably solid evidence that setting the system clock too often could case the cRIO to crash. I was trying to sync the cRIO clock from a master PC.

 

Something I wanted to try (but never got around to doing it) was to use a time sync protocol rather than "manually" doing it in the RT code.

[Jordan Kuehn]

I've encountered the cRIO rebooting when attempting to adjust the system clock.  This makes for some unpleasant results when trying to sync with a host PC unless you expect it.

[MikeC3]

What cRIO version (OS) were you using when the rebooting when adjusting system clock happened?  We have some systems using 1588 to sync clocks, haven't noticed any problems with that under VxWorks.

[Neil Pate]

 We have some systems using 1588 to sync clocks, haven't noticed any problems with that under VxWorks.

 

That I think is the root of my problem, I was *not* using 1588 (which I presume you set up in the RT system ini file?), rather manually getting time updates over UDP and manually adjusting the system clock (i.e. in code). Doing this too quickly (i.e. more than a couple of times a minute) seemed to cause my RT code to crash horribly (some bits were still running, but not most of it).

 

This happened on a bunch of new cRIO-9024s

[MikeC3]

1588 (which I presume you set up in the RT system ini file?)

 

Had to install the right software to the cRIO, then enable it via MAX (Time Settings tab, under Time Synchronization).  Then some tweaks to the configuration so that we had a consistent master, in order to avoid a situation where on every power cycle, a different clock in our network became master (which caused the time to jump around a lot more than we wanted).

 

Also using cRIO-9024 and 9025s.

Edited July 8, 2014 by MikeC3

[Neil Pate]

Had to install the right software to the cRIO, then enable it via MAX (Time Settings tab, under Time Synchronization).  Then some tweaks to the configuration so that we had a consistent master, in order to avoid a situation where on every power cycle, a different clock in our network became master (which caused the time to jump around a lot more than we wanted).

 

Also using cRIO-9024 and 9025s.

 

Well, hopefully my cRIOs are stable now. I have nine of them running a long term test, with uptimes of about 60 days so far, so hopefully my issues are resolved.

 

Next time I will definitely go the time sync route.