CAN Taking 100% RT CPU and another problem with the Reset

[Foreshadow]

I have been on this problem for a few weeks now and still unable to fix the problem:

Description of the problem:

1- When reading from 2 CAN channels with a Real-Time PXI Controller, the Processor runs between 70% and 80% (but most often at 80%). (Nothing else runs on the RT, but it will be required that more vis to run on it in the futur.) When running the vi on the Host PC, the processor on the RT controller tops at 100%. Some Data loss occurs.

2- Minor problem: Board doesn't always reset even with 2 resets.)

Program Content (view attachement for more details):

1- While Loop at 2500us containing a sequence structure to read 2 Canbus ports in sequence. To read the CAN, I use a ncGetAttr (Read Entries Pending) followed with a ncReadNetMult.v. Data is then unbundled and set into an array in order to use a shared variable (using Real-Time FIFO). This alone takes 80% of the CPU LOAD.

2- First sequence: Canbus Reset.vi. Second Sequence has the Config (set at 1MHz), Open then CActions of Reset and Start.

Description of Task:

Read 2 CAN channels from a UUT on a Real-Time PXI Controller (8196). 16 CAN frames per whole CAN Message per channel. The 16 CAN frames are sent in 2.5ms. The Host PC that reads the data from the Real-Time Controller at any rate (I usually set it at 4 times the loop time of the RT Target).

FIFO Configuration:

12 elements of the CAN frame + RT TickCounter. As 64bits Double or 64 bits U64. 13 elements per array. 512 Arrays (this number varied alot). The lower the # of arrays, the more random data loss occurs, but the higher it is, the more the duration of data loss is long. Note that sometimes i get the last frame and appears in the log as many times as loop size.

Host PC:

Reads the Shared Variable in a For Loop with the #the same as the FIFO size (or higher).

Note: Many Disable diagrams, but doesn't affect the program in any way. Its other tests done.

Thank you

[LAVA 1.0 Content]

Is the timed loop necessary?

Using a timed loop that fast with that much code in it is doomed to overrun it's timer. (something you don't check for).

I would read the values available, sent them via a queue element by element to a lower priority loop, that does the publishing of the shared variable.

Should the two can's read one after another? If so just place them in one frame with error clusters connected.

How you have programmed it you throw away the parallel power of LabVIEW.

What currently happens:

t->

PollC1----ReadC1----ProcessDataC1-----PublishDataC1

-----------------------------------------------------PollC2----ReadC2----ProcessDataC2-----PublishDataC2

What could happen (if you want it):

t->

PollC1----ReadC1----ProcessDataC1-----PublishDataC1

PollC2----ReadC2----ProcessDataC2-----PublishDataC2

Now if you start queueing data the following can happen:

t->

PollC1----ReadC1----SendQ1

PollC2----ReadC2----SendQ2

ReadQ1----ProcessDataC1-----PublishDataC1

ReadQ1----ProcessDataC2-----PublishDataC2

I think you will have more predictable timing schemes in the last case, give every row it's own loop.

Why do you need an RT read? The current setup does not read every 2.5 ms a can message. It reads every 2.5 ms all the can messages.

What is the observed length of your FIFO? If this increases your listener is to slow and might cause timing hickups.

Ton

[Foreshadow]

I tried with 2 while loops with each one having the loop content but without the sequence structure but it was horrible, there was a stop of reading from a CAN every now and then, so much data was lost. The reason why the while loop is at 2.5ms, is because we rely on the TickCounter of the RT Controller in order to get results of tests or to execute tests. I know we can use the difference of timing of the CAN timing vs the TickCount, but for some reason the project manager doesn't seem trust the timing of the Canbus as we do not know when the CAN timing is executed on the unit

I tried with Queues on my simulation (I created a Windows VI to simulate canbus data when I was unable to work on the testers with the unit and RT controller), but it took too more time to execute the Queues than the timing of the loop (was confirmed when using the profile tool). Unless the Queues are executed differently on the RT than on a Windows machine then i'll try it. I based my simulation with the NI Example Queue basics.vi (find example/Optimizing Applications/Syncho Task).

I'll try it on the RT Controller.

With a 2.5ms loop, it reads the 16 CAN frames, so the array is usually 15-17 CAN frames long. What would be the best ratio between the loop from the RT Target and the loop from the Host PC ? I think i'll also optimize the CReadMult to do all the operations of it on the Host PC. Oh and the timed loop is necessary as it runs on a RT Controller.

Thanks.

QUOTE (Ton @ Nov 14 2008, 12:32 PM)

Is the timed loop necessary?

Using a timed loop that fast with that much code in it is doomed to overrun it's timer. (something you don't check for).

I would read the values available, sent them via a queue element by element to a lower priority loop, that does the publishing of the shared variable.

Should the two can's read one after another? If so just place them in one frame with error clusters connected.

How you have programmed it you throw away the parallel power of LabVIEW.

What currently happens:

t->

PollC1----ReadC1----ProcessDataC1-----PublishDataC1

-----------------------------------------------------PollC2----ReadC2----ProcessDataC2-----PublishDataC2

What could happen (if you want it):

t->

PollC1----ReadC1----ProcessDataC1-----PublishDataC1

PollC2----ReadC2----ProcessDataC2-----PublishDataC2

Now if you start queueing data the following can happen:

t->

PollC1----ReadC1----SendQ1

PollC2----ReadC2----SendQ2

ReadQ1----ProcessDataC1-----PublishDataC1

ReadQ1----ProcessDataC2-----PublishDataC2

I think you will have more predictable timing schemes in the last case, give every row it's own loop.

Why do you need an RT read? The current setup does not read every 2.5 ms a can message. It reads every 2.5 ms all the can messages.

What is the observed length of your FIFO? If this increases your listener is to slow and might cause timing hickups.

Ton

[asbo]

What does Canbus Reset.vi look like?

[Götz Becker]

An (old) way to reset a NI CAN board can be found at: ncReset.vi. It looks like it still works, currently we are using it in a RT App.

With NI-CAN in fast loops we had timing problems too (1ms at first, now we are down to 5ms (just enough to satisfy an alive counter for the UUT)). It seems like that the NI-CAN boards are generally using much time on the system bus (no dma), thus are slow.

Have you tried the RT Execution Trace Toolkit to see were your time is lost? It adds some major overhead while tracing, but then you'll have a chance of narrowing the timing problems down. We found (actually we believe/hope it) for our application that the CAN frame API calls are using the PAL Thread which itself does the PCI bus communications. While these are running all other bus access can cause trouble for the loop (e.g. networking).

Have you tried setting your PXI Controller to Ethernet Polling Mode (probably will decrease transfer speed, but maybe enough for your app)? I have seen some really horrible traces in which the ethernet interrupt handling thread preempted high prio closed loop control loops causing timing problems.

Have you tried to lower the rate in which you write in the shared variables (processing in the for loops and later writing a larger chunk in the variable)?

Queues on RT should be fast enough!?! Have you tried them with fixed size and preallocation at creation? Did they ran full and then causing delay when enqueueing?

Just my 5 cents.