FYI: cRIO VxWorks Bootup Slowdown Issue

[mpodhor1]

FYI:

For those who use cRIO controllers, there is an acknowledged bug in LV8.5.1 (& possibly 8.6.*) which causes an incremental bootup delay for every boot of the system. These are the conditions under which this occurs:

• Your executable code makes a reference to vilib (signal processing, math, pt-by-pt, etc.); almost anything really.
• Every bootup causes an lvtxxxxxxxx.xml (where xxxxxxxx is unique for every boot) file to be written to /tmp on the cRIO controller. This .xml file has no purpose. I suspect it is some debugging artifact to show what vilib elements are referenced.
• These .xml files will accumulate in your /tmp directory. Though they are not large; they really mess-up VxWorks when enough of them are present. If you manually ftp from the command line to your processor and do an ls on /tmp it will sometimes take a minute to get a response. Your ftp client may timeout or lockup. I suspect the TCP/IP stack for this implementation of VxWorks was not tuned for dealing with this many files.
• Upon subsequent bootups; VxWorks; during initialization performs some type of verification (fsck?) on the flash file system. Since the presence these .xml files in /tmp cause this verification to take longer. My measurements show about a 1 second bootup delay for each 100 .xml files in /tmp. This means your cRIO processor will take 1 second longer to boot for every 100 bootup cycles. This will grow and grow until your controller will no longer boot. A sort of timebomb.
• I suspect in the VxWorks startup code; the file checking / verification routine is first run; then after a max delay your app is started. Thus after a critical number of .xml files are present; the file checking routines will still be running while your startup app begins to run. This has the potential of causing all sorts of problems with your real-time app; as the processor and memory load are all of a sudden different from when you developed and debugged your app.

The solution:

• In your LV code; the very first thing to do is call a subvi to delete all .xml files in /tmp. I also call request deallocation to release any memory resources as a protective measure.
• This will prevent the buildup of .xml files and stop the entire delayed time-bomb effect.

In my case; I was using these controllers in an automotive radar development application which started showing unusual behavior only after a month or two in the field. It took a while to figure out what was happening and why. NI is aware of this issue and I trust it will be fixed in LV2K9; but for all those developers with systems in the field this is one nasty bug to watch out for. My cRIO controllers usually boot in about 15 seconds or so; but I have measured a few that were taking 80+ seconds to boot. The seriousness of this bug for developers will depend on how often their controller gets powered up. For some lucky enough to have the controller always on; possibly a few interruptions per year; no big deal. With other applications that bootup once or more per day; this may be a serious issue.

NI

• Shame shame. This kind of bug should have been squashed long long ago.

[Bug]

Hi..

I m facing the same problem with my digital dyno controller CRIO... The CRIO boot up time slows down with every wake up ..

Can you tell me how can I delete the .xml files from the tmp folder programmitically..

Please help..

Regards,

Bug