CompactRIO Locking Up

[smarlow]

Having trouble with a cRIO locking up:

• cRIO-9047 with the Embedded UI enabled.
• LabVIEW 2023Q3 on host and cRIO
• Using 30 sec. restart Watchdog and whacking it ~1 Hz
• Not a really complicated diagram.  
• FPGA programming mode (not scan interface mode)
• I am monitoring the memory usage and it does not appear to be increasing.
• Usually locks up 2-20 hours after start.
• Program and display completely frozen.
• Cannot connect from LV Project or MAX.
• Watchdog controller does not reboot the controller.
• Hard reset required to get the controller running again.
• NI tech support says no internal known issues reported

Has anybody else seen this?  Thanks for any help.

[Neil Pate]

Probably not relevant, but the only time I ever managed to hard-crash (lock-up) a running cRIO was when I was using the (I forget the API exactly) API to set the RT clock. I was doing this once a second to keep some clocks sync'd and apparently the cRIO just didn't like it.

This was so long ago I forgot how I figured out exactly what the problem was. I think there were some log files somewhere that *may* have had a tiny bit of useful info in them.

[smarlow]

Thanks for the info.  I believe I got it fixed.  I ran the latest update for 2023Q3 (f2).  I also did not realize there is a new cRIO firmware image for the cRIO-9047 (23.5).  I was running 8.8.  I am surprised that MAX did not flag the 8.8 firmware as being too old when I installed the 2023 base image.  Anyway, after updating everything, the system has been running for 16 hour with no problem.  I'll let it run over the weekend.  Hopefully, the problem is resolved.  EDIT:  I did completely reformat the cRIO drive after updating the firmware.

Edited February 9 by smarlow

[smarlow]

*SIGH*  Lock-up issue has returned even after firmware upgrade.

[Mads]

We do not have anything identical, but I do relate. The rate of critical bugs on cRIO seem to have risen. We currently have cRIO-9030 and 9041 devices that run fine for anything from 1 day to 3 weeks and then kills an application that we had no issues with prior to moving from 2018 to 2020. No worrying memory-, CPU- or disk usage up front - and nothing in the error logs. Tedious troubleshooting. There was a recall of some 9030s not so long ago too due to faulty components. On some there are even issues with the BIOS (>50% CPU usage when idle anyone...?). The known issues lists contain quite a few serious bugs too.

Simplifying the RT application until the issue goes away is a tredious enough strategy. Add all the recent BIOS, OS, LabVIEW and driver issues and you are in a world of fun...

Here is a similar scenario from 6 years ago: 

 

[Jordan Kuehn]

I've found the newest files edited in this directory provide some insight when you have crashes or issues with your app starting. It's not always intuitive as to what all is there, but I have been able to identify problem components from the legible portions. 

/var/local/natinst/log

I don't share the same experience as Mads regarding reduced reliability or increased bugs, save for the compilation issue I commented on in another thread which was actually a software bug I introduced and discovered via that log directory. We utilize hundreds of cRIOs across the country running for weeks or months at a time in mission critical applications. In fact the reliability of Linux RT is a pretty big factor in keeping with the platform instead of utilizing expansion chassis and a central computer approach. Obviously built-in FPGA is another huge asset when needed.

[smarlow]

Thanks for the links and info.  I will keep plugging away.  BTW, I am logging at 1 Hz to an SD card.  I do not leave the file open.

[Rolf Kalbermatter]

8 hours ago, smarlow said:

Thanks for the links and info.  I will keep plugging away.  BTW, I am logging at 1 Hz to an SD card.  I do not leave the file open.

I would be very hesitant to use SD cards for anything but occasional backups (and not to keep the backups on there, but just to transfer them to a different more safe location). SD card reliability is a big issue and "the" major reason for Rasperry Pi applications in 24/7 unattended operation mode to eventually just die (well not the Raspberry Pi is of course dying but it won't boot up from the SD card anymore eventually). The S of SD is definitely a misnaming. 

Edited February 14 by Rolf Kalbermatter

[smarlow]

Rolf:  Thanks for the info.  I did suspect it might be related to the SD card.  However, I did see the issue before adding the logging module .  I will still try a run without to see if it helps, since the non-logging version that froze was before the updates.

[Rolf Kalbermatter]

2 minutes ago, smarlow said:

Rolf:  Thanks for the info.  I did suspect it might be related to the SD card.  However, I did see the issue before adding the logging module .  I will still try a run without to see if it helps, since the non-logging version that froze was before the updates.

I didn't mean to indicate that the SD card is the reason for the lockup, although it is not entirely impossible. But SD cards are not a good solution for anything but occasional data transfer. It is not a question if they will fail, but when!

[Mads]

Did you figure out this crash @smarlow?

[smarlow]

I believe it is related to having the Embedded UI enabled.  I turned off the embedded UI and ran the system for a few days without the problem.  I did not see this issue with LabVIEW 2020 with the app I am running, so I believe to be a new issue introduced since then (I updated the system to LV 2023 Q3)

Edited March 25 by smarlow
spelling