cRIO-9040 freezes after several days - Embedded UI and SSH unresponsive, no CPU/RAM saturation

[Ghis]

Hi everyone,

Has anyone experienced a cRIO-9040 freezing after several days of operation, with no obvious CPU or memory overload?

We’re running LabVIEW RT 2021 with Embedded UI, NI Linux RT 8.10 (kernel 5.10.83-rt58), and controller firmware 25.5.0f112-x64.

The application was doing simple temperature regulation and buffered TDMS recording. No plots were displayed and there had been almost no user interaction over the weekend.

Around 4 AM, all logging stopped. That morning:

• The pointer still moved, but neither the LabVIEW UI nor the desktop menu responded.

• SSH was also inaccessible.

• A manual restart was required.

The last resource samples showed roughly 11% / 22% CPU usage and 51% physical memory usage. No explicit crash was found in the retained logs. Unfortunately, /var/log was volatile, so its contents were lost on restart.

Does this sound familiar? What would you enable beforehand to diagnose the next occurrence, especially if SSH becomes unavailable?

I have the logs and more details about the application if useful. Thanks!

[Mads]

It is most probably something project specific, but we did see similar issues with one of our projects on 2021 and all the way to and including 2024, the freezes occured on 9041, 9030 and 9053, but not 9063(!)..

Never found any proper explanations or solutions and had to stick to 2020 until 2025 came out. Have you tried running the same project on the 2020 stack?

Since the 2025 version it has been stable on all the aforementioned targets though...

[Ghis]

Thanks for sharing your experience. We’ve already tried 2025 and observed similar symptoms, then tried 2026 as well.

When you mention the 2025 stack, do you mean LabVIEW RT, the Linux RT image and drivers together?

During your freezes, was SSH also unreachable? In our case, both the application and the embedded desktop stopped responding, and we couldn’t connect via SSH or SFTP.

Edited 7 hours ago by Ghis

[hooovahh]

If you do reach out to NI they will likely ask for a crash dump.  Here is a page on how to enable it and get it:

https://nilrt-docs.ni.com/troubleshooting/lvrt-crash-dumps.html

I've tried sending NI projects with weird behavior, in the hopes they could find and fix LabVIEW bugs.  Some times it works out, but because of the nature of some configurations, and weird compile cache and deployment errors, it can be hard for NI to pinpoint a cause, and create a fix.