cRIO-9040 freezes after several days - Embedded UI and SSH unresponsive, no CPU/RAM saturation

[Ghis]

Hi everyone,

Has anyone experienced a cRIO-9040 freezing after several days of operation, with no obvious CPU or memory overload?

We’re running LabVIEW RT 2021 with Embedded UI, NI Linux RT 8.10 (kernel 5.10.83-rt58), and controller firmware 25.5.0f112-x64.

The application was doing simple temperature regulation and buffered TDMS recording. No plots were displayed and there had been almost no user interaction over the weekend.

Around 4 AM, all logging stopped. That morning:

• The pointer still moved, but neither the LabVIEW UI nor the desktop menu responded.

• SSH was also inaccessible.

• A manual restart was required.

The last resource samples showed roughly 11% / 22% CPU usage and 51% physical memory usage. No explicit crash was found in the retained logs. Unfortunately, /var/log was volatile, so its contents were lost on restart.

Does this sound familiar? What would you enable beforehand to diagnose the next occurrence, especially if SSH becomes unavailable?

I have the logs and more details about the application if useful. Thanks!

[Mads]

It is most probably something project specific, but we did see similar issues with one of our projects on 2021 and all the way to and including 2024, the freezes occured on 9041, 9030 and 9053, but not 9063(!)..

Never found any proper explanations or solutions and had to stick to 2020 until 2025 came out. Have you tried running the same project on the 2020 stack?

Since the 2025 version it has been stable on all the aforementioned targets though...

[Ghis]

Thanks for sharing your experience. We’ve already tried 2025 and observed similar symptoms, then tried 2026 as well.

When you mention the 2025 stack, do you mean LabVIEW RT, the Linux RT image and drivers together?

During your freezes, was SSH also unreachable? In our case, both the application and the embedded desktop stopped responding, and we couldn’t connect via SSH or SFTP.

Edited October 8 by Ghis

[hooovahh]

If you do reach out to NI they will likely ask for a crash dump.  Here is a page on how to enable it and get it:

https://nilrt-docs.ni.com/troubleshooting/lvrt-crash-dumps.html

I've tried sending NI projects with weird behavior, in the hopes they could find and fix LabVIEW bugs.  Some times it works out, but because of the nature of some configurations, and weird compile cache and deployment errors, it can be hard for NI to pinpoint a cause, and create a fix.

[salty_falcon51]

 

Ghis said:

Hi everyone,

Has anyone experienced a cRIO-9040 freezing after several days of operation, with no obvious CPU or memory overload?

We’re running LabVIEW RT 2021 with Embedded UI, NI Linux RT 8.10 (kernel 5.10.83-rt58), and controller firmware 25.5.0f112-x64.

The application was doing simple temperature regulation and buffered TDMS recording. No plots were displayed and there had been almost no user interaction over the weekend.

Around 4 AM, all logging stopped. That morning:

The pointer still moved, but neither the LabVIEW UI nor the desktop menu responded.

SSH was also inaccessible.

A manual restart was required.

The last resource samples showed roughly 11% / 22% CPU usage and 51% physical memory usage. No explicit crash was found in the retained logs. Unfortunately, /var/log was volatile, so its contents were lost on restart.

Does this sound familiar? What would you enable beforehand to diagnose the next occurrence, especially if SSH becomes unavailable?

I have the logs and more details about the application if useful. Thanks!

 

I'd be suspicious of a blocked kernel or filesystem path rather than the app itself, since both UI and SSH disappear while the box still looks alive. The volatile logs bit is brutal, it always fails right when you need the evidence

[Rolf Kalbermatter]

Have you tried running the application without UI? The NI Linux RT UI subsystem is a handy debug tool and tempting to use as local operator interface, despite its somewhat arcane look, but I remember having had quite some trouble in the past when using that for normal operation. Admittingly it's some years ago, ca LabVIEW 2018/19 but certain setups with specific additional software tended to lock up frequently in very similar ways. Sometimes it was only the local UI that hung but often the entire controller seemed kind of hung somewhere and even remote connections couldn't reach it anymore without a power reset. Debug logs sent to NI at that time indicated that resource management in the UI seemed to be buggy and accumulate stale memory objects until the controller run out of resources and the whole OS kind of stopped responding to anything.

Our resolution back then was to only use the local UI for testing and debugging and disable it for in production operation.

Edited 1 hour ago by Rolf Kalbermatter