Advice for building high-reliability LabVIEW process control VI?

[Onno]

Hi all,

Here at my university, someone asked me for advice on how to build a process control VI for a scientific experiment. What he described seemed like a stereotypical LabVIEW application: the computer has to control a couple of pumps and switches, and apply some basic PID control to keep, for example, the pH (acidity) of a solution constant.

All was well, until he added that this experiment has to run for at least a year; any interruption of a few seconds in the process control would mess up the experiment. Oops.

So, it seems this should be a high-reliability LabVIEW application — something on a scale with which I don't have any experience. So I'd like to ask you for some advice: how to handle this? Any experiences you'd like to share?

1. A few things that seem obvious:
   
   • disconnect the controlling PCs from the network;
     
   • disable Windows Updates (but that follows from the previous point, really);
     
   • have two PCs, and implement some sort of failover mechanism (but how?);
     
   • get a UPS, or maybe even a backup generator;
     
   • test extensively, by simulating all sorts of failure scenarios (any tips here?);
     
   • extensive logging and monitoring.
     

[*]Anything else?

[*]Do you know about any good whitepapers or examples? (I have tried Googling around, but haven't been able to come up with much useful information).

[*]Or should we ditch the idea of building this ourselves, and get some professionals to look at it?

Any pointers would be very welcome; thanks a lot in advance!

Best,

Onno

[crelf]

I strongly suggest you get a dedicated controller for the mission-critical tasks (think cFP or cRIO) and handle the non-mission critical tasks on the PC (like displaying data, logging to a database or wherever. This means that if the PC goes offline for some reason, the controller continues humming away at what it does best: determinisitc control. Then, you only really need to worry about infrastrucute support (eg: UPS) for the controller - it can continue to control and save data locally while the other system is offline. Also, if you go to the PC to get data off it (I don't expect you're going to run the system completely untouched for a year, right? Not even look at any of the data), doing so won't interupt the process.

have two PCs, and implement some sort of failover mechanism (but how?);

This is a common option in mission critical systems: we built a similar system a few years ago with 3 parallel controllers (PXI) that could take over from each other within 1ms of a detected failure (those specs are probably overkill for your application, but the technology remains the same). We achevied this using reflected memory (a PXI card with a fiber optic link between them, that all "shared" the same memory) - this worked really well. Another option is to stream reflective memory over a local LAN dedicated to the controllers.

Or should we ditch the idea of building this ourselves, and get some professionals to look at it?

I've never been to Amsterdam...

The answer to that question depends on just how important that determinism is. If you trust your engineers to make something that won't fail, then maybe do it alone. That said, if they misplace one bit, and the whole thing comes crashing down in the last month of the experiment, you might be cranky An even worse scenario (which I've seem many times) is when it *looks* like everything worked fine, but there's an offset or skew in your data that you don't find out about until you've published - *that* would be a nightmare!

[Roderic]

There are many solutions for your problem. Crelf already gave you some good advices.

Forget about windows, you'll need a deterministic system (Real Time) and looking at the small amount of sensors I guess a cRIO would be perfect (maybe two cRIO).

Have you considered RAID disks for data backup?

IMHO, if you don't feel like you can do it alone you should contact an NI Alliance partner. I'm sure you'll find someone with a great backround in deterministic systems.

[Tim_S]

I concur with the above. The actual control, depending on how complex it is, could be a closed loop controller with two digital outputs. Something more complex could be a PLC or equivalent (cRIO, cFP, etc.). These devices are meant to run day-in-and-out for years on end. The data logging can be a real-time or PC system with other features depending on how critical data loss could be.

[Onno]

Thank you, all, for the advice; it's very helpful!

An especially warm "thank you" to you, crelf, for the extensive post I'd never have thought of the cFP/cRIO systems, since I have absolutely nil experience with them. But yes, come to think of it, it makes a lot of sense.

The guy in charge of the project should be back from holiday by the end of August, so I'll discuss it with him then. But as far as I'm concerned, I'd definitely consider bringing in some external assistance. Just not sure if the research budget would allow for it!

If I end up getting involved in this, I'll post back some time in the future with my experiences!

Edited July 28, 2012 by Onno

[crelf]

An especially warm "thank you" to you, crelf, for the extensive post

No problem - that's what LAVA's here for. Now it's your turn to go assist someone else here that needs your assistance.

But as far as I'm concerned, I'd definitely consider bringing in some external assistance. Just not sure if the research budget would allow for it!

Well, I'd argue the question isn't about being able to afford bringing in external assistance when there's an appreciable risk of loosing all your data - can you imagine the "well why didn't we spend a little more money up front?!?" conversation then?

[Roderic]

Maybe you can limit the external help by having someone building the architecture and you coding all the sub VIs...

I'm sure with some effort you can find all the required info and coding techniques to achieve your system. But like crelf said, if you end up with a buggy system spending some money may be the best option.

[Jordan Kuehn]

don't forget the sbRIO

[Jeffrey Habets]

Sounds indeed like cRIO for the deterministic part would fit in perfectly here.

And I just happen to know a fine Alliance Member here in the Netherlands that loves this sort of projects and is quite proficient with LabVIEW and cRIO.

Hey, that seems to be a link to their site down there..

[crelf]

Hey, that seems to be a link to their site down there..

[Onno]

No problem - that's what LAVA's here for. Now it's your turn to go assist someone else here that needs your assistance.

Haha, maybe I'm lurking too much But I'm doing my best to chime in where it seems useful!

Well, I'd argue the question isn't about being able to afford bringing in external assistance when there's an appreciable risk of loosing all your data - can you imagine the "well why didn't we spend a little more money up front?!?" conversation then?

I couldn't agree more. But it's not my project, and I don't get to decide anything... But I'll definitely do my best to get that message across!

And I just happen to know a fine Alliance Member here in the Netherlands that loves this sort of projects and is quite proficient with LabVIEW and cRIO.

Hey, that seems to be a link to their site down there..

What a coincidence I might just happen to, by pure chance, pass on your contact info, but it's going to take another month until the guy in charge of the project is back here. So don't hold your breath

[Jeffrey Habets]

What a coincidence I might just happen to, by pure chance, pass on your contact info, but it's going to take another month until the guy in charge of the project is back here. So don't hold your breath

I won't.. I'll keep on breathing.. Hot Texan air for the coming week.