Program architecture

[Francois Aujard]

Hello everyone

I always wonder about the best method to use for data acquisition, filtration, formatting, calibration, etc... when using several different types of communication. 

Where can I find examples to inspire me? 

For example : 
I have : 
- an electronic board with 8 analog inputs, 3 analog outputs, 10 digital outputs and 5 digital inputs.
- an RS485 link with 5 Keller pressure sensors and 2 Bronkhorst flowmeters
- A CAN link with 6 SEW drives
- ModbusTCP link with 3 Kollmorgen drives
- Etc...

I use a lot of different equipment, with different communications. And I'd like to be able to make an adaptable program that can also store component lifetimes (number of solenoid valve openings, number of input and sensor activations if replaced before the board).

Another example of hesitation:
For a TCP-IP communication with 5 sensors, is it preferable to make a Vi that communicates with everyone, or 1 Vi re-entering with 5 instances in memory?

I've started to make Objects, 1 Hardware, 1 Communication, I was thinking of making 1 for scales, 1 for filtration and for calibration... but I'm always a bit afraid of being on the wrong track... and I spend a lot of time... 

Do you have any advice, examples or anything else that might help me? 

Edited September 5, 2023 by Francois Aujard

[ShaunR]

What I'm about to say will either "click" or be totally meaningless with respect to what you are asking. But here we go.

The No.1 goal should to separate the user interface / analysis from the acquisition. That is, how the information is viewed should be completely divorced from how the information was  acquired. To achieve that you will have a surprisingly few number of options with regards to a single, templated architecture. By far the most convenient way is via a database which isn't featured in any classical LabVIEW architectures..

So. We have decided to split the program into 3 halves [SIC ]. The first half (acquisition) will be any method we choose that populates the database with as much information as we need or can elicit. The second half will just be a UI and any analysis that executes queries on the the database and displays it. I am deliberately ignoring configuration but the database also solves this problem for us too.

So the acquisition is just a series of operations to populate the database. Our focus isn't on the LabVIEW architecture for achieving acquisition, it is a LabVIEW architecture for populating a database. How do we do that? It depends. It depends on whether in populating the database with a single piece of information we require more than 1 device to do so. For example. If we only have to read a DVM every 10 seconds then we can just call a VI to read the DVM and put the value in the database with the time . If we have to set a voltage, set a temperature, and then read a DVM then we require something more sophisticated to handle state and populate the database not only with the DVM reading and time , but with the voltage and temperature. However. We are not in the straight jacket of, say, Actor Framework or Producer Consumer et. al. To populate that information we can use any classical LabVIEW architectures for that database update.

As the program evolves, the reusable portions will reveal themselves and a particular classical architecture will become dominant based not only the demands of acquisition but the preferences of the designer (you). However. We are not limited to that one architecture. This process produces an  Emergent Architecture and is greatly expedited by experience. It is what Agile development tries (and fails, IMO) to capture.

So:

Quote

For a TCP-IP communication with 5 sensors, is it preferable to make a Vi that communicates with everyone, or 1 Vi re-entering with 5 instances in memory?

To populate the database if the 5 sensors are identical then you could use "1 Vi re-entering with 5 instances in memory"

To populate the database if the 5 sensors are all different then you could "make a Vi that communicates with everyone" (especially if they are SCPI compliant).

But there are other options....

To populate the database then you could launch an Actor for each device.

To populate the database then you message a Producer Consumer service that handles that device type.

--------------------------------------------------------

Now. Because we have separated the UI and analysis from the Acquisition, you can play with the acquisition implementation details or re-architect/reorganise the acquisition without affecting the analysis and presentation, So if you are operating in an agile environment, the acquisition can go through many tight iterations of the acquisition code and still maintain black-box test integrity for regression testing.

Other options are also valid such as "everything is an object so do POOP"

Edited September 5, 2023 by ShaunR

[Francois Aujard]

Thank you @ShaunR for your reply. 

I have some basic questions (sorry for that): 

- What is a VDM and IMO ?

- SIC = Acquisition / Interface / Configuration ? 

- When you write : "The first half (acquisition) will be any method we choose that populates the database with as much information as we need or can elicit"

Database like SQL or other? No problem with recording speed ? 

 

 

[ShaunR]

56 minutes ago, Francois Aujard said:

What is a VDM and IMO ?

Not sure about VDM but I did say DVM (Digital Volt Meter).

IMO - written abbreviation for "in my opinion": used when you tell someone your opinion, for example in an email.

SIC - sic erat scriptum, "thus was it written")[1] inserted after a quoted word or passage indicates that the quoted matter has been transcribed or translated exactly as found in the source text, complete with any erroneous, archaic, or otherwise nonstandard spelling, punctuation, or grammar. It also applies to any surprising assertion, faulty reasoning, or other matter that might be interpreted as an error of transcription.

1 hour ago, Francois Aujard said:

Database like SQL or other? No problem with recording speed ? 

SQLite is pretty fast but huge amounts of raw data are better stored elsewhere and referenced from a relational database. Key-value stores have their place too.

 

 

[Francois Aujard]

 

1 hour ago, ShaunR said:

Not sure about VDM but I did say DVM (Digital Volt Meter).

Sorry, I can't even copy.... yes, I meant DVM 🙂

 

1 hour ago, ShaunR said:

SIC - sic erat scriptum

ok, I was completely wrong 😄

 

1 hour ago, ShaunR said:

SQLite is pretty fast but huge amounts of raw data are better stored elsewhere and referenced from a relational database. Key-value stores have their place too.

Sorry, but I am not sure to understand... Where else do you want to store? What do you mean by "relational database" ? Are you copying into a double array stored in a global variable ? 

Edited September 5, 2023 by Francois Aujard

[ShaunR]

1 hour ago, Francois Aujard said:

Where else do you want to store? What do you mean by "relational database" ? Are you copying into a double array stored in a global variable ? 

MySQL, MariaDB, SQLite, PostGreSQL are examples of Relational Databases. They consist of tabular data that can be interrogated by their relationships. Key-Value (or Hash lookup) are another type of database and often referred to as NoSQL databases.

No global variables. You may think of the database as a global variable containing all information you have acquired or need to know (it's not really but it will suffice). If these concepts are new to you, then perhaps just concentrate on separating the analysis and display from the acquisition - that is the important part..

[Francois Aujard]

Ok, I Know MySQL. 

We've been using labview for a very long time, but unfortunately it's only been in the last few years that I've taken the time to open up to "best practice" programming methods. The problem I'm having is that I have a huge number of programs for a wide variety of machines. But the codes are all in one block and nothing is separated. Add to that the fact that I haven't capitalized on any code for about 2 decades 😔 ...  The advice you've given me confirms my choice.

Each time a new machine is sold, I have almost no programming time ... bah yes it's the same as the one that was sold with a rotten code... 😒

In short, I'm painfully trying to square everything up, but as some notions are a bit new, I've got an instinct that I'm heading in the right direction, but no certainties and no one to share my questions with... Except for you today 😊🤩

 

I know MySQL but I'd never thought of transferring the acquisition this way....  I was just getting to that point...  I have loops that start but how to transfer the data without going through global variables or FGV.... 

It's far from stupid, but it's going to make me do a lot more work... I'm working in stages between each machine.... It's still not very clear to me, but I can see the principle... 🤯

 

You don't know where I could find a simple example of the principle, do you?

[Reds]

For sure, the biggest hazard of LabVIEW is that it permits you to easily blur the lines between "data acqusition" and "user interface" as ShaunR points out. So dangerous.

I guess a SQL database is one way to draw a hard line in the sand between these two components. But I actually prefer to do it with an API (implemented as a LabVIEW Packed Library). In my opinion, the healthiest architecture decision you can make up front is that your Graphical User Interface will only be allowed to call an API (which you define) in order to configure, read, update, and delete the acquired data. If you have *any* instrument driver or communications code "above" your API, then you've violated your architectural contract.

Edited September 5, 2023 by Reds

[Ryan Podsim]

I would slightly disagree with @ShaunR on 1 point. I would split this into 4 parts, the DAQ, Data storage, UI, and Analysis.  I would use a Pub-Sub system here. The DAQ Publishes the data to whoever is listening( idea is the Storage and UI). This so depend on how 'real-time' you system needs to be and how the systems become live.

I can see the following layout:

• DAQ pushes to storage
• Storage Consumes from DAQ
• UI pulls from Storage
• Analysis pulls from Storage

Or 

• DAQ pushes to Storage and UI
• UI consumes from the DAQ
• Analysis pulls from Storage

Separating the Storage from everything will allow you change different elements in the future. Whether you choose pushing to the UI or the UI pulling will depend on your update rates. If you have a high DAQ update rate, then I would suggest having the UI pull from Storage. High meaning over 50 times per second.

[drjdpowell]

15 hours ago, Francois Aujard said:

I have loops that start but how to transfer the data without going through global variables or FGV.... 

Quite a lot of good architectures are based on using messages between semi-independent modules, with the modules being based on a standardized template.  There are a few such "frameworks" publicly available via vipm.io that you could look at.  Mine is "Messenger Library", which has some instructional videos.

[Francois Aujard]

Thank you all for your answers,

@Reds I've never done an API with Labview. I've used APIs before, but I've never created one. I don't know how to do that.... 😔

@Ryan Podsim  For the moment, my data storage is an FGV (measurement) which is filled by my acquisition loops. And my UI reads this FGV and makes requests to the acquisition loops via another FGV (Consgines). I was in the process of changing this mode to event subscriptions, but it's still not very clear in my head... When my UI requests specific information from one of my acquisition loops (e.g. flow totalizer reset), it's a dynamic event (A) that makes the request. However, to be sure that the acquisition loop has executed the request, and to validate that the action has been carried out, I need another dynamic event (B) to do so. I find this a bit complicated.... My loop time is between 2 and 30ms (depending on the hardware), since I'm using Windows.

@drjdpowell thanks for the link, I'll have a look... and I'll try to understand 😊

Edited September 6, 2023 by Francois Aujard