Alternative to Citadel for logging and displaying time series data

[TTGrey]

Hi,

My company has a software/hardware product which is deployed as a labview executable. The software is of the SCADA type and it typically monitors and controls 5-20 machines using modbus. We also log data from each machine (every ten minutes, 10-20 data points per machine). The gui allows the operator to trend data in 24 hour increments by selecting the machines and day of the year. We have been using DSC and Citadel to store the data as traces which allows us to locate the data with a start and stop time for display. However, my company is not happy about the licensing of DSC and has asked me to look into other options. 

Part of the issue with DSC is that although it is supported by NI, it is no longer being developed, fixed or improved. An example of this is the DSC modbus IO servers which map to shared variables. There are known issues with how certain write functions are implemented which make it impossible to write to many modbus devices. NI has acknowledged the problem but is not going to fix it. So we have developed our own modbus routines, making use of the NI modbus library.

On the data logging and trending side, I would like to know if there are reasonable alternatives within the labview environment that don't require a DSC license on each deployed machine? I know that citadel uses TDMS and it seems like a great format, but I'm not sure if it would be too much work to build out code for logging and retrieving time series data using the low level functions that labview provides. As you can probably tell, i'm familiar with labview but don't have a lot of background in databases and data management.

Does anyone have experience trying something similar?

Thanks

[smithd]

A team at NI has created an application engine which may help do what you want: http://ni.com/dcaf
Its a configurable plugin engine which maps a plugin (for example there is one which polls modbus values at a periodic rate) to scoped data storage inside of the engine and then maps that data out to other plugins (for example a TDMS writer). It can obviously get more complicated as you add custom logic, but I think they've been doing a pretty good job on getting that to be easier as well. If it sounds helpful, the guys working on it are very accessible so just message them or post in that group. For retrieving data from TDMS and processing it, I think most anyone at NI would recommend diadem, but its not really a scada tool so much as a fancy excel tool -- displaying it on the fly for an operator might be tougher.

I'm not personally aware of anything that would help with everything. Something that may help partially is the new OPC UA module, licensed separately from DSC (I think its something like $500 for a seat and then maybe $100 for a deploy, if I remember right). I say new because the outside looks the same, but it adds alarms and a historical server, built in. You'd essentially copy your modbus variables into a OPC UA server instance and then clients could read N samples worth of historical data (ie you could maybe store the previous day in memory). Once its in OPC UA land, I would bet you could find some other vendor with a good long term logger.

Along similar lines, the RTI DDS toolkit is a similar protocol library where the RTI folks sell add-on toolkits, like loggers, which consume the published data. So again you'd read modbus variables, copy them into DDS, and run a third-party service to do the logging and history.

Edited December 1, 2017 by smithd

[TTGrey]

25 minutes ago, smithd said:

A team at NI has created an application engine which may help do what you want: http://ni.com/dcaf
Its a configurable plugin engine which maps a plugin (for example there is one which polls modbus values at a periodic rate) to scoped data storage inside of the engine and then maps that data out to other plugins (for example a TDMS writer). It can obviously get more complicated as you add custom logic, but I think they've been doing a pretty good job on getting that to be easier as well. If it sounds helpful, the guys working on it are very accessible so just message them or post in that group. For retrieving data from TDMS and processing it, I think most anyone at NI would recommend diadem, but its not really a scada tool so much as a fancy excel tool -- displaying it on the fly for an operator might be tougher.

Stepping back to the more broad "i want to make a scada app in labview" concept, I'm not personally aware of anything that would help with everything. Something that may help partially is the new OPC UA module, licensed separately from DSC (I think its something like $500 for a seat and then maybe $100 for a deploy, if I remember right). I say new because the outside looks the same, but it adds alarms and a historical server, built in. You'd essentially copy your modbus variables into a OPC UA server instance and then clients could read N samples worth of historical data (ie you could maybe store the previous day in memory). Once its in OPC UA land, I would bet you could find some other vendor with a good long term logger.

Along similar lines, the RTI DDS toolkit is a similar protocol library where the RTI folks sell add-on toolkits, like loggers, which consume the published data. So again you'd read modbus variables, copy them into DDS, and run a third-party service to do the logging and history.

Thank you smithd. I have explored the dcaf option in the past. Ultimately, it looked like porting our LOOP system over to dcaf would create a system too hard to manage.

The OPC UA option is appealing because the OPC server could also handle the modbus communication. I wasn't aware that there were third party apps that could log the data and then access it through labview APIs. I will definitely look into it. However, it seems like the licensing could still be an issue, especially if we would have to buy a OPC UA license and a DSC runtime license for each deployment.

I'm still looking at your RTI DDS suggestion but I don't think I have enough of a grasp on it to understand if it's a viable solution.

[smithd]

1 hour ago, TTGrey said:

The OPC UA option is appealing because the OPC server could also handle the modbus communication. I wasn't aware that there were third party apps that could log the data and then access it through labview APIs. I will definitely look into it. However, it seems like the licensing could still be an issue, especially if we would have to buy a OPC UA license and a DSC runtime license for each deployment.

I'm still looking at your RTI DDS suggestion but I don't think I have enough of a grasp on it to understand if it's a viable solution.

For DDS, my (admittedly also vague) understanding is that they use the DDS publish subscribe model to develop various tools which consume the data. A concept diagram (ie marketing material) can be found on this page: https://www.rti.com/products/dds and the addon product would be the database integration service: https://www.rti.com/products/dds/add-on-products
Looking at it more thoroughly I think its probably way past what you need and geared towards a different use case.

As for the OPC UA, you probably wouldnt need a DSC runtime license in that situation, since you're using the separate modbus library. To make sure I'm being clear, I would see you running an embedded OPC UA server within your application, which itself could host a short term historical log. For longer term logging I'd imagine using a different vendor with the capability of pulling data from any OPC UA server, using OPC UA as the standard scada protocol backbone for your system. I don't have a specific recommendation for this, but a quick google "OPC UA data historian" comes up with this as an example: https://opcfoundation.org/products/view/prosys-opc-ua-historian and that company also has an opc ua client and this thing: https://www.prosysopc.com/products/opc-ua-modbus-server
All that having been said, if you have a very small system and licensing costs are a concern, all of the above is probably overkill. A lot of these integrated logging solutions seemed to be geared towards bigger systems and so is the pricing.

Edit:
These two occurred to me:
https://www.daq.io/what-daqio-is/
Looks like its a sort of integrated solution for historical data using a simple web-based labview API. I cannot speak to it past having seen a demo at some point.

https://www.influxdata.com/time-series-platform/telegraf/
I had thought about using this myself at one point, but my goals got redirected. As I recall its a time-series database with visualization tooling "chronograf" and a simple HTTPS interface for inserting data.

Edited December 1, 2017 by smithd

[TTGrey]

1 hour ago, smithd said:

For DDS, my (admittedly also vague) understanding is that they use the DDS publish subscribe model to develop various tools which consume the data. A concept diagram (ie marketing material) can be found on this page: https://www.rti.com/products/dds and the addon product would be the database integration service: https://www.rti.com/products/dds/add-on-products
Looking at it more thoroughly I think its probably way past what you need and geared towards a different use case.

As for the OPC UA, you probably wouldnt need a DSC runtime license in that situation, since you're using the separate modbus library. To make sure I'm being clear, I would see you running an embedded OPC UA server within your application, which itself could host a short term historical log. For longer term logging I'd imagine using a different vendor with the capability of pulling data from any OPC UA server, using OPC UA as the standard scada protocol backbone for your system. I don't have a specific recommendation for this, but a quick google "OPC UA data historian" comes up with this as an example: https://opcfoundation.org/products/view/prosys-opc-ua-historian and that company also has an opc ua client and this thing: https://www.prosysopc.com/products/opc-ua-modbus-server
All that having been said, if you have a very small system and licensing costs are a concern, all of the above is probably overkill. A lot of these integrated logging solutions seemed to be geared towards bigger systems and so is the pricing.

Edit:
These two occurred to me:
https://www.daq.io/what-daqio-is/
Looks like its a sort of integrated solution for historical data using a simple web-based labview API. I cannot speak to it past having seen a demo at some point.

https://www.influxdata.com/time-series-platform/telegraf/
I had thought about using this myself at one point, but my goals got redirected. As I recall its a time-series database with visualization tooling "chronograf" and a simple HTTPS interface for inserting data.

Smithd,

I really appreciate the help. It will take me a while to sift through everything. I've spent most of the day so far looking into the OPC UA solution. The daq.io product looks great for remote access to data, but i'm looking more for local access right now. The influxdata products look like they are powerful. I'm not sure how they would interface with labview yet. Thanks again.

[smithd]

On 12/1/2017 at 11:43 PM, TTGrey said:

The influxdata products look like they are powerful. I'm not sure how they would interface with labview yet. Thanks again.

Thats actually the easy part, which is why I was thinking about it as an option. The specific help doc is here (https://docs.influxdata.com/influxdb/v0.9/guides/writing_data/) but basically you'd just use the built-in HTTP(s) client library in labview with the url and data string as shown on that page, essentially "my_interesting_measurement,<metadata fields> value=123.456". 

 

Also for daq.io, it looked like they had a self-hosted option, so it wouldn't be putting in 'the cloud' so much as a local server at your company.

Edited December 3, 2017 by smithd

[viSci]

Has anyone looked at NI's new Systemlink / Skyline technology?  It looks like a type of DDS running on RabbitMQ that is being used for all future LV software and package deployment for Windows and Linux platforms.  Skyline 17.5 has Publish Subscribe tag's, tag viewer and a new web dashboard that can bind to tags.  I was informed that in Q2 Skyline will have historical capabilities but am not sure if it could serve as a citadel replacement.

[smithd]

On 12/19/2017 at 0:06 PM, viSci said:

Has anyone looked at NI's new Systemlink / Skyline technology?  It looks like a type of DDS running on RabbitMQ that is being used for all future LV software and package deployment for Windows and Linux platforms.  Skyline 17.5 has Publish Subscribe tag's, tag viewer and a new web dashboard that can bind to tags.  I was informed that in Q2 Skyline will have historical capabilities but am not sure if it could serve as a citadel replacement.

I'm familiar with it. My reaction at the time was that it seemed like another black box doing tag based communication via a background service to replace the shared variable black box doing tag based communication via a background service. I'd love for the new black box to be better than the old, but I'm skeptical.

It definitely appears to resolve some issues with shared variables (I believe they can be created programmatically), but its not clear how you might integrate it with non-lv applications, time will tell on performance, and time will really tell on usability, reliability, etc that variables are often questioned on (a straightforward example is, how do I find out that my peer is dead/disconnected? how long does it take to be detected as dead? etc.). Its also not clear why we'd want to use a custom proprietary tag protocol when NI is also investing in OPC UA.

[viSci]

Have been looking at the OPC UA toolkit, it looks pretty cool and as you pointed out has compatibility with a huge selection of 3rd party tools and utilities.  It seems like the licensing is free for Linux cRIO's so would just need to purchase the $500 dev toolkit.  OPC UA also has a full selection of scaler and array data types and a pretty nice LV API.  I was wondering if anyone knows about the OPC UA History Data vi's.  It seems like a type of memory resident time series historian but was wondering if it uses any sort of database under the hood.

[smithd]

The poor AE they had staffing the booth at NI week said it was in memory. I would normally take that with a grain of salt, but when I was evaluating it I don't remember seeing a path input for where to store any kind of database or file, so if it is persistent its a mystery to me. In memory would also make the most sense to me since it asks for a number of elements for each thing you want to store, so it probably just creates a fixed-size buffer. Not all that useful for a long term historical DB. Thats why in my post above I was suggesting some sort of 3rd-party tool for a more permanent storage mechanism.

For my use (I haven't written it yet, but I have a near-term plan), I would only use the historical information for sort of that short term use -- so if I have multiple HMIs, they don't all need to buffer values themselves and if they restart the HMI it has a centralized cache to pull from. Based on the API, it looks like it should be feasible to write the entire historical content to disk right before you shut down the server instance and then reload it when you restart the server instance. Of course, that depends on a clean shutdown...

Edited January 14, 2018 by smithd