Does it make sense to use REDIS instead of shared variables

[infinitenothing]

I feel like REDIS accomplishes most of the goals that shared variables attempt to cover (minus some UI sugar) in a nice cross platform way. Does anyone have an opinion on "in memory" key-value stores that are accessible over a network?

[ensegre]

For those ignorant as me, https://redis.io/ .
Given the shame that shared network variables are, I would tend to say that any other solution, no matter how much of a contraption, might be attractive. I'm curious to read opinions too, as well as ideas for a LV API.

[DTaylor]

On 4/28/2023 at 1:28 PM, infinitenothing said:

Does anyone have an opinion on "in memory" key-value stores that are accessible over a network?

I do. REDIS is great. It's much nicer to work with than shared variables or OPC. The downsides I can think of are:

1) You need a Linux server or Windows new enough to have WSL2 to run the server. There is no official Windows build

2) The protocol is simple, but the API is pretty big. I don't know if anyone has created a publicly available LabVIEW client. You can put together a client library with only the most common commands pretty quickly though.

[ensegre]

Reviving this thread. I'm looking for a distributed PV solution for a setup of some tens of linux PCs, each one writing some ten of tags at a rate of a few per sec, where the writing will mostly be done by Matlab bindings, and the supervisory/logging/alerting whatnot by clients written in a variety of languages not excluding LV. OSS is not strictly mandatory but essentially part of the culture.

I'd would be looking at REDIS, EPICS and Tango-controls (with its annexes Sardana, Taurus) in the first place, but I haven't yet dwelled into them order to compare own merits. In fact I had a project where I interfaced with Tango some years ago, and I contributed cleaning up the official set of LV bindings then. As for EPICS, linux excludes the usual Network Shared Variables stuff (or the EPICS i/o module), but I found for example CALab which looks on spot. Matlab bindings seem available for the three. The ability of handling structured data vs. just double or logical PV may be a discriminant, if one solution is particularly limited in that respect.

Has anyone recommendations? Is anyone aware of toolkits I could leverage onto?

[ShaunR]

13 hours ago, ensegre said:

Reviving this thread. I'm looking for a distributed PV solution for a setup of some tens of linux PCs, each one writing some ten of tags at a rate of a few per sec, where the writing will mostly be done by Matlab bindings, and the supervisory/logging/alerting whatnot by clients written in a variety of languages not excluding LV. OSS is not strictly mandatory but essentially part of the culture.

I'd would be looking at REDIS, EPICS and Tango-controls (with its annexes Sardana, Taurus) in the first place, but I haven't yet dwelled into them order to compare own merits. In fact I had a project where I interfaced with Tango some years ago, and I contributed cleaning up the official set of LV bindings then. As for EPICS, linux excludes the usual Network Shared Variables stuff (or the EPICS i/o module), but I found for example CALab which looks on spot. Matlab bindings seem available for the three. The ability of handling structured data vs. just double or logical PV may be a discriminant, if one solution is particularly limited in that respect.

Has anyone recommendations? Is anyone aware of toolkits I could leverage onto?

My tuppence is that anything is better than Network Shared Variables. My first advice is choose one or two, not a jenga tower of many.

What's the use-case here? Is the data real-time (you know what I mean) over the network? Are the devices dependent on data from another device or is the data accumulated for exploitation later? If it's the latter, I would go with SQLite locally (for integrity and reliability) and periodic merges to MySQL [Maria] remotely (for exploitation). Both of those technologies have well established API's in almost all languages.

Edited July 16 by ShaunR

[ensegre]

well, NSV are out of cause here first because it's a linux distributed system, second because of their own proven merits 😆...

The background is this, BTW. We have 17 PCs up and running as of now, expected to grow to 40ish. The main business logic, involving the production of control process variables, is done by tens of Matlab processes, for a variety of reasons. The whole system is a huge data producer (we're talking of TBs per night), but data is well handled by other pipelines. What I'm concerned with here is monitoring/supervision/alerting/remediation. Realtiming is not strict, latencies of the order of seconds could even be tolerated. Logging is a feature of any SCADA, but it's not  the main or only goal here; this is why I'd be happy with a side Tango or whatever module dumping to a historical database, but I would not look in the first place into a model "first dump all to local SQLs, then reread them and merge and ponder about the data". I'd think that local, in-memory PV stores, local first level remediation clients, and centralized system health monitoring is the way to go.

As for the jenga tower, the mixup of data producers is life, but it is not that EPICS or Tango come without a proven reliability pedigree! And of course I'd chose only one ecosystem, I'm at the stage of choosing which.

ETA: as for redis I ran into this. Any experience?

Edited July 16 by ensegre

[ShaunR]

3 hours ago, ensegre said:

well, NSV are out of cause here first because it's a linux distributed system, second because of their own proven merits 😆...

The background is this, BTW. We have 17 PCs up and running as of now, expected to grow to 40ish. The main business logic, involving the production of control process variables, is done by tens of Matlab processes, for a variety of reasons. The whole system is a huge data producer (we're talking of TBs per night), but data is well handled by other pipelines. What I'm concerned with here is monitoring/supervision/alerting/remediation. Realtiming is not strict, latencies of the order of seconds could even be tolerated. Logging is a feature of any SCADA, but it's not  the main or only goal here; this is why I'd be happy with a side Tango or whatever module dumping to a historical database, but I would not look in the first place into a model "first dump all to local SQLs, then reread them and merge and ponder about the data". I'd think that local, in-memory PV stores, local first level remediation clients, and centralized system health monitoring is the way to go.

As for the jenga tower, the mixup of data producers is life, but it is not that EPICS or Tango come without a proven reliability pedigree! And of course I'd chose only one ecosystem, I'm at the stage of choosing which.

ETA: as for redis I ran into this. Any experience?

I've no experience with Redis.

For monitoring/supervision/alerting, you don't need local storage at all. MQTT might be worth looking at for that.

Sorry I couldn't be more helpful but looks like a cool project.