Does it make sense to use REDIS instead of shared variables

[infinitenothing]

I feel like REDIS accomplishes most of the goals that shared variables attempt to cover (minus some UI sugar) in a nice cross platform way. Does anyone have an opinion on "in memory" key-value stores that are accessible over a network?

[ensegre]

For those ignorant as me, https://redis.io/ .
Given the shame that shared network variables are, I would tend to say that any other solution, no matter how much of a contraption, might be attractive. I'm curious to read opinions too, as well as ideas for a LV API.

[DTaylor]

On 4/28/2023 at 1:28 PM, infinitenothing said:

Does anyone have an opinion on "in memory" key-value stores that are accessible over a network?

I do. REDIS is great. It's much nicer to work with than shared variables or OPC. The downsides I can think of are:

1) You need a Linux server or Windows new enough to have WSL2 to run the server. There is no official Windows build

2) The protocol is simple, but the API is pretty big. I don't know if anyone has created a publicly available LabVIEW client. You can put together a client library with only the most common commands pretty quickly though.

[ensegre]

Reviving this thread. I'm looking for a distributed PV solution for a setup of some tens of linux PCs, each one writing some ten of tags at a rate of a few per sec, where the writing will mostly be done by Matlab bindings, and the supervisory/logging/alerting whatnot by clients written in a variety of languages not excluding LV. OSS is not strictly mandatory but essentially part of the culture.

I'd would be looking at REDIS, EPICS and Tango-controls (with its annexes Sardana, Taurus) in the first place, but I haven't yet dwelled into them order to compare own merits. In fact I had a project where I interfaced with Tango some years ago, and I contributed cleaning up the official set of LV bindings then. As for EPICS, linux excludes the usual Network Shared Variables stuff (or the EPICS i/o module), but I found for example CALab which looks on spot. Matlab bindings seem available for the three. The ability of handling structured data vs. just double or logical PV may be a discriminant, if one solution is particularly limited in that respect.

Has anyone recommendations? Is anyone aware of toolkits I could leverage onto?

[ShaunR]

13 hours ago, ensegre said:

Reviving this thread. I'm looking for a distributed PV solution for a setup of some tens of linux PCs, each one writing some ten of tags at a rate of a few per sec, where the writing will mostly be done by Matlab bindings, and the supervisory/logging/alerting whatnot by clients written in a variety of languages not excluding LV. OSS is not strictly mandatory but essentially part of the culture.

I'd would be looking at REDIS, EPICS and Tango-controls (with its annexes Sardana, Taurus) in the first place, but I haven't yet dwelled into them order to compare own merits. In fact I had a project where I interfaced with Tango some years ago, and I contributed cleaning up the official set of LV bindings then. As for EPICS, linux excludes the usual Network Shared Variables stuff (or the EPICS i/o module), but I found for example CALab which looks on spot. Matlab bindings seem available for the three. The ability of handling structured data vs. just double or logical PV may be a discriminant, if one solution is particularly limited in that respect.

Has anyone recommendations? Is anyone aware of toolkits I could leverage onto?

My tuppence is that anything is better than Network Shared Variables. My first advice is choose one or two, not a jenga tower of many.

What's the use-case here? Is the data real-time (you know what I mean) over the network? Are the devices dependent on data from another device or is the data accumulated for exploitation later? If it's the latter, I would go with SQLite locally (for integrity and reliability) and periodic merges to MySQL [Maria] remotely (for exploitation). Both of those technologies have well established API's in almost all languages.

Edited July 16 by ShaunR

[ensegre]

well, NSV are out of cause here first because it's a linux distributed system, second because of their own proven merits 😆...

The background is this, BTW. We have 17 PCs up and running as of now, expected to grow to 40ish. The main business logic, involving the production of control process variables, is done by tens of Matlab processes, for a variety of reasons. The whole system is a huge data producer (we're talking of TBs per night), but data is well handled by other pipelines. What I'm concerned with here is monitoring/supervision/alerting/remediation. Realtiming is not strict, latencies of the order of seconds could even be tolerated. Logging is a feature of any SCADA, but it's not  the main or only goal here; this is why I'd be happy with a side Tango or whatever module dumping to a historical database, but I would not look in the first place into a model "first dump all to local SQLs, then reread them and merge and ponder about the data". I'd think that local, in-memory PV stores, local first level remediation clients, and centralized system health monitoring is the way to go.

As for the jenga tower, the mixup of data producers is life, but it is not that EPICS or Tango come without a proven reliability pedigree! And of course I'd chose only one ecosystem, I'm at the stage of choosing which.

ETA: as for redis I ran into this. Any experience?

Edited July 16 by ensegre

[ShaunR]

3 hours ago, ensegre said:

well, NSV are out of cause here first because it's a linux distributed system, second because of their own proven merits 😆...

The background is this, BTW. We have 17 PCs up and running as of now, expected to grow to 40ish. The main business logic, involving the production of control process variables, is done by tens of Matlab processes, for a variety of reasons. The whole system is a huge data producer (we're talking of TBs per night), but data is well handled by other pipelines. What I'm concerned with here is monitoring/supervision/alerting/remediation. Realtiming is not strict, latencies of the order of seconds could even be tolerated. Logging is a feature of any SCADA, but it's not  the main or only goal here; this is why I'd be happy with a side Tango or whatever module dumping to a historical database, but I would not look in the first place into a model "first dump all to local SQLs, then reread them and merge and ponder about the data". I'd think that local, in-memory PV stores, local first level remediation clients, and centralized system health monitoring is the way to go.

As for the jenga tower, the mixup of data producers is life, but it is not that EPICS or Tango come without a proven reliability pedigree! And of course I'd chose only one ecosystem, I'm at the stage of choosing which.

ETA: as for redis I ran into this. Any experience?

I've no experience with Redis.

For monitoring/supervision/alerting, you don't need local storage at all. MQTT might be worth looking at for that.

Sorry I couldn't be more helpful but looks like a cool project.

[ensegre]

Coming back to report.

Scavenging the net I've found essentially three set of connectors between Redis and LV:

1. what can be downloaded from https://forums.ni.com/t5/Example-Code/REDIS-database-LabVIEW-toolkit/tac-p/3508611 taking into account the corrections listed in the thread. This seems to be the more widespread, considering even that it was shown as an option at the CERN LV user group this year (see https://indico.cern.ch/event/1388470/contributions/5911487/attachments/2843544/4971934/lugm_LabVIEW_at_CERN.pdf, slide 22). Dates originally to 2014.
2. Nick Folse's https://github.com/tauterra/Redis-Client-for-LabVIEW of about three years ago, according to its author no further developed. Found a couple of flaws, easily corrected.
3. https://github.com/Bas-vE/LV-Redis , which  claims to be an evolution of 1., promoted to LVOOP. Most recent of the three.

The philosophy of the three toolboxes differs somewhat from one to the other, the first one being more of the kind "one VI for each Redis command", the others putting perhaps more the accent on the transaction protocol than on the completeness of the commands implemented. Redis's huge command set also expanded during the years in question. However, I found in all three something which looks to me a bit of a no brainer, which is that TCP client connection are opened and then closed for each elementary operation. While that might have a minor performance impact, I found that the approach prevents Redis' MULTI pipelining.

I have forked 1. in https://github.com/EastEriq/redis-in-labview and 2. in https://github.com/EastEriq/Redis-Client-for-LabVIEW for dwelving into. Finally, I have resolved for adopting my fork and augmentation of 1. in my project, but only after I modified it so that TCP connections can be kept open throughout the client sessions.

Edited September 30 by ensegre

[Antoine Chalons]

I fully agree that anything, even smoke signals in the wind, are better that NSVs.

I'm using Redis, I also started from an existing lib found on the interweb, tweaked a bit for my needs.

We now have multiple golang services as well as LabVIEW services connecting to Redis for cache and using RabbitMq for messaging.

For some small tests on my Windows machine I've used Redis on WSL, easy to use.

At the next local user group meeting I might do a short presentation about it, if you have faced any issues, I'd be interested in your feedback.

Cheers

Edited October 1 by Antoine Chalons
fix typos