What are some choices for producer-consumer protocols?

[Reds]

I have a LabVIEW application that produces a continuous stream of binary measurement data. Right now, this measurement data streams into a LabVIEW notifier, and multiple "listeners" can "subscribe" to that data and monitor what is happening. Both the producer (publisher) and consumers (subscribers) are all native to LabVIEW right now, so it  all works great with LabVIEW notifiers.

But, now I want to have a non-LabVIEW listener. What to do?

The semi-obvious choice would be to switch the code so that the data would be streamed on TCP/IP instead. But what higher-layer protocol would you want to use for that? It seems to make like the RESTFul and gRPC models are more client-server oriented. I want something this a very loosely couple producer-consumer model, where you have one producer and any number of listeners.

What are some good candidates?

[ensegre]

Redis is certainly high performance and suited to multiple, loose writers, readers and subscribers, with bindings for so many ecosystems. One of its several features, which I haven't perused, are Streams. I'd be curious too to know whether continuous cross-app data streaming could be efficiently implemented using them.

Edited Wednesday at 08:15 AM by ensegre

[Antoine Chalons]

I would suggest rabbitmq, i want(ed) to present it at a LabVIEW user group (LUGE) but haven't done it yet.

It's very powerful.

I use redis and did a quick presentation (in french) at LUGE recently, i haven't used the stream feature though, I only used it as cache.

 

[ShaunR]

QUIC

Edited 21 hours ago by ShaunR

[Reds]

3 hours ago, ShaunR said:

QUIC

In theory maybe....

But how would I get LabVIEW to create QUIC connections?

[ShaunR]

14 hours ago, Reds said:

In theory maybe....

But how would I get LabVIEW to create QUIC connections?

Tough one for LabVIEW. It's a bit like RTSP - there's nothing until someone writes it. 

OpenSSL has QUIC support. M$ have .NET support and MsQuic.

RabbitMQ is discussing whether they should or not. I will support it later this year (waiting for OpenSSL version 3.6).

I have some LabVIEW prototypes that I've played with and it's damned fast with almost no overhead. It has in-built failover and multiplexed channels making it ideal for your use case. 

Your idea of streaming directly is a good one, though. You don't really need a higher protocol unless you can identify a particular problem you need to overcome. Websockets is another that would be a good start because you would be able to have an Apache/Nginx server for routing or even send the data directly to browsers.

Hmmm. Yes. Considering where you are now I would recommend Websockets. They are probably the easiest and get you almost all the way to where you want to be and a good start for expansion later for streaming in other more efficient and scalable protocols like QUIC. I think there are a few LabVIEW Websocket implementations to choose from so you wouldn't have to write your own.

[ShaunR]

As an addendum...

I did something very similar to you many moons ago. Maybe it will give you some ideas. Here is "Dispatcher". Should work fine for you as long as you don't select TLS. Just open the project and run the file Demo - Run ME.vi (then press Register and Subscribe buttons). It also does stuff like compression and blowfish encryption. It's a simple custom, home-grown,  protocol that I wrote to explore high-speed data streaming before the more modern protocols were available. The Dispatcher, Publishers and Subscribers can run on any machine that has network access-they don't all have to be local to each other. There is an API that can be used to implement it in applications.

It's not really a product so not much in the way of documentation, I'm afraid.

Dispatcher.zip

Edited 1 hour ago by ShaunR