zeromq - simple fast lightweight communication - does it fill a gap in NIs offering?

[Jordan Kuehn]

On 11/27/2013 at 10:24 AM, M_P said:

I'm particularly looking into this (I'm also known as MattP on the NI forums, I work in the SE group like GoGators). From my perspective, it fills a gap that was partially filled by the AMC library (using same API for inter-loop and inter-node communication), network streams (automatic reconnect in background abstracted from the read/write, good performance), and shared variables (the pub-sub use case).  

 

What it handles better, IMHO, is that it handles not just the reconnect, but also the initial connect and intermediate send/receive queues entirely asynchronously.  It also easily handles N:1 or 1:N using several different transports.  It's not so much the base set of patterns that are useful, but the combinations of them that perrmit complex problems to be solved with very small quantities of efficient code.  Me likey.

 

Has there been any recent movement on this? Currently from NI I see a clunky method of flushing streams suggested for native messaging/command control, and the STM library and I don't really see anything that natively addresses the issues mentioned here.

[smithd]

I'm confused by this question -- there is an available zeromq library if that fits your needs. Its not perfect but its good, and its up to version 4.2.5 (current main lib release is 4.3.2).

If you don't need linuxrt support you can just use https://github.com/zeromq/netmq

[Jordan Kuehn]

19 hours ago, smithd said:

I'm confused by this question -- there is an available zeromq library if that fits your needs. Its not perfect but its good, and its up to version 4.2.5 (current main lib release is 4.3.2).

If you don't need linuxrt support you can just use https://github.com/zeromq/netmq

As I read the OP the question was does zeromq fill this gap in native NI offerings? I quoted a mention of some of the holes that exist and suggest that I think they still exist. Perhaps I should add some context. I remember when this conversation first popped up as well as rabbitmq. I thought that it was just an effort to include a third-party communication protocol to make rt targets able to communicate with 3rd party devices and I don't really have *that* need. However, filling some of the holes in NI offerings is something I am interested in seeing if this is the case. I work with cRIOs daily and have a working toolset, but I was curious to see where this (or other) protocols sit today in terms of the original question. If I misunderstood the OP, my apologies, and I do realize this is an old thread, but that isn't always frowned on here.

[MarkCG]

On 12/26/2019 at 10:29 AM, smithd said:

I'm confused by this question -- there is an available zeromq library if that fits your needs. Its not perfect but its good, and its up to version 4.2.5 (current main lib release is 4.3.2).

If you don't need linuxrt support you can just use https://github.com/zeromq/netmq

we got it to work on cRIO

https://sourceforge.net/p/labview-zmq/discussion/general/thread/87780372ed/

[smithd]

41 minutes ago, MarkCG said:

we got it to work on cRIO

https://sourceforge.net/p/labview-zmq/discussion/general/thread/87780372ed/

oh yeah i meant you just have to download for windows. I know its pretty easy to compile for lrt, but you have to do the compile.

[smithd]

On 12/27/2019 at 9:02 AM, Jordan Kuehn said:

As I read the OP the question was does zeromq fill this gap in native NI offerings? I quoted a mention of some of the holes that exist and suggest that I think they still exist. Perhaps I should add some context. I remember when this conversation first popped up as well as rabbitmq. I thought that it was just an effort to include a third-party communication protocol to make rt targets able to communicate with 3rd party devices and I don't really have *that* need. However, filling some of the holes in NI offerings is something I am interested in seeing if this is the case. I work with cRIOs daily and have a working toolset, but I was curious to see where this (or other) protocols sit today in terms of the original question. If I misunderstood the OP, my apologies, and I do realize this is an old thread, but that isn't always frowned on here.

I see, yeah I don't know of any movement on the ni side to improve data comms. Rabbit is 'available' via systemlink, but...requires systemlink. Its also just about 2020 and we don't have any secure network api built into the product.

[Jordan Kuehn]

On 12/28/2019 at 11:50 AM, smithd said:

Its also just about 2020 and we don't have any secure network api built into the product.

That's also a very good point.

[viSci]

It's not exactly native but the RTI DDS api has been included in the past few LabVIEW releases.

Edited December 30, 2019 by viSci

[viSci]

 

On ‎12‎/‎28‎/‎2019 at 12:50 PM, smithd said:

I see, yeah I don't know of any movement on the ni side to improve data comms. Rabbit is 'available' via systemlink, but...requires systemlink. Its also just about 2020 and we don't have any secure network api built into the product.

It is disheartening for those of us who develop for the cRIO to see NI come out with a cool product like systemlink but in a misguided marketing strategy have totally ignored those of us toughing it out in the trenches trying to develop secure and robust DDS capability for the cRIO.  Even basic stuff like cRIO image management and software deployment, which systemlink handles nicely have been bundeled exclusively in the prohibitively expensive package that NI is trying to sell to electrical utilities. 

[Jordan Kuehn]

29 minutes ago, viSci said:

It's not exactly native but the RTI DDS api has been included in the past few LabVIEW releases.

I've seen this mentioned a few times and it didn't make it onto my radar until recently. Have you had occasion to use it much in LabVIEW?

[viSci]

Yes I have been using it quite a bit on a roadway monitoring project with 50 cRIO's.  I use it to handle all of the cRIO system state and tag data publishing.  We are still in the early stages of testing but it seems to be working very well.  The LabVIEW version of RTI DDS is a subset of its full capability.  RTI has been slowly adding features but it still does not support basic things like events.  Judging from the forum posts, the toolkit is largely unknown in the LV community.  I think if more people adopted it, it would garner more love and attention by RTI.

[MarkCG]

10 hours ago, viSci said:

Yes I have been using it quite a bit on a roadway monitoring project with 50 cRIO's.  I use it to handle all of the cRIO system state and tag data publishing.  We are still in the early stages of testing but it seems to be working very well.  The LabVIEW version of RTI DDS is a subset of its full capability.  RTI has been slowly adding features but it still does not support basic things like events.  Judging from the forum posts, the toolkit is largely unknown in the LV community.  I think if more people adopted it, it would garner more love and attention by RTI.

what kind of data rates and number of tags can it handle?

[viSci]

https://www.rti.com/products/benchmarks

[MarkCG]

would you say the system is easy to set up or does it basically require someone dedicated to running it full time?

[viSci]

Pretty easy to setup and keep running.   If you give me more specifics on your application I could provide more relevant information...

[ShaunR]

Is anybody using COAP?

Edited January 1, 2020 by ShaunR

[smithd]

I've looked at it in the past. Seems weird to target devices with performance such that full http is too expensive to parse but with sufficient performance for dtls over udp and for parsing/returning json and xml strings. I just ended up using full http.

[ShaunR]

3 hours ago, smithd said:

I've looked at it in the past. Seems weird to target devices with performance such that full http is too expensive to parse but with sufficient performance for dtls over udp and for parsing/returning json and xml strings. I just ended up using full http.

Indeed. It does have some useful features though like congestion control targeted at small packets and discovery.

Seems to be more performant too.

Edited January 1, 2020 by ShaunR

[smithd]

On 1/1/2020 at 1:36 PM, ShaunR said:

Indeed. It does have some useful features though like congestion control targeted at small packets and discovery.

Seems to be more performant too.

Its important not to miss the details on that performance one. They went for the specific use case of very very underpowered devices, very infrequent sending of data, etc. For example they assume a new TCP connection for every data packet, unless I misread*, while on the DTLS/coap side they ignore security handshaking, assuming you have a factory installed pre-shared key instead. It looks like it also ignores the fact that CoAP uses a rest model, meaning a request-response cycle. If you include that information they are basically saying "UDP protocols with no reliability except a super super barebones re-transmit feature work better than TCP if you close and reopen the connection once per second"..which...yeah.

 

 

*

Quote

For TCP, the session is terminated after each sensor report was received in the other end of the connection

 

Edited January 3, 2020 by smithd

[ShaunR]

5 hours ago, smithd said:

Its important not to miss the details on that performance one. They went for the specific use case of very very underpowered devices, very infrequent sending of data, etc. For example they assume a new TCP connection for every data packet, unless I misread*, while on the DTLS/coap side they ignore security handshaking, assuming you have a factory installed pre-shared key instead. It looks like it also ignores the fact that CoAP uses a rest model, meaning a request-response cycle. If you include that information they are basically saying "UDP protocols with no reliability except a super super barebones re-transmit feature work better than TCP if you close and reopen the connection once per second"..which...yeah.

That was my takeaway too (udp vs tcp). However. I've been working with LoRa recently (they just launched a couple of satellites) so my choices were MQTT or COAP to run on ESP8266 and ESP32s. I'm considering COAP mainly for the discovery aspect (which MQTT doesn't have but you can kind of make your own by having a topic to annouce to). The UDP aspect might come into its own if they enable UDP Multicast over the internet as standard though.

[Jordan Kuehn]

3 hours ago, ShaunR said:

That was my takeaway too (udp vs tcp). However. I've been working with LoRa recently (they just launched a couple of satellites) so my choices were MQTT or COAP to run on ESP8266 and ESP32s. I'm considering COAP mainly for the discovery aspect (which MQTT doesn't have but you can kind of make your own by having a topic to annouce to). The UDP aspect might come into its own if they enable UDP Multicast over the internet as standard though.

The auto discovery aspect is one appealing thing about the DDS package viSci was mentioning. However, from what I'm looking at I may still need to create some sort of announce topic like you mentioned using in MQTT to know what topics/keys are available. Does COAP provide a clean way of providing a list of all available pubs/topics/etc.? I haven't had a chance to look into that a lot yet. Also, are there LabVIEW toolkits for it? There are several MQTT toolkits, the shipping RTI DDS toolkit, and then back to the OP some 0mq and labbitmq available (not sure these all belong in the same list, but they are all in this thread).

[ShaunR]

3 hours ago, Jordan Kuehn said:

Does COAP provide a clean way of providing a list of all available pubs/topics/etc.?

Yes. It's part of the protocol and has service and resource discovery.

3 hours ago, Jordan Kuehn said:

Also, are there LabVIEW toolkits for it?

Nope. 

Edited January 3, 2020 by ShaunR

[Jordan Kuehn]

Perhaps beginning to diverge from OP, but I've been spending some time with the RTI DDS toolkit. It does seem useful, but it also seems like the LabVIEW integration needs some improvement. Perhaps I'm not past the learning curve peak yet though. ViSci are you using it as more of a tag engine or are you using it for streaming and/or commands as well? It seems like implementation as a tag engine could get cumbersome on the BD publishing individual tags. Clusters work after using their tool, but that also seems cumbersome.