Jump to content

zeromq - simple fast lightweight communication - does it fill a gap in NIs offering?


Recommended Posts

On 11/27/2013 at 10:24 AM, M_P said:

I'm particularly looking into this (I'm also known as MattP on the NI forums, I work in the SE group like GoGators). From my perspective, it fills a gap that was partially filled by the AMC library (using same API for inter-loop and inter-node communication), network streams (automatic reconnect in background abstracted from the read/write, good performance), and shared variables (the pub-sub use case).  

 

What it handles better, IMHO, is that it handles not just the reconnect, but also the initial connect and intermediate send/receive queues entirely asynchronously.  It also easily handles N:1 or 1:N using several different transports.  It's not so much the base set of patterns that are useful, but the combinations of them that perrmit complex problems to be solved with very small quantities of efficient code.  Me likey.

 

Has there been any recent movement on this? Currently from NI I see a clunky method of flushing streams suggested for native messaging/command control, and the STM library and I don't really see anything that natively addresses the issues mentioned here.

Link to post
Share on other sites
  • Replies 72
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Been there, done that.  I have G implementations of ZMTP (at least most bits I need) and AMQP (0.9.1 not the 1.0 trainwreck).  I tend to use AMQP most often since my code is more complete (and there w

Just came across this port.  Sys Eng at NI is interested in zeromq and rabbitmq.  We've been playing around with both and are considered making an open source library if it fits a need.  One of the SE

I'm particularly looking into this (I'm also known as MattP on the NI forums, I work in the SE group like GoGators). From my perspective, it fills a gap that was partially filled by the AMC library (u

19 hours ago, smithd said:

I'm confused by this question -- there is an available zeromq library if that fits your needs. Its not perfect but its good, and its up to version 4.2.5 (current main lib release is 4.3.2).

If you don't need linuxrt support you can just use https://github.com/zeromq/netmq

As I read the OP the question was does zeromq fill this gap in native NI offerings? I quoted a mention of some of the holes that exist and suggest that I think they still exist. Perhaps I should add some context. I remember when this conversation first popped up as well as rabbitmq. I thought that it was just an effort to include a third-party communication protocol to make rt targets able to communicate with 3rd party devices and I don't really have *that* need. However, filling some of the holes in NI offerings is something I am interested in seeing if this is the case. I work with cRIOs daily and have a working toolset, but I was curious to see where this (or other) protocols sit today in terms of the original question. If I misunderstood the OP, my apologies, and I do realize this is an old thread, but that isn't always frowned on here.

Link to post
Share on other sites
On 12/26/2019 at 10:29 AM, smithd said:

I'm confused by this question -- there is an available zeromq library if that fits your needs. Its not perfect but its good, and its up to version 4.2.5 (current main lib release is 4.3.2).

If you don't need linuxrt support you can just use https://github.com/zeromq/netmq

we got it to work on cRIO

https://sourceforge.net/p/labview-zmq/discussion/general/thread/87780372ed/

Link to post
Share on other sites
On 12/27/2019 at 9:02 AM, Jordan Kuehn said:

As I read the OP the question was does zeromq fill this gap in native NI offerings? I quoted a mention of some of the holes that exist and suggest that I think they still exist. Perhaps I should add some context. I remember when this conversation first popped up as well as rabbitmq. I thought that it was just an effort to include a third-party communication protocol to make rt targets able to communicate with 3rd party devices and I don't really have *that* need. However, filling some of the holes in NI offerings is something I am interested in seeing if this is the case. I work with cRIOs daily and have a working toolset, but I was curious to see where this (or other) protocols sit today in terms of the original question. If I misunderstood the OP, my apologies, and I do realize this is an old thread, but that isn't always frowned on here.

I see, yeah I don't know of any movement on the ni side to improve data comms. Rabbit is 'available' via systemlink, but...requires systemlink. Its also just about 2020 and we don't have any secure network api built into the product.

Link to post
Share on other sites

 

On ‎12‎/‎28‎/‎2019 at 12:50 PM, smithd said:

I see, yeah I don't know of any movement on the ni side to improve data comms. Rabbit is 'available' via systemlink, but...requires systemlink. Its also just about 2020 and we don't have any secure network api built into the product.

It is disheartening for those of us who develop for the cRIO to see NI come out with a cool product like systemlink but in a misguided marketing strategy have totally ignored those of us toughing it out in the trenches trying to develop secure and robust DDS capability for the cRIO.  Even basic stuff like cRIO image management and software deployment, which systemlink handles nicely have been bundeled exclusively in the prohibitively expensive package that NI is trying to sell to electrical utilities. 

Link to post
Share on other sites
29 minutes ago, viSci said:

It's not exactly native but the RTI DDS api has been included in the past few LabVIEW releases.

I've seen this mentioned a few times and it didn't make it onto my radar until recently. Have you had occasion to use it much in LabVIEW?

Link to post
Share on other sites

Yes I have been using it quite a bit on a roadway monitoring project with 50 cRIO's.  I use it to handle all of the cRIO system state and tag data publishing.  We are still in the early stages of testing but it seems to be working very well.  The LabVIEW version of RTI DDS is a subset of its full capability.  RTI has been slowly adding features but it still does not support basic things like events.  Judging from the forum posts, the toolkit is largely unknown in the LV community.  I think if more people adopted it, it would garner more love and attention by RTI.

Link to post
Share on other sites
10 hours ago, viSci said:

Yes I have been using it quite a bit on a roadway monitoring project with 50 cRIO's.  I use it to handle all of the cRIO system state and tag data publishing.  We are still in the early stages of testing but it seems to be working very well.  The LabVIEW version of RTI DDS is a subset of its full capability.  RTI has been slowly adding features but it still does not support basic things like events.  Judging from the forum posts, the toolkit is largely unknown in the LV community.  I think if more people adopted it, it would garner more love and attention by RTI.

what kind of data rates and number of tags can it handle?

Link to post
Share on other sites

I've looked at it in the past. Seems weird to target devices with performance such that full http is too expensive to parse but with sufficient performance for dtls over udp and for parsing/returning json and xml strings. I just ended up using full http.

Link to post
Share on other sites
3 hours ago, smithd said:

I've looked at it in the past. Seems weird to target devices with performance such that full http is too expensive to parse but with sufficient performance for dtls over udp and for parsing/returning json and xml strings. I just ended up using full http.

Indeed. It does have some useful features though like congestion control targeted at small packets and discovery.

Seems to be more performant too.

Edited by ShaunR
Link to post
Share on other sites
On 1/1/2020 at 1:36 PM, ShaunR said:

Indeed. It does have some useful features though like congestion control targeted at small packets and discovery.

Seems to be more performant too.

Its important not to miss the details on that performance one. They went for the specific use case of very very underpowered devices, very infrequent sending of data, etc. For example they assume a new TCP connection for every data packet, unless I misread*, while on the DTLS/coap side they ignore security handshaking, assuming you have a factory installed pre-shared key instead. It looks like it also ignores the fact that CoAP uses a rest model, meaning a request-response cycle. If you include that information they are basically saying "UDP protocols with no reliability except a super super barebones re-transmit feature work better than TCP if you close and reopen the connection once per second"..which...yeah.

 

 

*

Quote

For TCP, the session is terminated after each sensor report was received in the other end of the connection

 

Edited by smithd
Link to post
Share on other sites
5 hours ago, smithd said:

Its important not to miss the details on that performance one. They went for the specific use case of very very underpowered devices, very infrequent sending of data, etc. For example they assume a new TCP connection for every data packet, unless I misread*, while on the DTLS/coap side they ignore security handshaking, assuming you have a factory installed pre-shared key instead. It looks like it also ignores the fact that CoAP uses a rest model, meaning a request-response cycle. If you include that information they are basically saying "UDP protocols with no reliability except a super super barebones re-transmit feature work better than TCP if you close and reopen the connection once per second"..which...yeah.

That was my takeaway too (udp vs tcp). However. I've been working with LoRa recently (they just launched a couple of satellites) so my choices were MQTT or COAP to run on ESP8266 and ESP32s. I'm considering COAP mainly for the discovery aspect (which MQTT doesn't have but you can kind of make your own by having a topic to annouce to). The UDP aspect might come into its own if they enable UDP Multicast over the internet as standard though.

Link to post
Share on other sites
3 hours ago, ShaunR said:

That was my takeaway too (udp vs tcp). However. I've been working with LoRa recently (they just launched a couple of satellites) so my choices were MQTT or COAP to run on ESP8266 and ESP32s. I'm considering COAP mainly for the discovery aspect (which MQTT doesn't have but you can kind of make your own by having a topic to annouce to). The UDP aspect might come into its own if they enable UDP Multicast over the internet as standard though.

The auto discovery aspect is one appealing thing about the DDS package viSci was mentioning. However, from what I'm looking at I may still need to create some sort of announce topic like you mentioned using in MQTT to know what topics/keys are available. Does COAP provide a clean way of providing a list of all available pubs/topics/etc.? I haven't had a chance to look into that a lot yet. Also, are there LabVIEW toolkits for it? There are several MQTT toolkits, the shipping RTI DDS toolkit, and then back to the OP some 0mq and labbitmq available (not sure these all belong in the same list, but they are all in this thread).

Link to post
Share on other sites
3 hours ago, Jordan Kuehn said:

Does COAP provide a clean way of providing a list of all available pubs/topics/etc.?

Yes. It's part of the protocol and has service and resource discovery.

3 hours ago, Jordan Kuehn said:

Also, are there LabVIEW toolkits for it?

Nope. 

Edited by ShaunR
Link to post
Share on other sites

Perhaps beginning to diverge from OP, but I've been spending some time with the RTI DDS toolkit. It does seem useful, but it also seems like the LabVIEW integration needs some improvement. Perhaps I'm not past the learning curve peak yet though. ViSci are you using it as more of a tag engine or are you using it for streaming and/or commands as well? It seems like implementation as a tag engine could get cumbersome on the BD publishing individual tags. Clusters work after using their tool, but that also seems cumbersome.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Content

    • By prime8
      We found this bug when updating from 2010 to 2012 last week. Certain types of clusters will not update over datasocket or network-published shared variable. You will only get their default data. Here are the conditions that will cause the problem:
      The cluster contains an array of clusters The array of clusters is not the last item in the cluster The array of clusters is empty

      All three conditions must be fulfilled in order for the bug to appear. An AE was able to reproduce it and it has CAR #385089.
    • By mike5
      Hi all,
      I'm starting using the DSC module, and since it has it's own shared variable data type, I'm forced to use DataSockets for reading and writing data.
      So, the DataSocket Read function has a "connection in" interface. This can take (among other things) connection reference from DataSocket Open function, or a DSC Shared Variable wire.
      If I use the latter, is the connection closed implicitly, or what? And if this implies Open/Read/Close how much overhead do Open and Close this bring?
      Thanks for any insight.
      Br, Mike



×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.