MQTT [Decision to make.]

[ShaunR]

Hi all.

Before you get too excited, this is a sort of discussion to help me figure out what to do for a commercial product. I'm in two minds so I thought more minds could out-vote either of them.

I'm sure some people of the community are ware of ECL. I recently released a new version which fulfilled my bug-bears about certain features I've wanted for a while. Now I'm free to implement anything I want in terms of nice-to-haves instead of "I really want this". So I settled on MQTT as it fits with my aspirations to increase the protocols and ECL now has everything at the lower levels that would be needed for virtually any high level protocol.

So the decision for supporting MQTT is made but I'm in two minds whether to use a 3rd party pre-built library and provide LabVIEW bindings of whether to write one from scratch in pure LabVIEW.

Writing from scratch would be an interesting learning exercise and licencing would not be a problem. It'd take some time but nowhere near as much as I have before a new feature release is due. I've already read the spec and prototyped some stuff and concluded it'd be a pain but perfectly doable once I put more energy in to it. The only sub-decisions I'd have to make is if it would be more elegant to do things like packet and bit stuffing in a DLL rather than LabVIEW as, for my first foray,  I have the impression it would be inelegant in LabVIEW.  But that's not really a factor, just an implementation detail (there are plenty of DLLs that ECL relies on already). I could also support the latest version but it might be a pain to support multiple protocol versions to begin with (rules, eh?).

My other route is to use a 3rd party library and write LabVIEW wrappers. Certainly the easier route for implementation but most libraries only support MQTT 3.x rather than version 5. Then we get to the dreaded licencing. The most complete implementations have awkward licences and the permissive ones are a little incomplete in terms of features and version support. I would definitely need a DLL (not really a problem) and the library would need modification to use existing ECL transports. Current implementations tend to handle the TLS and websockets themselves but I obviously want to use the ECL. Those transports are far more configurable and I want it to fit in with the methods used in all the other protocols (we don't need multiple different ways of defining a X.509 certificate for use, do we?). Additionally, most 3rd party libraries use OpenSSL 1.1.1 whereas ECL uses 3.x so some internal functions must be rewritten  to account for deprecations.

So from scratch (lets call it 1.) means no licence headaches, support latest MQTT version but longer to implement (I estimate 2x longer) but using a 3rd party (lets call it 2.) will be much quicker but require modification, headaches with licences, maybe only MQTT 3.x  and probably a lot less hair at the end of it.

Where I am at the moment is I have both 1 & 2 as stand-alone (which was part of my feasability investigation) where I can connect to a server and subscribe to a topic (no publish, no authentication or funny stuff like Will, QoS and Retain).

So I have to decide which route to go the rest of the way with (1 or 2) or maybe there is another?

Thoughts?

Edited January 19, 2023 by ShaunR

[Jordan Kuehn]

I assume you are familiar with these already?

https://github.com/LabVIEW-Open-Source/LV-MQTT-Broker

https://www.wireflow.com/products/software/wf-wirequeue-mqtt-toolkit/

I use the latter extensively, both in Windows and Linux RT. I have some curiosity regarding the first one, both the client and the broker, but have yet to try it out. I do not know how many of us there are, but I do use MQTT heavily and would welcome an additional (paid or OSS) toolkit. One feature of a native broker that I would love to see is the ability to cluster brokers like EQMX or other.

[ShaunR]

I won't check those links out as I don't want to be accused of plagiarism if I design from scratch. 

18 minutes ago, Jordan Kuehn said:

One feature of a native broker that I would love to see is the ability to cluster brokers like EQMX or other.

Clustering requires either global storage or high bandwidth replication. That would be outside the scope of the initial foray and require (probably)a database for the former.

A long time ago I created some software called Dispatcher (the first version was on Lavag for a while). It had a nascent ability to load balance between multiple services which would register with it. It would be possible to do something similar which would be 1/2 of a cluster. With a load balancer you can do a very rudimentary cluster by making Brokers subscribe to other brokers (Client/Broker hybrids). However. Clustering isn't part of the MQTT specification and is a domain in it's own right.

It would probably pan out as a phased release. First a client, then maybe a broker with consideration for clustering coming a very distant 3rd.

[Jordan Kuehn]

Fair enough. Good luck and please keep us up to date as you progress or need beta testers!

[Porter]

18 hours ago, ShaunR said:

I won't check those links out as I don't want to be accused of plagiarism if I design from scratch.

FYI the first link is licensed under BSD 0-Clause: LV-MQTT-Broker/LICENSE at master · LabVIEW-Open-Source/LV-MQTT-Broker · GitHub

I've used the MQTT client in a project already. Its well written and actively maintained. Definitely worth checking out.

[ShaunR]

39 minutes ago, Porter said:

FYI the first link is licensed under BSD 0-Clause

Granted. But it could still be claimed that some parts of a solution that I come up with from scratch are taken from it and therefore have to adhere to their licence. It's just better to not look and increase the likelihood that solutions to the same problems are solved in different ways and be able to show working of how one arrived at that solution should the need require it..

That's not a consideration if I use a 3rd party library, of course. And there is no reason that the 3rd party library couldn't be a LabVIEW one. I haven't taken that decision though (hence the thread).

[jacobson]

My first thought would be why even think about building your own solution if you haven't explored the existing alternatives. I only did some minor tests with Francois' API but the client seemed to work well in both Windows and Linux so if you built your own API and posted it to LAVA the first question I would probably ask is why I would want to choose your API over the one I already was using. I think you have to spend some time evaluating existing APIs so you can have a set of reasons as to why you're building yet another API.

Even if you have reasons for not liking an existing API it might still be a good use of time to see if you can address those issues by working from the existing API. As an example, if you find that the performance isn't good enough in pure LabVIEW you might not need to start from scratch. If you can get a big performance boost by replacing a few key internal VIs with DLL calls maybe you can bring that to the owner of the existing API and see if that's a change they would make.

If they're unwilling to make those changes (maybe they find value in keeping everything in pure LabVIEW) then you can start working on your own API with very clear reasons as to why someone would even want to use it. 

[ShaunR]

1 hour ago, jacobson said:

the first question I would probably ask is why I would want to choose your API over the one I already was using.

It's an additional protocol to an existing commercial product. If you only want MQTT and not the rest of the features (TLS1.3, SSH, SFTP, Websockets, ICMP etc), then keep using what you have.

1 hour ago, jacobson said:

Even if you have reasons for not liking an existing API it might still be a good use of time to see if you can address those issues by working from the existing API.

I've already explained about licencing.

1 hour ago, jacobson said:

If they're unwilling to make those changes (maybe they find value in keeping everything in pure LabVIEW) then you can start working on your own API with very clear reasons as to why someone would even want to use it. 

I already know why people want (and how) to use MQTT. I just want a bit of input as to which way to jump on a decision I'm in two minds over. FYI. The decision prior to this was CoAP or MQTT. I decided MQTT.

 

Edited January 20, 2023 by ShaunR

[ShaunR]

OK. Decision made - reinvent the wheel 

The final nail in the coffin of using something like Paho MQTT was DLL distribution and integration with current features.

[ShaunR]

What a crappy, poorly thought-out protocol. 

• You have a variable length integer (up to 4 bytes) but lets also have some fixed sized integer types less than 4 bytes for the giggles.
• You have an error byte...sometimes...with command/response contextual meanings. 
• You have properties, that may or may not be there, but only for some commands and responses.
• You have command specific, fixed, fields before the properties for some commands/responses.
• Some commands ack, except when they don't.
• Half the state is in the client and the other half is in the server.
• A significant number of the "features" are to get around the previous bullet point.

 

[Rolf Kalbermatter]

On 2/13/2023 at 12:44 AM, ShaunR said:

What a crappy, poorly thought-out protocol. 

• You have a variable length integer (up to 4 bytes) but lets also have some fixed sized integer types less than 4 bytes for the giggles.
• You have an error byte...sometimes...with command/response contextual meanings. 
• You have properties, that may or may not be there, but only for some commands and responses.
• You have command specific, fixed, fields before the properties for some commands/responses.
• Some commands ack, except when they don't.
• Half the state is in the client and the other half is in the server.
• A significant number of the "features" are to get around the previous bullet point.

It's most likely organically grown. Guaranteed green and maybe even vegan. 😁

And grown and groomed by many different "developers".

[ShaunR]

2 hours ago, Rolf Kalbermatter said:

It's most likely organically grown. Guaranteed green and maybe even vegan. 😁

And grown and groomed by many different "developers".

It was obviously grown like a furry mold.

The way they have designed it means that the user has to know so much about the protocol and what properties to bind to what messages. I'm having a hard time burying the complexity for the user. I wouldn't be surprised if a lot of the LabVIEW solutions out there are only partial implementations and ignore properties completely.

Edited February 14, 2023 by ShaunR

[ShaunR]

5 hours ago, Rolf Kalbermatter said:

It's most likely organically grown. Guaranteed green and maybe even vegan. 😁

And grown and groomed by many different "developers".

Aha! 

So properties are MQTT-V5 only. V3.1.1 has those fixed fields and doesn't have a variable integer at all. That explains the fixed fields and fixed sized integers I moaned about - backwards compatibility. I'm ambivalent if I can forgive them for that as it's a noble trade-off but they could have handled it much better than they have. 

[ShaunR]

Well. It didn't take long to prototype the MQTT client as I was expecting - once I figured out it wasn't my code and it was just properties weren't supported by the public Mosquito server.  The Hive MQTT public server doesn't seem to support V5 either.

Anyone know of a public V5 server?

 

[JKSH]

4 hours ago, ShaunR said:

Anyone know of a public V5 server?

It's not clear to me whether the EMQX public broker supports v5 or not... https://www.emqx.com/en/mqtt/public-mqtt5-broker

...but their downloadable one does, apparently: https://www.emqx.com/en/try?product=broker

[ShaunR]

4 hours ago, JKSH said:

It's not clear to me whether the EMQX public broker supports v5 or not... https://www.emqx.com/en/mqtt/public-mqtt5-broker

...but their downloadable one does, apparently: https://www.emqx.com/en/try?product=broker

Sweet. I'll check it out.

[ShaunR]

Ok. Turns out that Mosquitto and Hive partially support MQTT v5. Mosquito more than Hive (or it may just be more tolerant) but not all the properties in the spec. So they are probably v4.x. EMQX is a 14 day trial so I didn't bother with it in the end.

I'm just embarking on the cleanup and documentation (the fun's over ) but I have a quick question about the packet ID's. The spec doesn't define what they should be apart a u16 and non-zero (when they are there). So I decided to make them random (usually a good choice in crypto stuff). Mosquitto uses sequential incrementing by 1 and Hive uses sequential incrementing by 50. Technically, they are only used internally by the clients and servers but I was wondering if anyone relies on them being sequential for other things where a random packet ID would be a pain?

 

[Rolf Kalbermatter]

19 hours ago, ShaunR said:

I'm just embarking on the cleanup and documentation (the fun's over ) but I have a quick question about the packet ID's. The spec doesn't define what they should be apart a u16 and non-zero (when they are there). So I decided to make them random (usually a good choice in crypto stuff). Mosquitto uses sequential incrementing by 1 and Hive uses sequential incrementing by 50. Technically, they are only used internally by the clients and servers but I was wondering if anyone relies on them being sequential for other things where a random packet ID would be a pain?

I haven't looked at MQTT but this kind of ID is usually only really used to match responses to the request and/or verify that you got the right response. It should not contain any crypto relevant value, so I would not expect it to matter. More important would be that you can reasonably guarantee that your random generator never will result in the same ID being returned within a small time window, as that makes the whole purpose of being able to match the response to the request pretty difficult. In that sense a monotonically increasing ID that eventually wraps around is actually more useful than a true random ID.

Usually the client generates the ID and sends it with the request. The server simply copies it into the response without doing anything else. The client can then verify that the received response is actually for the sent out request and not some other stray request. In a connection based protocol using TCP/IP, this is usually kind of superfluous but for UDP or serial port communication it can be useful.

Edited February 21, 2023 by Rolf Kalbermatter

[ShaunR]

1 hour ago, Rolf Kalbermatter said:

I haven't looked at MQTT but this kind of ID is usually only really used to match responses to the request and/or verify that you got the right response. It should not contain any crypto relevant value, so I would not expect it to matter. More important would be that you can reasonably guarantee that your random generator never will result in the same ID being returned within a small time window, as that makes the whole purpose of being able to match the response to the request pretty difficult. In that sense a monotonically increasing ID that eventually wraps around is actually more useful than a true random ID.

Indeed. It's not so much a time window. It's if packets are unacknowledged for the QOS modes. The Packet ID is shared between the Server and client in order for them to synchronise state for the duration of a packet in-flight. That's not an issue for the library as it necessarily needs to do that but I expose the Packet ID to the user in order that they can do session packet tracing. A server, for example, will take into account which packets are unacknowledged for a session and may resend or an unknown Packet ID is received and the connection must be closed.

1 hour ago, Rolf Kalbermatter said:

Usually the client generates the ID and sends it with the request. The server simply copies it into the response without doing anything else. The client can then verify that the received response is actually for the sent out request and not some other stray request. In a connection based protocol using TCP/IP, this is usually kind of superfluous but for UDP or serial port communication it can be useful.

True. However MQTT can retain sessions across connections. In order to do that it keeps a track of the unacknowledged Packet ID's (QOS 1&2). Technically, they should be able to cope with a random vs sequential Packet via lookup tables if the user really needs to but I thought maybe there was a use case for sequentially increasing Packet ID's to, maybe, identify packet order.

[ShaunR]

On 1/19/2023 at 7:26 PM, Jordan Kuehn said:

One feature of a native broker that I would love to see is the ability to cluster brokers like EQMX or other.

This is an interesting feature that I guess is one of the mechanisms for clustering. It's basically like an HTTP redirect but I'm not sure how I'm going to handle it ATM. CONNACK is fine but a disconnect is usually the end of the conversation with an error 66.  Interesting that they don't mention a redirect depth.

Quote

 

4.11 Server redirection

A Server can request that the Client uses another Server by sending CONNACK or DISCONNECT with

Reason Codes 0x9C (Use another server), or 0x9D (Server moved) as described in section 4.13. When

sending one of these Reason Codes, the Server MAY also include a Server Reference property to

indicate the location of the Server or Servers the Client SHOULD use.

 

The Reason Code 0x9C (Use another server) specifies that the Client SHOULD temporarily switch to

using another Server. The other Server is either already known to the Client, or is specified using a

Server Reference.

 

The Reason Code 0x9D (Server moved) specifies that the Client SHOULD permanently switch to using

another Server. The other Server is either already known to the Client, or is specified using a Server

Reference.

 

 

[ShaunR]

What is it they say about the last 10% of a project is 90% of the work?

I stand by my original statement "What a crappy, poorly thought-out protocol" 

So. Like I said earlier, 1/2 the state is in the client and 1/2 in the server. I'll add now "they have to agree". So what if they don't agree? Well, here we go.

 

Client and server maintain a list of packets (Packet Identifier).

Quote

• MUST assign an unused Packet Identifier when it has a new Application Message to publish

Ok. Can do that.

Quote

MUST treat the PUBLISH packet as “unacknowledged” until it has received the corresponding PUBREC packet from the receive.

MUST NOT re-send the PUBLISH once it has sent the corresponding PUBREL packet

OK. But what if I don't get the PUBREC? I can send a duplicate right?

Quote

The DUP flag MUST be set to 1 by the Client or Server when it attempts to re-deliver a PUBLISH packet

Sweet

Oh wait. 

Quote

When a Client reconnects with Clean Start set to 0 and a session is present, both the Client and Server
MUST resend any unacknowledged PUBLISH packets (where QoS > 0) and PUBREL packets using their
original Packet Identifiers. This is the only circumstance where a Client or Server is REQUIRED to resend
messages. Clients and Servers MUST NOT resend messages at any other time

Ok  So we save them up for connect only. Job done, right?

Quote

Clients and Servers control the number of unacknowledged PUBLISH packets they receive by using a Receive Maximum.

If it receives more than Receive Maximum QoS 1 and QoS 2 PUBLISH packets where it has not sent a PUBACK or PUBCOMP in response, the Server uses a DISCONNECT packet with Reason Code 0x93 (Receive Maximum exceeded)

Ummm.  So we can only send duplicates at connect (and only if it's a persistent session) and there can only be so many unacknowledged ACK's otherwise we get spanked. So what happens if the unacknowledged packets reaches the limit?

Quote

 It MAY continue to send PUBLISH packets with QoS 0, or it MAY choose to suspend sending these as well.

Really? Go silent or hope for the best depending on the whim of the developer? 

Hmmm. So what does this look like in the real world.

• EMQX: 2 hours before getting Receive Maximum exceeded.
• Hive: 20 minutes before getting Receive Maximum exceeded.
• Mosquitto: 1 hour before getting Receive Maximum exceeded.

That's no good. Lets send duplicates if we have some acks and are getting near the limit (that's what we should be doing IMHO)

• EMQX: 40 minutes before getting getting disconnected (Packet Identifier in use). Can't blame them. Crappy protocol.
• Hive: 24 hrs and still going but getting duplicates in the subscriber (that's not right for QOS 2 but could live with it.)
• Mosquitto: 24 hrs. and still going. No problems. We have a winner!

This is a doozy too:

Quote

 

For Reason Code 0x91 (Packet identifier in use), the response to this is either to try to fix the state, or to reset the Session state by connecting using Clean Start set to 1, or to decide if the Client or Server implementations are defective.

 

Fix the state (normally that would be with duplicates), restart (without persistent session-poor subscribers  ) or blame someone else.  

Am I missing something?

 

Edited February 27, 2023 by ShaunR

[ShaunR]

Anyone interested in CoAP?

[Rolf Kalbermatter]

On 3/11/2023 at 3:32 PM, ShaunR said:

Anyone interested in CoAP?

Not yet 😁

[Jordan Kuehn]

On 3/11/2023 at 8:32 AM, ShaunR said:

Anyone interested in CoAP?

I have had a passing interest. I think you were extoling its virtues some time ago. But when I saw I'd be needing to build the LV implementation of the protocol from scratch I lost interest. I also didn't see a lot of wide adoption at the time in areas where I was working, but that is probably worth a fresh look.

[ShaunR]

14 hours ago, Jordan Kuehn said:

I have had a passing interest. I think you were extoling its virtues some time ago. But when I saw I'd be needing to build the LV implementation of the protocol from scratch I lost interest. I also didn't see a lot of wide adoption at the time in areas where I was working, but that is probably worth a fresh look.

Building the protocol from scratch isn't a barrier for me - I'm on a roll  The difficulty is that it requires DTLS (the UDP version of TLS). DTLS is something I've played with in the past and it was somewhat awkward to integrate into what I have currently so I moved past it and on to other features that I desperately wanted. CoAP would force me to look at DTLS again as it is something I've wanted, but never had a need for.

IMO CoAP is a far superior protocol to MQTT. I don't really understand why MQTT gets so much love.