MQTT [Decision to make.]

[Rolf Kalbermatter]

1 hour ago, ShaunR said:

Building the protocol from scratch isn't a barrier for me - I'm on a roll  The difficulty is that it requires DTLS (the UDP version of TLS). DTLS is something I've played with in the past and it was somewhat awkward to integrate into what I have currently so I moved past it and on to other features that I desperately wanted. CoAP would force me to look at DTLS again as it is something I've wanted, but never had a need for.

IMO CoAP is a far superior protocol to MQTT. I don't really understand why MQTT gets so much love.

Most likely because of its use of DTLS. 😁 OpenSSL's support of this was fairly "flaky" back when I did my Network library. Many problems were surrounding it, some of them were actually kind of unfixable with the DTLS standard at that time. Now this was around OpenSSL 0.9.6 or so, so I would assume that a lot has changed since.

And yes I got it to work, but only had done minimum testing with it. It was clear that more extended use of it would sooner or later bring out troubles with it. Some for sure in my interpretation of the OpenSSL API at that time, but some also unfixable for me without changing OpenSSL itself.

Edited March 14, 2023 by Rolf Kalbermatter

[ShaunR]

2 hours ago, Rolf Kalbermatter said:

Most likely because of its use of DTLS. 😁 OpenSSL's support of this was fairly "flaky" back when I did my Network library. Many problems were surrounding it, some of them were actually kind of unfixable with the DTLS standard at that time. Now this was around OpenSSL 0.9.6 or so, so I would assume that a lot has changed since.

And yes I got it to work, but only had done minimum testing with it. It was clear that more extended use of it would sooner or later bring out troubles with it. Some for sure in my interpretation of the OpenSSL API at that time, but some also unfixable for me without changing OpenSSL itself.

Last time I looked it was about 1.1.1e. I don't think it was much better. I bypassed it in the end because it needed callbacks for cookies-wasn't prepared to do that at the time. I'm hoping they've moved on from there with full blown certificate verification but if they haven't, I now have a place for callbacks in the API.

[ShaunR]

So. that's a resounding "don't bother" on the CoAP then 

[Rolf Kalbermatter]

9 minutes ago, ShaunR said:

So. that's a resounding "don't bother" on the CoAP then 

Well. It's more likely a very resounding "I have no idea if I'm ever going to need that. For now I just refrain from commenting on the matter!" 😎

[ShaunR]

12 minutes ago, Rolf Kalbermatter said:

Well. It's more likely a very resounding "I have no idea if I'm ever going to need that. For now I just refrain from commenting on the matter!" 😎

As you are the only one that has commented on it al all (indirectly). I think that's a resounding "don't bother".

Also means I don't have to look too closely at DTLS just yet 

[Jordan Kuehn]

5 hours ago, ShaunR said:

As you are the only one that has commented on it al all (indirectly). I think that's a resounding "don't bother".

*ahem*

Not that I'm suggesting that you do bother haha.

[ShaunR]

40 minutes ago, Jordan Kuehn said:

*ahem*

Not that I'm suggesting that you do bother haha.

Aha! Yes. Apologies.

[ShaunR]

Tada!

Just released 4.3.0 with MQTT support (and, of course, examples). Have fun!

[Antoine Chalons]

Any chance the Encryption Compendium ever supports Linux (Ubuntu)?

[Rolf Kalbermatter]

5 minutes ago, Antoine Chalons said:

Any chance the Encryption Compendium ever supports Linux (Ubuntu)?

Technically very much so. But it is an effort to support multiple platforms. If you intend to commit some backing to Shaun/LVS-Tools I’m fairly convinced he can come up with some proposal. And if he wants to I can even offer some support. But I’m very sure he can do it also alone if there is some incentive. 😀

[ShaunR]

1 hour ago, Antoine Chalons said:

Any chance the Encryption Compendium ever supports Linux (Ubuntu)?

Back at around version 2 Linux and Mac were supported but only Windows was distributed (see licensing below). Linux was a real pain to maintain so I dropped that. No one was interested in Mac so I dropped that too.

There are several reasons why Linux support is unlikely in the future:

1. The NI licencing scheme only works on Windows.
2. There is quite a lot of Windows specific stuff now like the use of the Windows Certificate Store and Windows Messaging for ICMP (off the top of my head-but there's more).
3. Linux is a real pain to maintain for. (50% chance of any one distribution of any software working out-of-the-box).
4. LabVIEW may be synonymous with the Dodo in the next year or two.

1. Has a solution but I'm tentative. 2. Is doable with a quite some of effort. 3 just isn't worth the aggro and 4. Well. Interesting times.

There are lots of reasons why *not*. Very few of why it should.

Edited March 25, 2023 by ShaunR

[ShaunR]

On 3/14/2023 at 1:54 PM, ShaunR said:

Last time I looked it was about 1.1.1e. I don't think it was much better. I bypassed it in the end because it needed callbacks for cookies-wasn't prepared to do that at the time. I'm hoping they've moved on from there with full blown certificate verification but if they haven't, I now have a place for callbacks in the API.

So. DTLS still not all that great. A few functions you need are macros (one or two are missing). Despite starting off with a similar API to TLS, you are quickly reduced to BIO manipulations and trickery. Honestly. It should only be a choice of instantiating DTLS or TLS object but, like so much in OpenSSL, they make you jump through hoops with wildly varying low-level API's (that they will probably deprecate in the future anyway).

I guess my brain just works differently to these developers. So much of what I write is to hide the complexity that they put in.

Edited April 1, 2023 by ShaunR

[Rolf Kalbermatter]

On 4/1/2023 at 2:04 PM, ShaunR said:

So. DTLS still not all that great. A few functions you need are macros (one or two are missing). Despite starting off with a similar API to TLS, you are quickly reduced to BIO manipulations and trickery. Honestly. It should only be a choice of instantiating DTLS or TLS object but, like so much in OpenSSL, they make you jump through hoops with wildly varying low-level API's (that they will probably deprecate in the future anyway).

I guess my brain just works differently to these developers. So much of what I write is to hide the complexity that they put in.

I can clearly see what you are talking about. DTLS is a second class citizen as you have to indeed do BIO trickery. For TLS OpenSSL has convenient wrapper functions. 
 

However I appreciate the OpenSSL developers troubles in trying to shove an intermediate layer into the socket interface, and to make matters worse try to get it to work on multiple platforms While both Linux sockets and WinSock are in principle based on the BSD sockets and this works surprisingly smooth on all these platforms with the same source code for basic TCP and even UDP sockets, things start to get nasty soon when trying to do advanced things such as what OpenSSL needs to do. Under Windows this should be really solved with a socket filter driver that can be installed into WinSock but that is considerably different to how things would be done on BSD and pretty much impossible on Linux without integrating it in the kernel or trying to hack it into pcap.

OpenSSL is clearly a compromise. The alternative would be OS specific protocol filter drivers and there are very few of them and none that supports multiple OSes.

 

 

[ShaunR]

On 4/2/2023 at 3:20 PM, Rolf Kalbermatter said:

I can clearly see what you are talking about. DTLS is a second class citizen as you have to indeed do BIO trickery. For TLS OpenSSL has convenient wrapper functions. 
 

However I appreciate the OpenSSL developers troubles in trying to shove an intermediate layer into the socket interface, and to make matters worse try to get it to work on multiple platforms While both Linux sockets and WinSock are in principle based on the BSD sockets and this works surprisingly smooth on all these platforms with the same source code for basic TCP and even UDP sockets, things start to get nasty soon when trying to do advanced things such as what OpenSSL needs to do. Under Windows this should be really solved with a socket filter driver that can be installed into WinSock but that is considerably different to how things would be done on BSD and pretty much impossible on Linux without integrating it in the kernel or trying to hack it into pcap.

OpenSSL is clearly a compromise. The alternative would be OS specific protocol filter drivers and there are very few of them and none that supports multiple OSes.

I'm not sure of the "convenient wrapper functions" that you refer to since I had to write quite complicated wrappers around reading TLS packets to get it to behave like the LabVIEW primitives.

I think the main issue I have with the DTLS are the cookies which a) are callbacks  and b) are not standardized. They also introduced the SSL_Stateless - a TLS equivalent of DTLS_Listen but, as they state:

Quote

TLSv1.3 is designed to operate over a stream-based transport protocol (such as TCP). If TCP is being used then there is no need to use SSL_stateless()

So I've no idea what that's about.

All the examples I've seen so far all use a different method for generating the cookies so how interoperability is supposed to work - I have no idea. And, unlike TLS , you have to listen before creating a socket and then tell the bio it's connected. That's a completely different procedure and not one easy to merge.

I've also seen reports of the DTLv1_Listen being broken is some versions but not seen anything about it being addressed.

It's a mess!

[Rolf Kalbermatter]

4 minutes ago, ShaunR said:

All the examples I've seen so far all use a different method for generating the cookies so how interoperability is supposed to work - I have no idea. And, unlike TLS , you have to listen before creating a socket and then tell the bio it's connected. That's a completely different procedure and not one easy to merge.

I think the cookies callbacks are not the problem. It's simply a "magic" value being generated and included in the encrypted hello response and the client then copies that back in its subsequent messages to verify that it is itself and not some intermediate adversary that intercepted the messages. So the generation and verification is done on the same side and there is no interoperability issue as the cookie is treated as an opaque binary blob by all intermediate channels.

Why they even require callbacks to be installed rather than providing a default mechanism itself (that could be overridden by a callback if so desired) is a bit beyond me however.

[ShaunR]

OK. Got it working (sort of).

So you don't "need" the callbacks (anti denial of service) but if you do use them then you need a way of matching the cookies between the generate and verify. In the examples they use an application global which is not very useful as I need it per CTX. I had a similar problem with ICMP echo which I partially solved by polling data on the socket (peeking) and dequeuing when the "cookie" matched. That's not an option here though.

The callbacks aren't very useful unless the cookie can be stored with the CTX or SSL session ... somehow. At least not without a lot of effort to create a global singleton with critical sections and lookups.

Any ideas?

 

[A little later]

Hmmm. Maybe I can use SSL_get_session and use the pointer to generate a cookie? Or maybe a BIO reference that I can get from the session?

Edited April 20, 2023 by ShaunR

[ShaunR]

DTLS version 1.0 doesn't work.

It looks like they don't compile the TLS1.0 and TLS1.1 cipher suites on the default build with legacy. There is an option to compile without certain cipher suites (no-{}) which implies they should be enabled by default. Additionally. The no-{} compile options remove the applicable methods for that ciphersuite. However, they are all available. Compiling with enable-{} doesn't get around the problem. Using openssl.exe ciphers -s -tls1_1 yields an empty list. This also means that TLS1.0 and TLS 1.1 don't work either.

Using openssl.exe s_server  -dtls1 -4 -state -debug and openssl.exe s_client  -dtls1 -4 -state -debug yields a Protocol Version Error (70)

DTLS 1.2 is fine, however.

[Rolf Kalbermatter]

31 minutes ago, ShaunR said:

DTLS version 1.0 doesn't work.

It looks like they don't compile the TLS1.0 and TLS1.1 cipher suites on the default build with legacy. There is an option to compile without certain cipher suites (no-{}) which implies they should be enabled by default. Additionally. The no-{} compile options remove the applicable methods for that ciphersuite. However, they are all available. Compiling with enable-{} doesn't get around the problem. Using openssl.exe ciphers -s -tls1_1 yields an empty list. This also means that TLS1.0 and TLS 1.1 don't work either.

Using openssl.exe s_server  -dtls1 -4 -state -debug and openssl.exe s_client  -dtls1 -4 -state -debug yields a Protocol Version Error (70)

DTLS 1.2 is fine, however.

Which version of OpenSSL is that in? TLS 1.0. and 1.1 are/were scheduled to be depreciated for quite some time already. And it seems by default to be disabled in OpenSSL 1.1(.1).

https://github.com/SoftEtherVPN/SoftEtherVPN/issues/1358

 

[ShaunR]

31 minutes ago, Rolf Kalbermatter said:

Which version of OpenSSL is that in? TLS 1.0. and 1.1 are/were scheduled to be depreciated for quite some time already. And it seems by default to be disabled in OpenSSL 1.1(.1).

https://github.com/SoftEtherVPN/SoftEtherVPN/issues/1358

 

3.1.0.

They weren't disabled in 1.1.1. That post seems to be specifically for Debian since it says "OpenSSL on Debian 10 is built with TLS 1.0 disabled.".

You used "no-{tls1|tls1_1} to disable them at compile time. Using that compile option also removes the TLS1 methods from the binary as well. The TLS1 methods are available in the 3.1.0 binary.

[ShaunR]

OK.

So seems it's to do with the security level. They are compiled in but disabled at run-time.

SSL_CTX_set_security_level states:

Quote

 

Level 1

The security level corresponds to a minimum of 80 bits of security. Any parameters offering below 80 bits of security are excluded. As a result RSA, DSA and DH keys shorter than 1024 bits and ECC keys shorter than 160 bits are prohibited. All export cipher suites are prohibited since they all offer less than 80 bits of security. SSL version 2 is prohibited. Any cipher suite using MD5 for the MAC is also prohibited. Note that signatures using SHA1 and MD5 are also forbidden at this level as they have less than 80 security bits. Additionally, SSLv3, TLS 1.0, TLS 1.1 and DTLS 1.0 are all disabled at this level.

 

That last sentence isn't in 1.1.1.

The default security level is 1. You have to set it to 0 to get the rest.

Now we're cooking! 

Edited April 27, 2023 by ShaunR

[Rolf Kalbermatter]

On 4/20/2023 at 10:45 AM, ShaunR said:

OK. Got it working (sort of).

So you don't "need" the callbacks (anti denial of service) but if you do use them then you need a way of matching the cookies between the generate and verify. In the examples they use an application global which is not very useful as I need it per CTX. I had a similar problem with ICMP echo which I partially solved by polling data on the socket (peeking) and dequeuing when the "cookie" matched. That's not an option here though.

The callbacks aren't very useful unless the cookie can be stored with the CTX or SSL session ... somehow. At least not without a lot of effort to create a global singleton with critical sections and lookups.

Any ideas?

 

[A little later]

Hmmm. Maybe I can use SSL_get_session and use the pointer to generate a cookie? Or maybe a BIO reference that I can get from the session?

The way I saw it done in some example code was to generate an application global random secret on first use and use that as key for a HMAC over the actual connection peer address (binary address + port number).

BIO_dgram_get_peer(SSL_get_rbio(ssl), &peer);

Then use the HMAC result as cookie.

Yes it is not super safe as an attacker could learn the key eventually if he tries to attack the server long enough (and knows that that key for the cookie generation is actually constant) but if you don't use an abnormally bad HMAC hash code (SHA256 should be enough), it should be pretty safe.

Edited April 28, 2023 by Rolf Kalbermatter

[ShaunR]

7 minutes ago, Rolf Kalbermatter said:

The way I saw it done in some example code was to generate an application global random secret on first use and use that as key for a HMAC over the actual connection peer address (binary address + port number).

BIO_dgram_get_peer(SSL_get_rbio(ssl), &peer);

Then use the HMAC result as cookie.

Yes it is not super safe as an attacker could learn the key eventually if he tries to attack the server long enough (and knows that that key for the cookie generation is actually constant) but if you don't use an abnormally bad HMAC hash code (SHA256 should be enough), it should be pretty safe.

It's not so much safety but I can have multiple connections (on say 127.0.0.1) and I don't want a global for all the connections. A random per callback would be OK but there is no way to tell the verifying callback what the generator chose (hence they have a global). It would have been preferably to be able to define the cookie to be compared so that the cookie generation could be done in the application rather than inside the callback.

I'm not sure HMAC is all that useful here either (they use SHA1-HMAC by the way). Effectively we are just saying "is it mine? rather than "is it from who it says it is"? They are really relying on the port number from the same address (127.0.0.1, say) and that definitely isn't random and could be repeated.

What I've done is just SHA1 the result of SSL_get_rbio(ssl) It's not "cryptographically" random but is probably random enough for this purpose (this is for DDOS rather than hiding secrets-similar reasoning that we use broken hashes for file integrity) and, unlike their global, it changes on each connect. I could do the whole HMAC thing using the SSL_get_rbio(ssl)  as the random but I'm not sure it's really worth the overhead. Can you give an argument in favour?

[Rolf Kalbermatter]

If you absolutely want to store information on a session level, you could use the CRYPTO_get_ex_new_index(CRYPTO_EX_INDEX_SSL/SSL_CTX, 0, "Name", NULL, NULL, NULL);

Then store the information on the ssl or ctx with SSL_set_ex_data() or SSL_CTX_set_ex_data().

Retrieve it with the according SSL_get_ex_data()/SSL_CTX_get_ex_data().

[ShaunR]

10 hours ago, Rolf Kalbermatter said:

If you absolutely want to store information on a session level, you could use the CRYPTO_get_ex_new_index(CRYPTO_EX_INDEX_SSL/SSL_CTX, 0, "Name", NULL, NULL, NULL);

Then store the information on the ssl or ctx with SSL_set_ex_data() or SSL_CTX_set_ex_data().

Retrieve it with the according SSL_get_ex_data()/SSL_CTX_get_ex_data().

Ooooh. I shall have a play.

[ShaunR]

13 hours ago, Rolf Kalbermatter said:

If you absolutely want to store information on a session level, you could use the CRYPTO_get_ex_new_index(CRYPTO_EX_INDEX_SSL/SSL_CTX, 0, "Name", NULL, NULL, NULL);

Then store the information on the ssl or ctx with SSL_set_ex_data() or SSL_CTX_set_ex_data().

Retrieve it with the according SSL_get_ex_data()/SSL_CTX_get_ex_data().

Having played a bit, it doesn't look that straightforward.

The main idea, it seems, is that you create callbacks that allocate and free the CRYPTO_EX_DATA (which is required for the get and set) but if they are all set to NULL in CRYPTO_get_ex_new_index then you must use CRYPTO_EX_new which would have to be a global and there is no way to associate it with the SSL session.

This seems a lot harder than it should be so maybe I'm not seeing something.