Slow MD5

[Rolf Kalbermatter]

10 minutes ago, Lipko said:

Never mind, maybe I used File/directory info.vi.

That does take some time as LabVIEW has to enumerate the directory contents for all files to get the  size which is the number of files in the directory.

[ShaunR]

5 hours ago, Rolf Kalbermatter said:

The LabVIEW file IO nodes maintain internally a file offset (actually it's the underlying OS file IO functions which do and advance that pointer along as you read)

Error 4 (EOF)still really annoys me. I still maintain it should be a warning and not an error. Every time I have to use my filter error to prevent passing it through I curse and wish a plague on NI. lol

[Rolf Kalbermatter]

55 minutes ago, ShaunR said:

Error 4 (EOF)still really annoys me. I still maintain it should be a warning and not an error. Every time I have to use my filter error to prevent passing it through I curse and wish a plague on NI. lol

Error handling is always a heated discussion topic. You could argue about the same for timeout errors on network and VISA nodes. And some people get in their frillies about the VISA Read returning a warning when it reads as many characters as you have specified it to read.

A warning wouldn't be better as you still would have to read both the status=FALSE and code==4 to detect it. Also I never really work with the EOF error status as I don't read a file until it errors out but until I reach its size. And if you want to work with the EOF status there is a very easy thing. Using the Clear Errors.vi for error 4 you actually get a boolean status if this error was removed from the error cluster if you need that. Otherwise just terminate the loop on the error cluster anyways, clear error 4 in all cases and go on.

[hooovahh]

Here is my minor contribution.  Following a thread on the NI forums it seems Open SSL support when installed (through MAX as an optional software package) the compatible binary is libeay32.so.  So I updated the code to remove the Get/Set file positions, and specify the file path using a conditional disable structure.

I tested it this morning on my RT VM and it worked great.  The same 1.4GB file took 20s using native G and only 2.2s using the Open SSL version.

MD5_my8.vi

[ShaunR]

3 hours ago, Rolf Kalbermatter said:

Using the Clear Errors.vi for error 4 you actually get a boolean status if this error was removed from the error cluster if you need that.

Not mine. It's just two unconnected error clusters. For this reason I have a VI that I wrote in 2009 similar to what NI eventually eventually implemented (except it defaults to 4).

. This is the diagram 

 

3 hours ago, hooovahh said:

I tested it this morning on my RT VM and it worked great.  The same 1.4GB file took 20s using native G and only 2.2s using the Open SSL version.

[ShaunR]

6 hours ago, hooovahh said:

Here is my minor contribution.  Following a thread on the NI forums it seems Open SSL support when installed (through MAX as an optional software package) the compatible binary is libeay32.so.  So I updated the code to remove the Get/Set file positions, and specify the file path using a conditional disable structure.

I tested it this morning on my RT VM and it worked great.  The same 1.4GB file took 20s using native G and only 2.2s using the Open SSL version.

MD5_my8.vi 26.1 kB · 1 download

Some things that need to be done so it doesn't crash arbitrarily (not particularly you, just commenting on the latest incarnation).

EVP_DigestFinal_ex mdlen parameter needs to be pointer to value instead of value as it returns the length,

Return values need to be used (and checked). Currently the functions return Void when they should be I32.

Need to check the MD_CTX pointer isn't 0 for each function.

 

Edit.

almost forgot.

Remove EVP_cleanup(). In versions prior to 1.1.0 it will crash other functions that use EVP and after 1.1.0 it's a no-op.

Edited September 10, 2021 by ShaunR

[JKSH]

21 hours ago, hooovahh said:

the compatible binary is libeay32.so

That means it is using OpenSSL 1.0.x or earlier.

In OpenSSL 1.1.0, "libeay32" was renamed to "libcrypto"

[hooovahh]

On 9/11/2021 at 7:42 AM, JKSH said:

In OpenSSL 1.1.0, "libeay32" was renamed to "libcrypto"

Thanks for the info.  Looks like a wildcard can't be used to select the new binary based on the naming.  I could add code to say if we are on Linux, and the first call to "OpenSSL_add_all_digests" returns an error 7 (file not found) when using the path "libeay32.so" to the binary, then to try one more time using the path of "libcrypto.so".  I mention this method because I don't want to specify the full path, but the Check If File or Folder Exists doesn't search for a file in global path locations.  Not sure if there is a more elegant solution.

Speaking of hard coded paths the Windows method is obviously not ideal at the moment too since it is a hard codded full path.  What would be the best solution here?  Reading registry information on first call?

[hooovahh]

Okay attached is an updated version that I think supports the changes I made.  It reads the registry in 32 or 64 bit LabVIEW finding the path to the shared installation location.  In Linux it tries both the libeay32.so and libcrypto.so.  In both cases it keeps the found path in a feedback node for later calls.  I also added outputs instead of void for calls, but am now realizing this version doesn't do anything with them yet.

MD5_my9.vi

[Antoine Chalons]

Has anyone been able to make this work on Linux desktop?
I use LV 2020 SP1 (therefore 64 bit) on Ubuntu 20

I ran

Quote

apt-get update
apt-get install libssl-dev

to install libcrypto.so but at the first call of the so "OpenSSL_add_all_digests" I get error 15 "resource not found" and indeed if in the CLFN I put the full libcrypto.so path, the "OpenSSL_add_all_digests" is not in the "function" list

edit : I guess the answer is RTFM

With help from a co-worker we could make it work on Linux, multiple functions have different names, will post the VI soon.

 

Edited August 29, 2022 by Antoine Chalons

[Rolf Kalbermatter]

39 minutes ago, Antoine Chalons said:

Has anyone been able to make this work on Linux desktop?
I use LV 2020 SP1 (therefore 64 bit) on Ubuntu 20

I ran

to install libcrypto.so but at the first call of the so "OpenSSL_add_all_digests" I get error 15 "resource not found" and indeed if in the CLFN I put the full libcrypto.so path, the "OpenSSL_add_all_digests" is not in the "function" list

That's most likely because of this in the OpenSSL headers:

# if OPENSSL_API_COMPAT < 0x10100000L

These functions were required to be called in OpenSSL before 1.1.0but since 1.1.0 OpenSSL automatically initializes its engines on the first call of any function that creates a context of similar session.

And since 1.1.0 is already EOL too and you should either use 1.1.1 or even better 3.0.x, it would be indeed strange if your OpenSSL library still included those APIs. I think the old libssleay.so still contained it in 1.0.1 but it was unnecessary to call that, but when changing the shared library names they also axed many of those compatibility hacks too.

You should probably call

OPENSSL_init_crypto(OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS | OPENSSL_INIT_LOAD_CONFIG, NULL);

and yes this means reading the headers to see what numeric values those constant defines have.

 

Edited August 29, 2022 by Rolf Kalbermatter

[Antoine Chalons]

Just now, Rolf Kalbermatter said:

That's most likely because of this in the OpenSSL headers:

# if OPENSSL_API_COMPAT < 0x10100000L

These functions were required to be called in OpenSSL before 1.1.1 but since 1.1.1 OpenSSL automatically initializes its engines on the first call of any function that creates a context of similar session.

And since 1.1.0 is already EOL too and you should either use 1.1.1 or even better 3.0.x, it would be indeed strange if your OpenSSL library still included those APIs. I think the old libssleay.so still contained it in 1.0.1 but it was unnecessary to call that, but when changing the shared library names they also axed many of those compatibility hacks too.

You should probably call

OPENSSL_init_crypto(OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS | OPENSSL_INIT_LOAD_CONFIG, NULL);

 

and yes this means reading the headers to see what numeric values those constant defines have.

 

Your reply came as I was typing my edit... Thanks

[Antoine Chalons]

On 9/13/2021 at 4:59 PM, hooovahh said:

Okay attached is an updated version that I think supports the changes I made.  It reads the registry in 32 or 64 bit LabVIEW finding the path to the shared installation location.  In Linux it tries both the libeay32.so and libcrypto.so.  In both cases it keeps the found path in a feedback node for later calls.  I also added outputs instead of void for calls, but am now realizing this version doesn't do anything with them yet.

MD5_my9.vi 32.87 kB · 9 downloads

why on earth did you use a ring and a property node on that ring to get the method (md4, md5, etc.)?
an enum with a format into string seems nicer.

A, also, if your VI used on Linux and built as a shared library (in order to run the app as a service) then it causes a crash.
With enum+format into string, it works.

[hooovahh]

2 hours ago, Antoine Chalons said:

why on earth did you use a ring and a property node on that ring to get the method (md4, md5, etc.)?
an enum with a format into string seems nicer.

A, also, if your VI used on Linux and built as a shared library (in order to run the app as a service) then it causes a crash.

That is some ugly ass code for sure.  I'm fairly certain I didn't create that ring, and instead just copied it from some other example set of code.  I can never see myself center justifying a control like that so I'm guessing I just got it from something else, and then cut and pasted code until it worked.  Enum and format into string is the way to go.  That being said I'm pretty sure I would have tested this on a Linux RT machine and didn't see a crash at least running in source.

[ShaunR]

As this has popped up again and someone may find it useful. Here are some comparative benchmarks between LabVIEW and OpenSSL hashes (sorted fastest to slowest).

[Novgorod]

Interesting topic.. Has anyone successfully played around with the built-in (undocumented) Labview manager function "GetMD5Digest"? I assume it's used internally for VI block hashing and other stuff and I'm wondering what's the performance compared with the "public" G code MD5 utilities. If it's faster, it could be a very practical alternative because the manager functions are platform-independent (right?)..

[Rolf Kalbermatter]

11 hours ago, Novgorod said:

Interesting topic.. Has anyone successfully played around with the built-in (undocumented) Labview manager function "GetMD5Digest"? I assume it's used internally for VI block hashing and other stuff and I'm wondering what's the performance compared with the "public" G code MD5 utilities. If it's faster, it could be a very practical alternative because the manager functions are platform-independent (right?).

Yes that function is used for generating hashes in LabVIEW itself for various things including password protection. It was never meant for hashing huge buffers and is basically the standard Open Source code for MD5 hashing that floats on the internet for as long as the internet exists. It is similar to what the OpenSSL code would do, although OpenSSL is likely a little more optimized/streamlined. This function exists in LabVIEW since about at least 5.0 or thereabout and likely has never been looked at again since, other than use it.

You could of course call it, and dig in the LabVIEW attic with its many rusty nails, but why? Just because you can does not mean you should! 🙂

If you absolutely insist you might try yourself. This is the prototype:

MgErr GetMD5Digest(LStrHandle message, LStrHandle digest);

 

Edited August 30 by Rolf Kalbermatter

[Novgorod]

5 hours ago, Rolf Kalbermatter said:

Yes that function is used for generating hashes in LabVIEW itself for various things including password protection. It was never meant for hashing huge buffers and is basically the standard Open Source code for MD5 hashing that floats on the internet for as long as the internet exists. It is similar to what the OpenSSL code would do, although OpenSSL is likely a little more optimized/streamlined. This function exists in LabVIEW since about at least 5.0 or thereabout and likely has never been looked at again since, other than use it.

You could of course call it, and dig in the LabVIEW attic with its many rusty nails, but why? Just because you can does not mean you should! 🙂

If you absolutely insist you might try yourself. This is the prototype:

MgErr GetMD5Digest(LStrHandle message, LStrHandle digest);

 

W00t, thanks! Where did you steal the forbidden .h file? 😁

Of course you're right, it should be sufficient in almost all use cases to simply use the provided vi.lib version. It's more a curiosity thing and also a neat trick to have very compact portable tools. For example, the built-in zip/deflate/inflate functions eliminate the need for external dlls, since (to my knowledge) there is no pure G code zip implementation, but that's a story for another thread.

I tried using the GetMD5Digest function but I'm getting strange behavior (unsurprisingly, I know)... The MgErr type is not defined in extcode.h, but I assume it's just an int32 error code (right?), so that shouldn't be the issue. Here's a minimal implementation:

The digest output string seems to require at least a 16-byte pre-allocated buffer or it returns an empty string. The confusing part is that the input string buffer gets overwritten with a seemingly random 16-byte string and the output MD5 digest is the correct MD5 hash for that random 16-byte string and NOT for the actual input string. That random string also changes every time the function is called. Maybe it needs some kind of initializer? I'd appreciate if you have any more info on that, just for the sake of curiosity... 😉

Edited August 30 by Novgorod

[LogMAN]

23 minutes ago, Novgorod said:

The MgErr type is not defined in extcode.h

It is defined in mine:

Quote

/** @brief Manager error code type. */
typedef int32 MgErr;

Edit: I believe it is also defined in the product documentation but NI's website is down again.

Edited August 30 by LogMAN

[Novgorod]

44 minutes ago, LogMAN said:

It is defined in mine:

What the heck? You're right, my Ctrl+F didn't search the whole document, my bad...

[hooovahh]

1 hour ago, Novgorod said:

Of course you're right, it should be sufficient in almost all use cases to simply use the provided vi.lib version. It's more a curiosity thing and also a neat trick to have very compact portable tools. For example, the built-in zip/deflate/inflate functions eliminate the need for external dlls, since (to my knowledge) there is no pure G code zip implementation, but that's a story for another thread.

I did try to implement inflate/deflate in pure G.  There was a website that explained in pretty easy to understand english how the compression works, and gave example code on how to implement it.  I went through it step by step but it still wasn't producing the correct data.  After trying in my free time for a bit I gave up. I thought it would be helpful on Linux RT for web server stuff where it could make PNG compressed images. There are alternatives.

[Rolf Kalbermatter]

43 minutes ago, hooovahh said:

I did try to implement inflate/deflate in pure G.  There was a website that explained in pretty easy to understand english how the compression works, and gave example code on how to implement it.  I went through it step by step but it still wasn't producing the correct data.  After trying in my free time for a bit I gave up. I thought it would be helpful on Linux RT for web server stuff where it could make PNG compressed images. There are alternatives.

Believe me, while it is certainly possible, this is one area where you do not want to use LabVIEW to implement it. Main reason are twofold:

It's a lot of bit twiddling and while LabVIEW supports rotate and boolean operators on numbers and is quite good at it, there is an inherent overhead since LabVIEW does do some extra safe keeping to avoid errors that you can easily do in programming languages like C. This typically is negligible except when you do it in a loop 8 * many million times. Then these small extra checks really can add up. So a LabVIEW inflate or deflate algorithm will always be significantly slower than the same in C, compiled as optimized code. This is because the boolean and rotate operations take an extremely short time to execute and the extra LabVIEW checks each time to be always safe are significant in comparison.

The second reason is the fact that bit twiddling is an extremely easy thing to get wrong and mess up. You may think to have copied the according C code exactly but in most cases there are more or less subtle things that will turn your results upside down. Debugging on that level is extremely frustrating. I have run code many many times in single step mode trying to find the error in those binary operators and missed the actual issue many times until finally nailing it. It's an utterly frustrating experience and it made me long ago promise to myself to not try to implement compression or decompression as well as encryption routines in LabVIEW except the most trivial ones. ZLIB however is not trivial, not at all!

[Rolf Kalbermatter]

18 hours ago, Novgorod said:

I tried using the GetMD5Digest function but I'm getting strange behavior (unsurprisingly, I know)... The MgErr type is not defined in extcode.h, but I assume it's just an int32 error code (right?), so that shouldn't be the issue. Here's a minimal implementation:

The digest output string seems to require at least a 16-byte pre-allocated buffer or it returns an empty string. The confusing part is that the input string buffer gets overwritten with a seemingly random 16-byte string and the output MD5 digest is the correct MD5 hash for that random 16-byte string and NOT for the actual input string. That random string also changes every time the function is called. Maybe it needs some kind of initializer? I'd appreciate if you have any more info on that, just for the sake of curiosity... 😉

Well, that's what you get for strolling in the LabVIEW attic. Seems the prototype and functionality as I posted it was correct at LabVIEW 5.x times until sometimes later, but later got changed by NI for some reasons. Since it was an undocumented API, they are of course free to do that anytime they like (and this is also one of the main reasons to not document such APIs). The only one that could get hurt is themselves when they happen to forget to adapt a caller somewhere to the new function interface (and of course people like me and you who try to sneak in anyways).

It's also while this is called the LabVIEW attic with many rusty nails laying in the open.

[Novgorod]

6 minutes ago, Rolf Kalbermatter said:

Well, that's what you get for strolling in the LabVIEW attic. Seems the prototype and functionality as I posted it was correct at LabVIEW 5.x times until sometimes later, but later got changed by NI for some reasons.

I'm not complaining about an undocumented/non-public API change after 20 years, just being curious .. I poked around a bit more - the prototype looks alright, it's just that the function is not intended to provide an MD5-hash of a caller-supplied message. The "message" parameter is actually an output that returns 16 random bytes (there are RNG/GUID calls at least under Windows), which get MD5 hashed. So it outputs a 16-byte random string (GUID) and its MD5 hash for some very specific application. The actual MD5 subroutine is not exposed, so it was all a red herring stinking up the attic...

[Rolf Kalbermatter]

4 hours ago, Novgorod said:

I'm not complaining about an undocumented/non-public API change after 20 years, just being curious .. I poked around a bit more - the prototype looks alright, it's just that the function is not intended to provide an MD5-hash of a caller-supplied message. The "message" parameter is actually an output that returns 16 random bytes (there are RNG/GUID calls at least under Windows), which get MD5 hashed. So it outputs a 16-byte random string (GUID) and its MD5 hash for some very specific application. The actual MD5 subroutine is not exposed, so it was all a red herring stinking up the attic...

The most easy solution of all: But likely not extremely fast although fairly fast should do it!

https://forums.ni.com/t5/LabVIEW/Any-GUID-generating-VI-available/m-p/4392240#M1293609

Even this should be considered LabVIEW attic area as indicated by the poop brown color of the VI Server node, but with considerably less rusty nails. Worst case, it simply stops working one day. It's most likely using the MD5 code that the old GetMD5Hash() was using. Anything else would have been traditionally at least potentially trouble to implement on non Windows platforms without help of extra library installations such as OpenSSL, EmbedTSL or coreutils, etc.

Edited August 31 by Rolf Kalbermatter