Slow MD5

[Rolf Kalbermatter]

10 minutes ago, Lipko said:

Never mind, maybe I used File/directory info.vi.

That does take some time as LabVIEW has to enumerate the directory contents for all files to get the  size which is the number of files in the directory.

[ShaunR]

5 hours ago, Rolf Kalbermatter said:

The LabVIEW file IO nodes maintain internally a file offset (actually it's the underlying OS file IO functions which do and advance that pointer along as you read)

Error 4 (EOF)still really annoys me. I still maintain it should be a warning and not an error. Every time I have to use my filter error to prevent passing it through I curse and wish a plague on NI. lol

[Rolf Kalbermatter]

55 minutes ago, ShaunR said:

Error 4 (EOF)still really annoys me. I still maintain it should be a warning and not an error. Every time I have to use my filter error to prevent passing it through I curse and wish a plague on NI. lol

Error handling is always a heated discussion topic. You could argue about the same for timeout errors on network and VISA nodes. And some people get in their frillies about the VISA Read returning a warning when it reads as many characters as you have specified it to read.

A warning wouldn't be better as you still would have to read both the status=FALSE and code==4 to detect it. Also I never really work with the EOF error status as I don't read a file until it errors out but until I reach its size. And if you want to work with the EOF status there is a very easy thing. Using the Clear Errors.vi for error 4 you actually get a boolean status if this error was removed from the error cluster if you need that. Otherwise just terminate the loop on the error cluster anyways, clear error 4 in all cases and go on.

[hooovahh]

Here is my minor contribution.  Following a thread on the NI forums it seems Open SSL support when installed (through MAX as an optional software package) the compatible binary is libeay32.so.  So I updated the code to remove the Get/Set file positions, and specify the file path using a conditional disable structure.

I tested it this morning on my RT VM and it worked great.  The same 1.4GB file took 20s using native G and only 2.2s using the Open SSL version.

MD5_my8.vi

[ShaunR]

3 hours ago, Rolf Kalbermatter said:

Using the Clear Errors.vi for error 4 you actually get a boolean status if this error was removed from the error cluster if you need that.

Not mine. It's just two unconnected error clusters. For this reason I have a VI that I wrote in 2009 similar to what NI eventually eventually implemented (except it defaults to 4).

. This is the diagram 

 

3 hours ago, hooovahh said:

I tested it this morning on my RT VM and it worked great.  The same 1.4GB file took 20s using native G and only 2.2s using the Open SSL version.

[ShaunR]

6 hours ago, hooovahh said:

Here is my minor contribution.  Following a thread on the NI forums it seems Open SSL support when installed (through MAX as an optional software package) the compatible binary is libeay32.so.  So I updated the code to remove the Get/Set file positions, and specify the file path using a conditional disable structure.

I tested it this morning on my RT VM and it worked great.  The same 1.4GB file took 20s using native G and only 2.2s using the Open SSL version.

MD5_my8.vi 26.1 kB · 1 download

Some things that need to be done so it doesn't crash arbitrarily (not particularly you, just commenting on the latest incarnation).

EVP_DigestFinal_ex mdlen parameter needs to be pointer to value instead of value as it returns the length,

Return values need to be used (and checked). Currently the functions return Void when they should be I32.

Need to check the MD_CTX pointer isn't 0 for each function.

 

Edit.

almost forgot.

Remove EVP_cleanup(). In versions prior to 1.1.0 it will crash other functions that use EVP and after 1.1.0 it's a no-op.

Edited September 10, 2021 by ShaunR

[JKSH]

21 hours ago, hooovahh said:

the compatible binary is libeay32.so

That means it is using OpenSSL 1.0.x or earlier.

In OpenSSL 1.1.0, "libeay32" was renamed to "libcrypto"

[hooovahh]

On 9/11/2021 at 7:42 AM, JKSH said:

In OpenSSL 1.1.0, "libeay32" was renamed to "libcrypto"

Thanks for the info.  Looks like a wildcard can't be used to select the new binary based on the naming.  I could add code to say if we are on Linux, and the first call to "OpenSSL_add_all_digests" returns an error 7 (file not found) when using the path "libeay32.so" to the binary, then to try one more time using the path of "libcrypto.so".  I mention this method because I don't want to specify the full path, but the Check If File or Folder Exists doesn't search for a file in global path locations.  Not sure if there is a more elegant solution.

Speaking of hard coded paths the Windows method is obviously not ideal at the moment too since it is a hard codded full path.  What would be the best solution here?  Reading registry information on first call?

[hooovahh]

Okay attached is an updated version that I think supports the changes I made.  It reads the registry in 32 or 64 bit LabVIEW finding the path to the shared installation location.  In Linux it tries both the libeay32.so and libcrypto.so.  In both cases it keeps the found path in a feedback node for later calls.  I also added outputs instead of void for calls, but am now realizing this version doesn't do anything with them yet.

MD5_my9.vi

[Antoine Chalons]

Has anyone been able to make this work on Linux desktop?
I use LV 2020 SP1 (therefore 64 bit) on Ubuntu 20

I ran

Quote

apt-get update
apt-get install libssl-dev

to install libcrypto.so but at the first call of the so "OpenSSL_add_all_digests" I get error 15 "resource not found" and indeed if in the CLFN I put the full libcrypto.so path, the "OpenSSL_add_all_digests" is not in the "function" list

edit : I guess the answer is RTFM

With help from a co-worker we could make it work on Linux, multiple functions have different names, will post the VI soon.

 

Edited August 29, 2022 by Antoine Chalons

[Rolf Kalbermatter]

39 minutes ago, Antoine Chalons said:

Has anyone been able to make this work on Linux desktop?
I use LV 2020 SP1 (therefore 64 bit) on Ubuntu 20

I ran

to install libcrypto.so but at the first call of the so "OpenSSL_add_all_digests" I get error 15 "resource not found" and indeed if in the CLFN I put the full libcrypto.so path, the "OpenSSL_add_all_digests" is not in the "function" list

That's most likely because of this in the OpenSSL headers:

# if OPENSSL_API_COMPAT < 0x10100000L

These functions were required to be called in OpenSSL before 1.1.0but since 1.1.0 OpenSSL automatically initializes its engines on the first call of any function that creates a context of similar session.

And since 1.1.0 is already EOL too and you should either use 1.1.1 or even better 3.0.x, it would be indeed strange if your OpenSSL library still included those APIs. I think the old libssleay.so still contained it in 1.0.1 but it was unnecessary to call that, but when changing the shared library names they also axed many of those compatibility hacks too.

You should probably call

OPENSSL_init_crypto(OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS | OPENSSL_INIT_LOAD_CONFIG, NULL);

and yes this means reading the headers to see what numeric values those constant defines have.

 

Edited August 29, 2022 by Rolf Kalbermatter

[Antoine Chalons]

Just now, Rolf Kalbermatter said:

That's most likely because of this in the OpenSSL headers:

# if OPENSSL_API_COMPAT < 0x10100000L

These functions were required to be called in OpenSSL before 1.1.1 but since 1.1.1 OpenSSL automatically initializes its engines on the first call of any function that creates a context of similar session.

And since 1.1.0 is already EOL too and you should either use 1.1.1 or even better 3.0.x, it would be indeed strange if your OpenSSL library still included those APIs. I think the old libssleay.so still contained it in 1.0.1 but it was unnecessary to call that, but when changing the shared library names they also axed many of those compatibility hacks too.

You should probably call

OPENSSL_init_crypto(OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS | OPENSSL_INIT_LOAD_CONFIG, NULL);

 

and yes this means reading the headers to see what numeric values those constant defines have.

 

Your reply came as I was typing my edit... Thanks

[Antoine Chalons]

On 9/13/2021 at 4:59 PM, hooovahh said:

Okay attached is an updated version that I think supports the changes I made.  It reads the registry in 32 or 64 bit LabVIEW finding the path to the shared installation location.  In Linux it tries both the libeay32.so and libcrypto.so.  In both cases it keeps the found path in a feedback node for later calls.  I also added outputs instead of void for calls, but am now realizing this version doesn't do anything with them yet.

MD5_my9.vi 32.87 kB · 9 downloads

why on earth did you use a ring and a property node on that ring to get the method (md4, md5, etc.)?
an enum with a format into string seems nicer.

A, also, if your VI used on Linux and built as a shared library (in order to run the app as a service) then it causes a crash.
With enum+format into string, it works.

[hooovahh]

2 hours ago, Antoine Chalons said:

why on earth did you use a ring and a property node on that ring to get the method (md4, md5, etc.)?
an enum with a format into string seems nicer.

A, also, if your VI used on Linux and built as a shared library (in order to run the app as a service) then it causes a crash.

That is some ugly ass code for sure.  I'm fairly certain I didn't create that ring, and instead just copied it from some other example set of code.  I can never see myself center justifying a control like that so I'm guessing I just got it from something else, and then cut and pasted code until it worked.  Enum and format into string is the way to go.  That being said I'm pretty sure I would have tested this on a Linux RT machine and didn't see a crash at least running in source.

[ShaunR]

As this has popped up again and someone may find it useful. Here are some comparative benchmarks between LabVIEW and OpenSSL hashes (sorted fastest to slowest).