Missing external function in LV2019 64-bit executable

[Neil Pate]

Thanks Rolf, that sounds quite plausible.
AQ this is a brand new CPU with a Ryzen 3600 in it. I would consider changing the CPU if NI could give some guidance on what to change it to.

Is there any sort of hot fix or patch I can try?

For now I am out of trouble as I have rewritten the VIs that I needed to use from that library (mine were really simple) , but I doubt others will be so lucky.

 

[JKSH]

2 hours ago, Neil Pate said:

this is a brand new CPU with a Ryzen 3600 in it.

I don't know if this is related to your issue or not, but there are known problems with some brand new AMD Ryzen CPUs: https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd-microcode-bug-destroyed-my-weekend/

See if updating your BIOS helps at all. (Even if it doesn't solve this problem, I'd imagine you want a functional RNG on your PC!)

[Neil Pate]

Interesting, thanks JKSH I will see if there is a BIOS update.

[Rolf Kalbermatter]

9 hours ago, Aristos Queue said:

The MKL problem is in the wild? I thought that was something that was only affecting LV 2020 (now in beta) because we were updating to the latest MKL. The bug isn't on my team's plate... I just sit near the people who are handwringing a lot about it. If that's in the wild affecting already shipping MKL versions, then, yeah, that could be it. I still don't know how the node is rolling back to its last known good resolve path... I can't find that path stored anywhere... but if we assume it is binary encoded *somewhere* in the node's saved attributes, then this makes plausible sense. 

Workaround is to a) get a new CPU or b) wait until LV issues a new version where we do whatever we are doing to avoid calling certain CPU instructions (I'm unclear what the planned solution looks like). 

I definitely saw somewhere discussions about this that were not LabVIEW 2020 related. The solution was to create(edit) some environment variable for the MKL library itself where some thread configurations were forced explicitedly rather than letting MKL detect the right configuration automatically.

See this thread for some discussion of the problem and possible solutions. It seems to be related to latest AMD Ryzen CPUs with a specific SSE architecture.

That it falls back to trying to load from the penguin path is however rather strange. Generally these paths only exist in the executable as debug messages related to the source module the specific code was compiled from.

Edited March 23, 2020 by Rolf Kalbermatter

[Neil Pate]

Thanks Rolf, that fix worked for me too! 👍🙏🙇‍♂️

I did search but never found that thread.

Phew, mystery solved!

 

[Aristos Queue]

14 hours ago, Neil Pate said:

Phew, mystery solved!

For anyone else interested in how far this bug goes, I got this from one of my coworkers:

From my research it affects all processors based on the Zen 2 architecture (AMD processors that started being released in July of last year). AMD claimed that they dropped support in Zen 1 for fma4 instruction set, and the illegal instruction causing the crash is part of that set. The first series Ryzen processors sound like they may still work, but I didn't have access to any to verify one way or the other.

[Lipko]

I have the same problem on random machines (the same type though), though the application is built in LV 2015 (32bit), at it was built months ago, not nwelí built.
I figure setting enviromental value means setting it in windows.
That didn't solve the problem.
I will replace all Mean functions, but meh....

EDIT: Sorry for bumping, I didn't realize it's not 2020 any more...
Anyways, replacing the Mean function solved the issue. Another argument for reinventing everything for yourself 😋
 

Edited March 11, 2021 by Lipko

[X___]

On 3/22/2020 at 9:52 AM, Rolf Kalbermatter said:

lvanlys.dll is for the most part a thin wrapper around the Intel Math Kernel (MKL) library since many moons. It used to be a fully NI private implementation of the analysis functions but at some point NI realized that they never can beat the guys from Intel in making hyper-optimalized versions of those functions that work virtually on any CPU from early Pentiums to the latest iCore monsters with optimal performance (and also work on AMD CPUs as well).

Kind of out of topic, but if confirmed, that transposes the source code source from NI to Intel... MKL forums seem to be indicating that obtaining source code from Intel is a complex and costly proposition.

I wonder whether this is why it took 2 years to fix a numerical bug in LabVIEW (i.e. this was out of the hands of NI and had to be fixed by Intel): https://forums.ni.com/t5/LabVIEW/Bug-in-the-Kummer-Function/m-p/2387522

[Rolf Kalbermatter]

3 hours ago, X___ said:

Kind of out of topic, but if confirmed, that transposes the source code source from NI to Intel... MKL forums seem to be indicating that obtaining source code from Intel is a complex and costly proposition.

I wonder whether this is why it took 2 years to fix a numerical bug in LabVIEW (i.e. this was out of the hands of NI and had to be fixed by Intel): https://forums.ni.com/t5/LabVIEW/Bug-in-the-Kummer-Function/m-p/2387522

I'm not exactly privy to the details but most likely NI doesn't even build the MKL themselves. They simply take the binaries as released from Intel and package them with their LabVIEW wrapper and be done with it. And there are a number of issues with this that way:

1) NI can indeed not patch that library themselves anymore but has to wait on Intel to make bug fixes.

2) And NI won't pick a new release everytime Intel decides to make some more or less relevant change to that library. Instead they will likely review the list of changes since the last pick they did and decide if it is worth the hassle to rebuild a new MKL + LabVIEW package. This is not a one hour process of just adding the new DLLs to the old package build but instead involves a lot of extra work in terms of making sure everything is correct and lots and lots of testing too. The moment for such a review is likely usually a few months before a new release of LabVIEW. IF Intel happens to make this one single important change one month after this, NI will most likely not pick it up until the next review moment a few months before the next full LabVIEW release and then you can easily see how it can take 2 years.

[Lipko]

I wonder if that mean function has advantages over a self implementation apart from returning some error codes and (I guess) no float overflow errors.

[Neil Pate]

So I took the the same route initially and just implemented my own Mean.vi as it was trivial.

Then something else broke and I realised it would be a game of whack-a-mole I could not win so went to solve the actual problem.

[Lipko]

55 minutes ago, Neil Pate said:

So I took the the same route initially and just implemented my own Mean.vi as it was trivial.

Then something else broke and I realised it would be a game of whack-a-mole I could not win so went to solve the actual problem.

By "then something else broke" what kind off breaking do you mean?
If replacing that Mean with my own solved the problem for me (no .dll error, the exe runs without obvious strange behavior), can I feel safe?

My role is not a programmer, but a mere test engineer who makes tools for making test processes more convenient. So I don't really have the time to dig too deep into stuff like this, I need to have solutions for problems ASAP. Solving actual problems is not an easy task with a soo rigid IT department over me. I was surprised that I could add that environment value (which didn't solve the issue...)

Edited March 12, 2021 by Lipko

[Neil Pate]

1 hour ago, Lipko said:

By "then something else broke" what kind off breaking do you mean?
If replacing that Mean with my own solved the problem for me (no .dll error, the exe runs without obvious strange behavior), can I feel safe?

My role is not a programmer, but a mere test engineer who makes tools for making test processes more convenient. So I don't really have the time to dig too deep into stuff like this, I need to have solutions for problems ASAP. Solving actual problems is not an easy task with a soo rigid IT department over me. I was surprised that I could add that environment value (which didn't solve the issue...)

I mean I needed some other analysis functions and these were also broken. At that point I was worried I would hit something I would not be able to recreate myself. The environment variable solved it straight away for me so I am not sure what is going on with your PC.

 

[X___]

21 hours ago, Rolf Kalbermatter said:

I'm not exactly privy to the details but most likely NI doesn't even build the MKL themselves. They simply take the binaries as released from Intel and package them with their LabVIEW wrapper and be done with it. And there are a number of issues with this that way:

1) NI can indeed not patch that library themselves anymore but has to wait on Intel to make bug fixes.

2) And NI won't pick a new release everytime Intel decides to make some more or less relevant change to that library. Instead they will likely review the list of changes since the last pick they did and decide if it is worth the hassle to rebuild a new MKL + LabVIEW package. This is not a one hour process of just adding the new DLLs to the old package build but instead involves a lot of extra work in terms of making sure everything is correct and lots and lots of testing too. The moment for such a review is likely usually a few months before a new release of LabVIEW. IF Intel happens to make this one single important change one month after this, NI will most likely not pick it up until the next review moment a few months before the next full LabVIEW release and then you can easily see how it can take 2 years.

<OOT>

In the case of the Kummer function (which I was referring to above), this appears to be an internal algorithm as the function is called LV_Kummer (there are a number of similar LV_functions in the lvanlys.dll).

For the purely MKL ones, I guess we have to check the bug fixes here (without knowing which one is relevant for which LV version...): MKL Bug Fixes

</OOT>

Edited March 12, 2021 by X___

[Rolf Kalbermatter]

17 hours ago, X___ said:

In the case of the Kummer function (which I was referring to above), this appears to be an internal algorithm as the function is called LV_Kummer (there are a number of similar LV_functions in the lvanlys.dll).

For the purely MKL ones, I guess we have to check the bug fixes here (without knowing which one is relevant for which LV version...): MKL Bug Fixes

You might be right but please note that those LabVIEW VIs more or less all call "internal" functions in lvanlys.dll. But in reality all they do is massage the parameters from a LabVIEW friendly format into an Intel MKL C(++) API format and then call the actual MKL library. So the fact that a VI calls LV_something in that DLL means absolutely nothing in terms of if it is ultimately executed inside lvanlys.dll or actually simply forwarded to MKL to do the heavy number crunching part. It could be implemented fully in lvanlys.dll, because the MKL doesn't provide this function or not in the way the old NI library did, so for compatibility reason they maintained the old code but in most cases it is simply a forward to the MKL with minor parameter datatype translations.

Even if there is some real implementation part in lvanlys.dll for a function, it still will very often ultimately call the MKL for lower level functions so may still depend on a corrected MKL to fix a bug.

Edited March 13, 2021 by Rolf Kalbermatter

[X___]

I am not sure there is any special functions in the MKL (at least I could not find any in the description) and all these are LV_something in the DLL call. There is really no wrapping to be done here, as the arguments are only a few.

So maybe after all NI did code those (sometimes using lousy algorithms, as was the case for the Kummer function)...

[Rolf Kalbermatter]

On 3/13/2021 at 9:21 PM, X___ said:

I am not sure there is any special functions in the MKL (at least I could not find any in the description) and all these are LV_something in the DLL call. There is really no wrapping to be done here, as the arguments are only a few.

So maybe after all NI did code those (sometimes using lousy algorithms, as was the case for the Kummer function)...

The wrapping may be done anyways even if it is a clean pass through of all parameters. Simply because this was how the VIs always worked and it was actually easier. The lvanlys.dll had to be modified anyways, so just leave the original exports and redirect them wherever necessary to MKL, with or without any parameter massaging. This makes the tedious work of going into every single LabVIEW VI to edit the Call Library Node superfluous. And yes I have experience with wrapping DLLs from LabVIEW and can assure you that the last thing you want to do when changing something is to make a change that will require you to go into every single VI and make some more or less minor changes. Aside from being a mind numbing job, it is also VERY VERY easy to make stupid mistakes in such changes by forgetting a certain change in some of the VIs. And then you have to open each and every VI again to make sure that you did really change everything correctly, and to be safe, do that again and again. Tedious, painful and utterly unnecessary. Instead just leave the VIs alone, change the underlying DLL in whatever way you need and you are done. There is still a lot of testing after that, but at least one potential source of errors less.

Edited March 16, 2021 by Rolf Kalbermatter

[X___]

On 3/16/2021 at 6:57 AM, Rolf Kalbermatter said:

The wrapping may be done anyways even if it is a clean pass through of all parameters. Simply because this was how the VIs always worked and it was actually easier. The lvanlys.dll had to be modified anyways, so just leave the original exports and redirect them wherever necessary to MKL, with or without any parameter massaging. This makes the tedious work of going into every single LabVIEW VI to edit the Call Library Node superfluous. And yes I have experience with wrapping DLLs from LabVIEW and can assure you that the last thing you want to do when changing something is to make a change that will require you to go into every single VI and make some more or less minor changes. Aside from being a mind numbing job, it is also VERY VERY easy to make stupid mistakes in such changes by forgetting a certain change in some of the VIs. And then you have to open each and every VI again to make sure that you did really change everything correctly, and to be safe, do that again and again. Tedious, painful and utterly unnecessary. Instead just leave the VIs alone, change the underlying DLL in whatever way you need and you are done. There is still a lot of testing after that, but at least one potential source of errors less.

By "special functions", I mean "Special" functions such as Bessel, which can be found in the non-yellow subpalettes of the Elementary & Special Functions of the Mathematics palette. I am under the impression that those are not part of the MKL.