Formula node rounding issue

[Samapico]

First of all, I found a fix to this problem, but I'm just very curious on the reasons it could happen, so please don't reply with "just use XXX and it will work"

I had a very simple formula node that did this:

angle = acos(DP/(n1*n2));

In some cases, the result was NaN because it somehow calculated acos(1.000000001) or something, even though it was mathematically 1.0. DP is the dot product between V1 and V2, n1 and n2 are the length of V1 and V2. All these inputs are of type Double.

Doing the same thing with LabVIEW primitives gave the correct result (0) with the same inputs.

I tried two other things:

float64 angle;

angle = acos(DP/(n1*n2));

float64 angle;

float64 acosvalue;

acosvalue = DP/(n1*n2);

angle = acos(acosvalue);

Guess what... the first one gave NaN, and the second one gave 0. ??? Why?

This is using LV2009. I just tried it in 2010 and it works fine (both methods return 0). It's still pretty weird.

I attached a snippet, with the offending inputs as default value.

Edit: Just noticed... even without the 'is NaN?' check of the LV-primitive method, the result is correct. I just added it as extra precaution.

Also, here's a screenshot showing probes of the results:

Edited August 10, 2011 by Samapico

[darr]

The issue you are encountering here is not unique to LabVIEW as it involves the round-off error in the floating-point computations making up your example. Without the exact inputs values (DP, n1, n2), I can reliably reproduce your scenario but I can theorize as to why it's happening.

1. In your first example, the formula node is capable of storing the result of DP/(n1*n2) in an extended precision register. It is possible that the value in extended precision is slightly larger than 1 which is outside the real domain of [-1,1] for acos() which is what your surmised.

2. In your second example, the second version stores DP/(n1*n2) explicitly in a double before computing acos(). The additional round-off error of the computation in double can result in a value <= 1.

This would explain why the formula node results in 2009 are not the same between the versions.

As for your LV implementation in 2009, the compiler at that time did not keep the result of DP/(n1*n2) (i.e. the result on the wire) in an extended precision register so its behavior mimics the second version of example 2.

The consistent results you get in 2010 is a bit harder to explain but there were changes in the compiler and low-level math functions that impacted floating-point. Without the exact input values you were using, it's hard to say what exactly changed to produce the same results from both formula node versions.

[Samapico]

You can get the values I used with the VI snippet I posted, Vector 1 and Vector 2 have the same value and it's set as the default value.

I figured it was something about floating point calculations, but I wasn't aware the computations could go through extended precision registers, I thought 'double' was pretty much the most accurate it could get.

Thanks for the explanation.