The issue you are encountering here is not unique to LabVIEW as it involves the round-off error in the floating-point computations making up your example. Without the exact inputs values (DP, n1, n2), I can reliably reproduce your scenario but I can theorize as to why it's happening.
1. In your first example, the formula node is capable of storing the result of DP/(n1*n2) in an extended precision register. It is possible that the value in extended precision is slightly larger than 1 which is outside the real domain of [-1,1] for acos() which is what your surmised.
2. In your second example, the second version stores DP/(n1*n2) explicitly in a double before computing acos(). The additional round-off error of the computation in double can result in a value <= 1.
This would explain why the formula node results in 2009 are not the same between the versions.
As for your LV implementation in 2009, the compiler at that time did not keep the result of DP/(n1*n2) (i.e. the result on the wire) in an extended precision register so its behavior mimics the second version of example 2.
The consistent results you get in 2010 is a bit harder to explain but there were changes in the compiler and low-level math functions that impacted floating-point. Without the exact input values you were using, it's hard to say what exactly changed to produce the same results from both formula node versions.
