Fract/Exp String To Number Issue

[Dave Graybeal]

Alright, the issue I'm seeing is that the Fract/Exp String To Number VI is incorrectly translating some strings to numbers. For instance the number 1.3 is being converted to the number 1.30000000000000004. Yes, it is placing a 4 at the 17th place past the decimal. This happens with a few numbers that I've noticed (different numbers being tacked on the end). Attached is a VI that demonstrates this error. If anyone can think of anyway to get around this please let me know. In the mean time I'll be on the phone with NI to see what the issue is and how to work around it till it is resolved.

Also, while I was getting the VI ready to attach to this post I noticed that even just basic math on numbers can create this same issue. 130 / 100 is comming out with the same results.

Thanks,

Dave Graybeal

Download File:post-4149-1146232154.vi

[jpdrolet]

See this FAQ

[Dave Graybeal]

wow, I'm actually a bit surprised that I never came across this issue earlier. Thank you for the link, it was very interesting material to read. I guess i'll have to get away from these types of habits.

Thanks again,

Dave