Expected zero isn't

[Graeme]

Hi all,

Stumbled across this today in a larger application that I'm working on. Have stripped out the core and put it in the attached very simple VI. The VI takes a double precision number (which should be a multiple of 0.1) and, via a While loop with a shift register, successively subtracts 0.1 (also a double precision number) from it. When the result is zero the While loop should stop, except it doesn't: the result is never zero, only very nearly, e.g. 1.23456789E-16, or something similar.

I'm a bit baffled. As ever, you can work around it, but I'd like to understand what's going on. If anyone can enlighten me I'd appreciate it.

Regards.

[Ton Plomp]

You are right, this will (most likely) never work. Since a double is a floating point binary representation of a decimal number you always get rounding errors.

If you want to do something like this use integers!

Get your DBL multiply it by 10 and use round to nearest and then convert it to I16. Now do in every iteration a subtraction of 1 (one) and you will hit 0!

Ton

[PaulG.]

QUOTE(Graeme @ Apr 11 2007, 05:32 PM)

Hi all,

Stumbled across this today in a larger application that I'm working on. Have stripped out the core and put it in the attached very simple VI. The VI takes a double precision number (which should be a multiple of 0.1) and, via a While loop with a shift register, successively subtracts 0.1 (also a double precision number) from it. When the result is zero the While loop should stop, except it doesn't: the result is never zero, only very nearly, e.g. 1.23456789E-16, or something similar.

I'm a bit baffled. As ever, you can work around it, but I'd like to understand what's going on. If anyone can enlighten me I'd appreciate it.

Regards.

I saw this quite a few years ago. If you extend your significant digits out to about 18 you'll see where the "error" is coming from, but it's not really an error. It has something to do with the way your PC deals with dbl's. It can only handle so many bits.

[syrus]

Take a look at this article about Comparing Floating Point Numbers.

[Michael Aivaliotis]

[wiki]Numeric[/wiki]

[Graeme]

As ever, many thanks to those that replied. Well, you learn something every day, don't you! Thanks for your comments and links. Reading these it all seems rather obvious now, but it never crosses your (well, my) mind initially. So, unless I've misread the links, floating point maths is fine but, be careful comparing such numbers.

Regards, Graeme.