Simple number conversion , precision error!

[micman2]

Hi all,

I'm new user.

I have a simple number that ranges from 0 to 10 with 0.01 accuracy, this issue must send in the serial in the following format 00.00 :

Example :

#15.00

#02.10

#00.47

Now when I try to convert in any way the decimal number I have a problem, when a number varies between 4.49 example, I take 4 (ok) but varies 4.51 when I take 5

Sending screen

Why?

Download File:post-10885-1204032162.vi

[Francois Normandin]

You are using three different data types... Your slider is SGL-precision, your indicator DBL-precision and your conversion to string uses Integer...

Try to match your datatypes (right-click: representation) to get consistent results or to use "Round to -Infinity" to get 3,51 to be rounded to 3. It is definitely not a precision error to round 3,51 to 4.

[Phillip Brooks]

Try replacing the "Number to Decimal String" function with the "Format Into String" function.

Use a format string of "#%05.2f".

[shoneill]

If you are converting from string, then you need to take the decimal point character into consideration. You seem to have a comma set as the decimal point (3,51 not 3.51), so you need to let your "Scan from string" function know this is being used, not a ".".

You can do this with "%,;#%05.2f". The %,; tells the interpreter to use the "," as the decimal seperator.

Shane

PS Scratch that, it seems that what normandinf says is correct.

[micman2]

QUOTE(shoneill @ Feb 26 2008, 04:12 PM)

If you are converting from string, then you need to take the decimal point character into consideration. You seem to have a comma set as the decimal point (3,51 not 3.51), so you need to let your "Scan from string" function know this is being used, not a ".".

You can do this with "%,;#%05.2f". The %,; tells the interpreter to use the "," as the decimal seperator.

Shane

PS Scratch that, it seems that what normandinf says is correct.

No work

[BrokenArrow]

Pick one.

-Richard

[shoneill]

QUOTE(micman2 @ Feb 26 2008, 04:59 PM)

No work

Er yeah, like I edited my text: My interpretation of your original question was incorrect.

QUOTE

PS Scratch that, it seems that what

normandinf

says is correct.

Your problem lies elsewhere.

Shane.

[BrokenArrow]

Isn't the comma thing an artifact of the nationality of his LabVIEW version, and not so much a format deal?

[shoneill]

QUOTE(BrokenArrow @ Feb 26 2008, 08:16 PM)

Isn't the comma thing an artifact of the nationality of his LabVIEW version, and not so much a format deal?

OK, People, please forget I even posted here. I live in Switzerland, and I get confromted with the comma, point argument quite a lot, so this is why I thought the problem had to do with that. It doesn't. If it did, a 3.51 would NOT be rounded to 4, it would be rounded to 3.

As to the "format deal" I don't really now what you're referring to, but I meant that if scanning from string, one must let LV know what type of decimal seperator we are using, otherwise it gets all confused.

Shane.

[BrokenArrow]

Dude chill. :beer: