Jump to content

Simple number conversion , precision error!


micman2

Recommended Posts

Hi all,

I'm new user.

I have a simple number that ranges from 0 to 10 with 0.01 accuracy, this issue must send in the serial in the following format 00.00 :

Example :

#15.00

#02.10

#00.47

Now when I try to convert in any way the decimal number I have a problem, when a number varies between 4.49 example, I take 4 (ok) but varies 4.51 when I take 5

Sending screen

Why?

post-10885-1204032116.gif?width=400

post-10885-1204032131.gif?width=400

Download File:post-10885-1204032162.vi

Link to comment

You are using three different data types... Your slider is SGL-precision, your indicator DBL-precision and your conversion to string uses Integer...

Try to match your datatypes (right-click: representation) to get consistent results or to use "Round to -Infinity" to get 3,51 to be rounded to 3. It is definitely not a precision error to round 3,51 to 4.

Link to comment

If you are converting from string, then you need to take the decimal point character into consideration. You seem to have a comma set as the decimal point (3,51 not 3.51), so you need to let your "Scan from string" function know this is being used, not a ".".

You can do this with "%,;#%05.2f". The %,; tells the interpreter to use the "," as the decimal seperator.

Shane

PS Scratch that, it seems that what normandinf says is correct.

Link to comment

QUOTE(shoneill @ Feb 26 2008, 04:12 PM)

If you are converting from string, then you need to take the decimal point character into consideration. You seem to have a comma set as the decimal point (3,51 not 3.51), so you need to let your "Scan from string" function know this is being used, not a ".".

You can do this with "%,;#%05.2f". The %,; tells the interpreter to use the "," as the decimal seperator.

Shane

PS Scratch that, it seems that what normandinf says is correct.

No work :unsure:

Link to comment

QUOTE(BrokenArrow @ Feb 26 2008, 08:16 PM)

Isn't the comma thing an artifact of the nationality of his LabVIEW version, and not so much a format deal?

OK, People, please forget I even posted here. I live in Switzerland, and I get confromted with the comma, point argument quite a lot, so this is why I thought the problem had to do with that. It doesn't. If it did, a 3.51 would NOT be rounded to 4, it would be rounded to 3.

As to the "format deal" I don't really now what you're referring to, but I meant that if scanning from string, one must let LV know what type of decimal seperator we are using, otherwise it gets all confused.

Shane.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.