DaveX Posted July 15, 2009 Report Share Posted July 15, 2009 Hi all, My problem with Flattern to XML is rounding, see following example. DBL 0.123456789 result: <DBL> <Name>Value</Name> <Val>0.12346</Val> </DBL> Is there any way to set precision of conversion? BTW conversion back (unflattern) works just fine. Thanks&Regards, David Quote Link to comment
PaulG. Posted July 15, 2009 Report Share Posted July 15, 2009 As far as I know that is normal behavior. Try this workaround. Quote Link to comment
Mark Smith Posted July 15, 2009 Report Share Posted July 15, 2009 Hi all, My problem with Flattern to XML is rounding, see following example. DBL 0.123456789 result: <DBL> <Name>Value</Name> <Val>0.12346</Val> </DBL> Is there any way to set precision of conversion? BTW conversion back (unflattern) works just fine. Thanks&Regards, David I just tried this on my system (LV 8.6.1) and passed a double control into the Flatten To XML function and got <DBL> <Name>Value</Name> <Val>0.12345678900000</Val> </DBL> which preserves the precision of the double - In LV 8.5.1 I get the results you describe. I thought I remembered something about Flatten To XML having some bugs fixed lately. So, if you can, upgrade to 8.6 Mark Quote Link to comment
DaveX Posted July 15, 2009 Author Report Share Posted July 15, 2009 I'm still in lv8.5 :-( but we have available lv8.6 as well, so it time to move.... Thanks, David. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.