Sign in to follow this  
Followers 0
CanaanP

REAL datatype math problem

2 posts in this topic

Hello, I am working on a project where I measure material with accuracy to the tenth of a mm. I also store a value of blade thickness that will cut that material that is input as mm.m If I have a piece of material that is 119.2, my blade thickness is 3.2, then I should wind up with 2 58mm pieces. However, after subtracting one value from another I wind up with 57.999996 as the value of my second piece. I had asked for advice from another source on how to handle this and they said to use an implied decimal. Meaning, use DINT as datatype and just add a zero to the end of each value, then have HMI put the decimal there. Is this really the best solution? Or is there another route I can take where my datatypes are the proper type and I can limit accuracy to a tenth?

Share this post


Link to post
Share on other sites
Of course the problem is that floating point can't represent the 3.2 exactly. Though it can represent the 119.2 exactly. Subtracting you get an inexact number close to 116.0 but not exactly. As long as your single decimal place is exact then using whole integers (multiplying the original number by 10) may be the best route. The HMI can supply the proper positioning of the decimal point.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0