[teiid-issues] [JBoss JIRA] Resolved: (TEIID-159) Modify our handling of BigDecimal division to use MathContext.DECIMAL128 as our rounding mode to improve scale and precision of results
[
https://jira.jboss.org/browse/TEIID-159?page=com.atlassian.jira.plugin.sy...
]
Steve Hawkins resolved TEIID-159.
---------------------------------
Resolution: Done
Updated the floating to bigdecimal conversion to use a scale of 16 for doubles and 8 for
floats. Updated the logic in the bigdecimal x/y to use a scale of max(16, x.scale +
y.precision + 1), then the result is normalized to remove trailing zeros.
Modify our handling of BigDecimal division to use
MathContext.DECIMAL128 as our rounding mode to improve scale and precision of results
---------------------------------------------------------------------------------------------------------------------------------------
Key: TEIID-159
URL: https://jira.jboss.org/browse/TEIID-159
Project: Teiid
Issue Type: Feature Request
Components: Server
Affects Versions: 6.0.0
Reporter: Jerry Helbling
Fix For: 7.0
Using MathContext.DECIMAL128 should give a more usable scale and precision to the
results.
Current behavior
254403453878 / 21870000000 returns 12
and
254403453878.0001 / 21870000000 returns 11.6325
Improved behavior:
254403453878 / 21870000000 returns 11.632531
and
254403453878.0001 / 21870000000 returns 11.632531
We should also investigate what the impact of this change will be on users who may be
relying on our current behavior.
Reference: This enhancement was suggested by Issue Tracker Issue 175485, contributed by
Greg Haber
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira