Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[yugabyte#9867]: YSQL: Fix double type overflow in case of SET yb_tra…
…nsaction_priority_lower_bound/yb_transaction_priority_upperr_bound command Summary: YB transaction priority is stored in uint64_t variable. the `SET yb_transaction_priority_lower_bound/yb_transaction_priority_upperr_bound` command uses `double` to store factor (0..1) for priority bounds calculation. `double` has the following limitations for representing integers ``` Precision limitations on integer values: - Integers from −2^53 to 2^53 (−9,007,199,254,740,992 to 9,007,199,254,740,992) can be exactly represented - Integers between 2^53 and 2^54 = 18,014,398,509,481,984 round to a multiple of 2 (even number) -Integers between 2^54 and 2^55 = 36,028,797,018,963,968 round to a multiple of 4 ``` Due to this limitations the following statement causes overflow ``` return minValue + value * static_cast<double>(maxValue - minValue); ``` Solutions is to use `long double` instead of `double` because it can exactly represents int up to `2^113` Test Plan: Run existing unit test multiple times ``` ./yb_build.sh --cxx-test pgwrapper_pg_libpq-test --gtest_filter PgLibPqTest.MultiBankAccountSerializable -n 20 ``` Reviewers: alex Reviewed By: alex Subscribers: yql Differential Revision: https://phabricator.dev.yugabyte.com/D12789
- Loading branch information