Repository navigation
Migrate Number from float64 to big.Float #191
Description
Currently, Number always converts its value to float64. In many cases it works well, however float64 can't represent big integer values (larger than 2^62) without precision loss.
Actually, there is no native type that can represent ALL values of int64, uint64, and float64 without loss. However, there is a library type that can do it: big.Float (see https://pkg.go.dev/math/big).
Here are the steps:
-
replace underlying value of Number from float64 to big.Float
-
by default, use precision that is sufficient to represent all values of int64, uint64, and float64 without loss
-
allow user to set higher precision in Config (this may be useful if user wants to parse Number from String that contains bigger integer or float)
-
-
rework canonNumber() to return big.Float instead of float64; it should automatically detect precision and create big.Float from any of these types:
- all native float and integer types
- type definitions for those native types (e.g.
type MyInt int) - big.Float and big.Int
- json.Number
-
rework NewNumber constructor: it should accept interface{} instead of float64 and use canonNumber() to convert it to big.Float
-
rework all Number operations, so that they will operate on big.Float values (one stored in Number, another returned from canonNumber for operation argument)
-
rework Number.EqualDelta and Number.NotEqualDelta: they should accept interface{} instead of float64; this will allow user to pass big.Float instead of float64, when necessary
-
rework JSON decoding:
-
find places where we decode JSON: canonValue, Response, WebsocketMessage; configure JSON decoder to use json.Number instead of float64 (see https://pkg.go.dev/encoding/json#Decoder.UseNumber)
-
ensure that canonValue() and other canonization functions does not cause precision loss
-
-
rework Number creation
-
Expect.Number() should accept interface{} instead of float64
-
rework Value.Number() - it should not force float64 type
-
rework String.AsNumber() - it should correctly parse big floats and integers
-
find all invocations to newNumber() that have unnecessary casts to float64 and pass integers instead; e.g.
newNumber(a.chain, float64(len(a.value)))can be now replaced withnewNumber(a.chain, len(a.value))because Number now supports integers
-
-
adjust equality checks:
-
find all places that do deep equality checks (e.g. in Object, Array) using reflect.DeepEqual
-
ensure that they will work when original values have different representations for same numbers (e.g. json.Number vs float64 vs big.Float)
-
cover these cases with unit tests
-
-
add new getters for Number:
-
add 4 new methods: AsInt (returns int64), AsUint (return uint64), AsFloat (return float64), AsBig (return big.Float)
-
each method should report failure if underlying value can't be converted to requested types without loss
-
add IsBig (similar to other Is methods, see Implement Number IsInt, IsUint, IsFloat, IsNan #155)
-
mark Number.Raw as deprecated and ask user to use AsFloat instead; in httpexpect v3, when we will be able to break compatibility, we'll undeprecate Raw, but change its return type to big.Float
-
-
adjust DefaultFormatter to handle big.Float values:
-
if value is integer, print it as integer
-
otherwise print it in non-scientific form (see also Add DefaultFormatter.EnableScientific #190)
-
-
adjust tests:
-
add unit tests for Number to check how it works with big integers and floats
-
add unit tests for Value.Number() to check how it works with json.Number and integers that can't be represented without loss as float64
-
rework tests for String.AsNumber() - rework float_precision test, add tests for big floats and integers that can't be parsed with default precision, but can be parsed when setting higher precision in Config
-
cover big integers and floats, and different precisions, in e2e tests (e.g. in e2e_basic_test.go)
-
Activity
angshumanHalder commentedon Jan 10, 2023
Hey @gavv can assign this to me? I am little busy with switching jobs but this problem seems interesting. Would pickup as soon as I get some time.
gavv commentedon Jan 12, 2023
Sure, thanks!
angshumanHalder commentedon Jan 13, 2023
Hey @gavv converting
to
will cause
Raw()to fail. As it returns float64. The code won't compile unless we comment out the code instead of deprecating it or we can modify it to return float64 by converting it from big.Float. But if we decide to make changes on the function then we don't need to deprecate it. And AsFloat and Raw would be the same exact function. How do you suggest we proceed?gavv commentedon Jan 13, 2023
Yes, that's what I was suggesting.
Yes.
Yeah, sorry, I was not clear on this. The idea is the following:
This is actually a common approach for breaking APIs: deprecate, wait, then remove or change. Wait period gives users time to reacts to the change without hurry.
angshumanHalder commentedon Jan 30, 2023
Hey @gavv I have few questions.
newNumber()should those be replaced too?find places where we decode JSON: canonValue, Response, WebsocketMessage; configure JSON decoder to use json.Number instead of float64 (see https://pkg.go.dev/encoding/json#Decoder.UseNumber)
ensure that canonValue() and other canonization functions does not cause precision loss
For ☝️ do you suggest to have a type assertion wherever the functions are called? Because as far as I can see. All of the canon functions are using interfaces to decode json rather than any specific type.
gavv commentedon Jan 30, 2023
Yep
Sorry, I don't understand the question/suggestion, could you elaborate?
angshumanHalder commentedon Jan 31, 2023
So let's suppose we have this function -
How do you want it to be modified to use json.Number? Because from what I can see we are using
interface{}as input.Or Am I understanding this wrong? If so can you please explain this?
gavv commentedon Feb 1, 2023
Ah, I see.
canonValue and other methods use json.Unmarshal to unmarshal object, array, etc. When json.Unmarshal encounters a number somewhere in json, it unmarshals it into float64. This may cause precision loss if the number can't be represented as float without loss (if it's a large 64-bit integer for example).
encoding/jsonprovides an option to unmarshal numbers into json.Number instead of float64. (This need using json.Decoder instead of json.Unmarshal).We need to use this option everywhere when we unmarshal json, to avoid precision loss. Our maps, arrays, etc. should have json.Number instead of float64.
When the user will create
Numberfor such element, it will construct big.Float from json.Number.angshumanHalder commentedon Feb 3, 2023
Understood. Thanks. 😄
gavv commentedon Feb 4, 2023
Some thoughts:
To avoid code duplication, we need a function (I think in canon.go) that performs decoding with right options. We'll need to use it everywhere where we used json.Unmarshal before.
If you wish, you can think about splitting your PR into several, so that we can merge some of them earlier, to avoid accumulating of conflicts and unnecessary work on resolving them. For example:
You could create a PR that just switches Number to big.Float internally, but does not change Number interface: constructor will still accept float, Raw will still returns float, methods like InDelta still accept float, etc etc. We can quickly review and merge this, and you can continue work on other parts of the code without gaining conflicts on this part.
Another example: you could create a PR that changes all calls to json.Unmarshal to calls to a function in canon.go, but without changing behavior (it can still use json.Unmarshal or Decoder without options). In future PRs you can change this function to use json.Number when the code base will be ready for it.
Another example: DefaultFormatter will need adjustments to support big floats and json numbers. This can also be done in a separate PR.
2 remaining items
angshumanHalder commentedon Feb 5, 2023
But I already started making those fixes. 😅
gavv commentedon Feb 5, 2023
Oh I see, great :)
angshumanHalder commentedon Feb 11, 2023
Hey @gavv
Test cases are failing in
number_test.go. For exampleBelow is the output of test case.
The error is caused because
big.Floatis not parsed correctly tojsonincanonDecodemethod, since all the numbers are right now converted tobig.Float. Any idea how to decodebig.Floatorbig.Inttojson?gavv commentedon Feb 18, 2023
I think you need to convert big.Float to json.Number and then pass it to json decoder.
angshumanHalder commentedon Feb 18, 2023
I don't think we can directly convert big.Float to json.Number because big.Float is a struct. Rather I was using a reverse conversion and then passing it to json decoder. Check out this snippet -
gavv commentedon Feb 18, 2023
This would work too... does it handle wrapped types, e.g.
type myInt int?I'd rather use json.Number + json.Unmarshal because it's less code and we can be sure that we didn't miss any details.
json.Number is just a wrapped string, so "conversion" is something like
json.Number(value.String()). Though, we should check that it never loses precision and how it handles big numbers.angshumanHalder commentedon Feb 19, 2023
t := reflect.Indirect(reflect.ValueOf(target)).Kind()this takes care of wrapped types.The problem with using json.Number and json.Unmarshal is json.Number uses string. But the target can be of any type. So if an user wants to convert big.Float to int.
Decodefunction will fail -error: json: cannot unmarshal string into Go value of type int. Which is why thecanonNumberDecodefunction is required.angshumanHalder commentedon Feb 23, 2023
Hey @gavv there's one more issue. There is no NaN representation in big.Float/big.Int . There is only a type
ErrNaN(https://pkg.go.dev/math/big#ErrNaN) that panics if the input is NaN. How do you suggest we go about this issue? For example -The above snippet in
TestNumber_EqualNaNfails. Although I was thinking to use big.NewFloat(0) as a value but thenthis portion of the test case fails. Can I use
-InfasNaNrepresentation inbig.Float? If yes, then I have to write couple of extra checks on each method to see if the number is-Infwhich in this case would beNaNvalue representation inbig.Float.gavv commentedon Feb 27, 2023
Oh, cool.
Sorry, I don't know why I mentioned json.Number at all, it has nothing to do with this.
What I was trying to say is that we can convert big.Float to string and use stock json decoder, like this:
https://go.dev/play/p/uyl7R19QBYa
Can this work for us?
gavv commentedon Feb 27, 2023
Regarding NaN.
We can't use any other floating point value to represent NaN: zero, -inf, +inf - all are valid values that can be passed by user and we should not confuse them with NaN.
The only option we have is to handle NaN specially.
Since it seems that NaN is the only unsupported value (both -Inf and +Inf are seem to be supported), I think the easiest way to handle it is to use
*big.Floatinstead ofbig.Float, and usenilto representNaN.Then we'll need to check for nil in all methods. And I guess we'll need to add NaN tests for all methods.
angshumanHalder commentedon Mar 10, 2023
Hey @gavv#6022 I don't think our approach is right. For example :-
Look at the test function -
Check the var
target- it can either beinterface{}or[]intreface{}. Moreover the array can have structure as[]interface{}{}or[]interface{}{map[string]{}, {}interface{}, []interface{}}.Now when we decode the function will internally call
To handle the issue I wrote something like
But ☝️ will only work when the target is
{}interfaceanda.valueis[]interface{}Otherwise it will fail.Now, I don't see any way to convert json.Number back to either float64 or big.Float without manually checking for all the cases where the structure may vary.
How do you suggest we keep check for all these scenarios with too many variable structure?
Moreover this will introduce inconsistency. What do you think?
gavv commentedon Mar 23, 2023
Hi, and sorry for delay.
When we were discussing this in chat, my thoughts were the following (I think I didn't explain all of the idea accurately enough):
when we decode json from http response (in Response.JSON), we ask decoder to use json.Number; we'll get a map or slice or whatever object that occasionally may have json.Numbers inside
after decoding, we perform canonization (in Value constructor we call canonValue; we also call canon funcs in all other constructors);
during canonization, as suggested, we should replace json.Number with either float64 (if it's small) or big.Float (if it's large), hence preserving compatibility in cases when we were working correctly before; to achieve this, we should recursively inspect the object (map, slice, map of maps, whatever), find all json.Number instances, and convert them (I guess using reflect)
as discussed, the behavior of the previous step is controlled by an option; user can choose: always use float64 (will be default in v2), always use big.Float, automatic selection based on needed precision (will be default in v3)
after canonization, Value (and other structs like Object, Array, etc; everything except Number) will internally represent numbers as float64 or big.Float
when user calls Raw (e.g. Value.Raw, Object.Raw), they will get objects (maps, slices, etc) with numbers as float64 or big.Float; similarly, when user calls operations like IsEqual(), the operations will work with same representation of numbers
when user calls Decode (e.g. Value.Decode), it internally calls canonDecode; here we need special handling: we should again recursively find all numbers, convert them to json.Number, and the decode value into user's target using json decoder;
when user will eventually construct Number (e.g. call Value.Number() or Value.Array().Value(0).Number(), etc), Number constructor will receive interface{} for the numeric value
Number constructor will inspect received interface{}; it will be either float64 or big.Float; however Number will unconditionally convert it to big.Float, if it's not NaN, or to nil if it's NaN
all Number operations will work with big.Float that is stored internally; from user arguments, these operations (e.g. IsEqual) will accept interface{} that can be integer, float, or big.Float
Number.Raw will still return float64 for compatibility (and will be deprecated), however in v3 we'll change it to return big.Float (and will undeprecate it)
A lot of bullet points... but I tried to give comprehensive overview this time. In short, there are actually only a few key points:
during decoding from HTTP, use json.Number (as you already do)
during canonization, everywhere except Number, recursively find json.Number and covert to float64 or big.Float (new feature)
during decoding (in canonDecode), recursively find float64 and big.Float and covert to json.Number, then encode to json and decode into target (new feature)
during canonization in Number constructor, accept both float64 and big.Float, and convert to big.Float (as you already do)
And that's it.
Does this solve the problem you mentioned?
BTW I want to thank you for patience and all these long discussions on discord. I was not expecting all the obstacles when creating the task, and I think these discussions have generated a rather good solution.
angshumanHalder commentedon Mar 25, 2023
Now I understand. I although have an issue with the overhead of conversion. But I think this is inevitable if we are keep it compatible. In v3 we can remove this overhead easily as we will be allowed to have breaking changes.
But this makes sense. This was a clear explanation, thanks.
Even I wasn't expecting so many hurdles. But I guess it's good that had this detail conversation. It is much clear now.
gavv commentedon Mar 28, 2023
Regarding overhead, I suppose it's comparably with json marshalling and unmarshalling, which is as well based on reflect. So here we don't change much in terms of performance.
angshumanHalder commentedon Apr 2, 2023
Got it. Btw for now what I am following is every number that will not cause precision loss will be converted to float64 by default and numbers which will cause precision loss will be converted to big.Float. And this will be the default behaviour for now. Once I fix all the test cases and this PR is merged. I will add a config var which will use either float64 or big.Float as per the user choice. I think this would be the ideal way to go forward. Otherwise it will increase complexity to a already complex PR.
Also the PR is getting huge in terms of code changes. 😅