# Non-deterministic results with float 32 addition **URL:** \ **Category:** help\ **Created:** [June 15, 2022, 7:29pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009 "2022-06-15T19:29:39Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jinlow](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/jinlow/32/15238_2.png) [@jinlow](https://users.rust-lang.org/u/jinlow)\ **Post date:** [June 15, 2022, 7:29pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/1 "2022-06-15T19:29:39Z")
Hello all, First time posting here, let me know if there is a better place for this kind of question šŸ™‚ I am seeing non-deterministic behavior when summing `f32` values, and am looking for best practice/advice for how to deal with this. I am working on some machine learning algorithms in rust, as side projects. I am using `f32` values as this is fairly common in these algorithms for speed. However, I am seeing depending on how I sum a vector of `f32` values, I get different results. For example... ```rust let records = 300000; let vec = vec![0.23500371; records]; println!("Sum Result: {}", vec.iter().sum::()); // Sum Result: 70366.76 ``` Now the same value, using multiplication to simulate summing. ```rust println!("Multiplication Results {}", vec[0] * (records as f32)); // Multiplication Results 70501.11 ``` Finally, summing, but chunks of the vector, and then summing those values. This is to mimic, sometimes I need to sum a vector by another vector, and I would expect the sub-sums summed together would equal the same as summing the entire vector, but again different results... ```rust let chunks = vec.chunks_exact(10); let remainder = chunks.remainder().to_vec(); assert!(remainder.is_empty()); let vec_c = chunks.map(|v| v.iter().sum::()).collect::>(); println!("Batch sum Result: {}", vec_c.iter().sum::()); // Batch sum Result: 70521.31 ``` Here is the [link to this in the playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=04ff4a5f92e13ad5e60914621fd9dd71). I am OK, with the fact that there is going to be differences between these operations for instance between a `f32` or a `f64`, just because of precision, but it seems summing chunks of a vector, and then for instance summing the same vector in it's entirety should produce the same results? This is probably just my lack of knowledge around how floating point numbers work in rust. Is there anything I can do, or pre-processing of the numbers to ensure these different operations will produce the same results? ---
**Author:** ![kpreid](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/kpreid/32/22420_2.png) [@kpreid](https://users.rust-lang.org/u/kpreid)\ **Post date:** [June 15, 2022, 7:49pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/2 "2022-06-15T19:49:16Z")
This is a fundamental fact of how floating-point arithmetic works. The rounding error in the result of each addition operation necessarily depends on the relative magnitude of the inputs, since that determines how much the smaller number has to be rounded to fit in the mantissa, given the exponent of the sum. I'm not an expert in the area and can't recommend a specific solution for your problem, but for example, [Kahan summation](https://en.wikipedia.org/wiki/Kahan_summation_algorithm) is a famous algorithm to reduce error when summing many floating point numbers. In situations where you're summing a few numbers of statically known relationships, improvement can be gained simply by arranging to sum numbers of similar magnitude first ā€” since that has the least rounding error. ---
**Author:** ![scottmcm](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/scottmcm/32/4286_2.png) [@scottmcm](https://users.rust-lang.org/u/scottmcm)\ **Post date:** [June 15, 2022, 7:50pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/3 "2022-06-15T19:50:11Z")
> [@jinlow](#): > > but it seems summing chunks of a vector, and then for instance summing the same vector in it's entirety should produce the same results? No, because floating point addition is not associative. The losses from rounding can be different in different places. Here's a very simple example: [https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=a9eb4c5804c5de01d51c860684aeda00](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=a9eb4c5804c5de01d51c860684aeda00) ```rust [src/main.rs:2] (-2.0 + 2.0) + f64::EPSILON = 2.220446049250313e-16 [src/main.rs:3] -2.0 + (2.0 + f64::EPSILON) = 0.0 ``` Specifically, the problem is [Catastrophic cancellation - Wikipedia](https://en.wikipedia.org/wiki/Catastrophic_cancellation). Obligatory link to [What every computer scientist should know about floating-point arithmetic](https://doi.org/10.1145%2F103162.103163). If you want a quick way to improve the error behaviour for summations, you can replace `.sum()` with `.tree_fold1(f32::add).unwrap_or(0.0)` using [https://docs.rs/itertools/\*/itertools/trait.Itertools.html#method.tree\_fold1](https://docs.rs/itertools/*/itertools/trait.Itertools.html#method.tree_fold1). ---
**Author:** ![SkiFire13](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/skifire13/32/17959_2.png) [@SkiFire13](https://users.rust-lang.org/u/SkiFire13)\ **Post date:** [June 15, 2022, 7:52pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/4 "2022-06-15T19:52:13Z")
To add on the other answers, the results are deterministic. For example your same code in Java: ```java int records = 300000; float d = 0.23500371f; float sum = 0; for(int i = 0; i < records; i++) sum += d; System.out.println(sum); float prod = d * records; System.out.println(prod); float chunked = 0; for(int i = 0; i < records; i += 10) { float chunk = 0; for(int j = 0; j < 10; j++) chunk += d; chunked += chunk; } System.out.println(chunked); ``` Prints: ```rust 70366.76 70501.11 70521.31 ``` Which is the same answer as your rust code. ---
**Author:** ![jinlow](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/jinlow/32/15238_2.png) [@jinlow](https://users.rust-lang.org/u/jinlow)\ **Post date:** [June 16, 2022, 2:00pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/5 "2022-06-16T14:00:00Z")
Thanks All, this makes a lot of sense. I understand now, that this is just a floating point property I need to consider when designing applications. Also, thanks for the great resources! ---
**Author:** ![steffahn](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/steffahn/32/47569_2.png) [@steffahn](https://users.rust-lang.org/u/steffahn)\ **Post date:** [June 16, 2022, 2:45pm UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/6 "2022-06-16T14:45:26Z")
> [@kpreid](#): > > I'm not an expert in the area and can't recommend a specific solution for your problem, but for example, [Kahan summation](https://en.wikipedia.org/wiki/Kahan_summation_algorithm) is a famous algorithm to reduce error when summing many floating point numbers. I would assume a good takeaway from the idea of Kahan summation might be to simply use an `f64` accumulator if you sum up a lot of `f32`s. You can convert the final result back to `f32` if you like. Seems like a simpler, and more performant, and more accurate approach than trying to encode a higher-precision accumulator using two `f32`s. If Iā€™m not mistaken, summing up `f32`s this way should give _quite_ accurate results (almost up to the full accuracy that an `f32` can represent) for a max. of about 1 billion items. (In my head, Iā€™m roughly calculating: ~7 significant decimals for f32, and ~16 significant decimals for `f64`, then 16 - 7 = 9 and 1 billion is 109). ---
**Author:** ![stonerfish](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@stonerfish](https://users.rust-lang.org/u/stonerfish)\ **Post date:** [June 17, 2022, 1:33am UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/7 "2022-06-17T01:33:03Z")
> [@jinlow](#): > > I am using `f32` values as this is fairly common in these algorithms for speed. However, I am seeing depending on how I sum a vector of `f32` values, I get different results. Like the others said, but in three easy to understand concepts... 1 . Floating point gives a big range with the cost of loss of accuracy. 2 . Every floating point operation has the potential to introduce a small amount of error. 3 . If you use floating point you are only guaranteed close results. ---
**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/users.rust-lang.org/system/32/51372_2.png) [@system](https://users.rust-lang.org/u/system)\ **Post date:** [September 15, 2022, 1:33am UTC](https://users.rust-lang.org/t/non-deterministic-results-with-float-32-addition/77009/8 "2022-09-15T01:33:39Z")
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.