Skip to content
About
Main content

Float summation is not associative

Measure your machineNeeds Python

Problem

Floating-point addition rounds after every step, so the result depends on the order of the additions: adding a small number to a large one can lose it entirely, and the same numbers added in a different order lose different bits. A distributed SUM adds in an order set by partitioning and scheduling, so its result changes when the cluster is resized. An exactly rounded sum (math.fsum, a relative of summation) gives one stable value.

Sum one million doubles of very different magnitudes in five random orders with a plain left-to-right loop (Python 3.12's built-in sum() already compensates, so it would hide the effect), and compare with math.fsum.

The question. How many different sums do five orders give, and how far are they from the exactly rounded sum?


Predict

Summing the same million floats in five different orders: how many different sums come out?

Run it

  1. 1Install uv once. It fetches Python and the lab's packages for you.

    Terminal
    curl -LsSf https://astral.sh/uv/install.sh | sh
  2. 2Get the script. It is the whole lab: read it, then download it.

    float-summation-is-not-associative.py41 linesDownload
    View the code
    float-summation-is-not-associative.py
    # /// script
    # requires-python = ">=3.10"
    # ///
    # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
    # https://thedataengineering.com/lab-code/LICENSE.txt
    """Float summation is not associative.
    
    One million doubles of very different magnitudes, summed in five random
    orders with a plain left-to-right loop (Python 3.12's built-in sum()
    compensates, so it would hide the effect), then with math.fsum, which is
    exactly rounded.
    """
    import argparse, json, math, random
    
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--n", type=int, default=1_000_000, help="Numbers.")
    p.add_argument("--orders", type=int, default=5, help="Random orders to sum in.")
    p.add_argument("--seed", type=int, default=0, help="Random seed.")
    p.add_argument("--json", help="Also write the results to this JSON file.")
    args = p.parse_args()
    random.seed(args.seed)
    
    xs = [random.uniform(-1, 1) * 10 ** random.randint(0, 12) for _ in range(args.n)]
    def plain(v):
        s = 0.0
        for x in v: s += x                                                  # what a simple engine or a reducer does
        return s
    rows = []
    for i in range(args.orders):
        random.shuffle(xs); rows.append({"order": i + 1, "plain_sum": plain(xs), "fsum": math.fsum(xs)})
    exact = math.fsum(xs)
    for r in rows: r["plain_minus_exact"] = r["plain_sum"] - exact
    summary = {"distinct_plain": len({r["plain_sum"] for r in rows}), "distinct_fsum": len({r["fsum"] for r in rows}),
               "exact": exact, "max_abs_error": max(abs(r["plain_minus_exact"]) for r in rows)}
    for r in rows: print(f"order {r['order']}: plain loop {r['plain_sum']!r:<24} fsum {r['fsum']!r}")
    print(f"{summary['distinct_plain']} different plain sums, {summary['distinct_fsum']} distinct fsum value(s); largest error {summary['max_abs_error']:.4g}")
    
    if args.json:
        with open(args.json, "w") as f:
            json.dump({"summary": summary, "tables": {"orders": rows}}, f)
    
  3. 3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

    Terminal
    uv run float-summation-is-not-associative.py
    uv run float-summation-is-not-associative.py --help     # every knob
Knobs
Knob · defaultWhat it changes
--n1,000,000 Numbers.
--orders5 Random orders to sum in.
--seed0 Random seed.

Our run

The same numbers in five orders

The same numbers in five orders
OrderPlain loopmath.fsumPlain − exact
183233593475878.2883233593475876.861.42
283233593475879.8983233593475876.863.03
383233593475876.8383233593475876.86-0.031
483233593475880.9783233593475876.864.11
583233593475878.9883233593475876.862.13
Reference runApple M2 · 8 cores · 8 GB
Output · uv run float-summation-is-not-associative.py
order 1: plain loop 83233593475878.28        fsum 83233593475876.86
order 2: plain loop 83233593475879.89        fsum 83233593475876.86
order 3: plain loop 83233593475876.83        fsum 83233593475876.86
order 4: plain loop 83233593475880.97        fsum 83233593475876.86
order 5: plain loop 83233593475878.98        fsum 83233593475876.86
5 different plain sums, 1 distinct fsum value(s); largest error 4.109
macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

Five orders gave 5 different plain sums, off by up to 4.11 from the exactly rounded value; math.fsum gave 1 value in every order.


Read it

This is why a distributed SUM of doubles changes when the cluster is resized, and why money belongs in decimal or fixed-point types. If floats are unavoidable, use compensated or exactly rounded summation, or accept and document a tolerance in every reconciliation.