Day 9 — Error, rounding & floating-point idea

Updated

July 30, 2026

Day 9 — Error, rounding & floating-point idea

Stage I · concept day
Goal: Define absolute and relative error; distinguish rounding vs truncation; exhibit catastrophic cancellation; model floating-point as binary scientific notation with machine epsilon idea (not a full IEEE deep dive); reason about simple error propagation.

Why this matters

Every sensor, every float, every rounded display lies a little. Algorithms that are correct in exact $\mathbb{R}$ can fail in floating-point (or succeed with care). Relative error is how you say “accurate to $0.1\%$.” Cancellation bugs look like “random” wrong digits.

Theory

True value, approximation, error

Let $x$ be a true value and $\hat{x}$ an approximation.

Absolute error: \[E_{\mathrm{abs}}=|\hat{x}-x|.\]

Relative error (for $x\neq 0$): \[E_{\mathrm{rel}}=\frac{|\hat{x}-x|}{|x|}.\]

Percent error: $100\cdot E_{\mathrm{rel}}\%$.

Relative error is scale-free: being off by $0.1$ matters differently near $1$ vs near $1000$.

Rounding vs truncation

Representations use finitely many digits.

Truncation (chopping): drop extra digits.
$\pi\approx 3.14159\ldots$ truncated to $4$ decimals: $3.1415$.

Rounding (half-up common school rule): look at the next digit; if $\ge 5$, round away from zero in the last kept place (variants exist: banker’s rounding / round-to-even in IEEE).

$\pi$ rounded to $4$ decimals: $3.1416$.

Rounding error bound (decimal, $k$ places after point): absolute error $\le \frac{1}{2}\cdot 10^{-k}$ under standard round-to-nearest.

Sources of error

Measurement noise
Representation (finite digits / bits)
Truncation of series / algorithms (math model cut off)
Propagation through arithmetic

Catastrophic cancellation

Subtracting two nearly equal quantities loses leading significant digits.

Example. In $6$-digit decimal arithmetic, let \[x=1.00000,\quad y=1.00001\quad\text{(true)},\] but suppose we only store $x'=1.00000$, $y'=1.00000$ after rounding coarse measurements—difference $0$ vs true $10^{-5}$.

Classic: compute $1-\cos\theta$ for tiny $\theta$ in floating-point loses digits; mathematically equivalent $2\sin^2(\theta/2)$ is stabler (identity reminder, not a coding lab).

Algebraic example. $(1+x)^2-1=2x+x^2$. For tiny $x$, left side cancels; right side is better.

Floating-point model (binary scientific)

A simplified floating-point number looks like

\[\pm\, (1.b_1 b_2\ldots b_{p-1})_2 \times 2^{e},\]

with a fixed precision $p$ (significand bits including the leading $1$ for normals) and exponent $e$ in a finite range $e_{\min}..e_{\max}$.

(This is the idea of binary floating point; IEEE 754 adds subnormals, NaN, $\pm\infty$, rounding modes—acknowledge, do not memorise the whole standard today.)

Machine epsilon (idea): $\varepsilon_{\mathrm{mach}}$ is roughly the gap between $1$ and the next representable number above $1$. Relative spacing near $1$ is order $\varepsilon_{\mathrm{mach}}$.
For a $p$-bit significand (binary), $\varepsilon_{\mathrm{mach}}\sim 2^{1-p}$ (order of magnitude).

Consequence: relative representation error for “typical” numbers is on the order of machine epsilon (when not underflowing/overflowing).

Decimal analogue: with $d$ significant digits, relative spacing $\sim 10^{1-d}$.

Propagation of error (lite)

Suppose $x$ and $y$ have absolute errors $\delta_x,\delta_y$ (bounds).

Sum: $|(x+\delta_x)+(y+\delta_y)-(x+y)|\le |\delta_x|+|\delta_y|$ — absolute errors add (triangle inequality).

Product (sketch): $(x+\delta_x)(y+\delta_y)=xy+x\delta_y+y\delta_x+\delta_x\delta_y$.
Relative error approximately $\frac{|\delta_x|}{|x|}+\frac{|\delta_y|}{|y|}$ when second-order term neglected and $x,y\neq 0$.

Division: similar approximate addition of relative errors.

Multiplication by constant $c$: absolute error scales by $|c|$; relative error unchanged.

Conditioning (intuition only)

A problem is ill-conditioned if small input changes cause large output changes (e.g. subtracting nearly equal numbers). Even exact arithmetic on slightly wrong inputs fails; floating-point makes it worse.

Worked examples

Example 1 — Absolute and relative

True $x=200$, approximation $\hat{x}=201$.
$E_{\mathrm{abs}}=1$, $E_{\mathrm{rel}}=1/200=0.005=0.5\%$.

True $x=2$, $\hat{x}=3$: same absolute error $1$, relative $50\%$.

Example 2 — Rounding vs truncation

$x=\frac{2}{3}=0.6666\ldots$
Truncate to $3$ d.p.: $0.666$. Absolute error $\approx 0.000666\ldots$
Round to $3$ d.p.: $0.667$. Absolute error $\approx 0.000333\ldots$

Example 3 — Percent error

Measure $9.8$ for true $9.81$: $E_{\mathrm{rel}}=0.01/9.81\approx 0.102\%$.

Example 4 — Cancellation

Exact: $(1+10^{-8})-1=10^{-8}$.
In a decimal system with $7$ significant digits, $1+10^{-8}$ may store as $1.000000$, difference $0$.

Example 5 — Algebraic rewrite

Evaluate $f(x)=\sqrt{x+1}-\sqrt{x}$ for large $x$. Rationalize: \[f(x)=\frac{1}{\sqrt{x+1}+\sqrt{x}}.\] Right form avoids subtracting close square roots.

Example 6 — Sum error bound

If each of $100$ terms has absolute error $\le 10^{-6}$, sum absolute error $\le 100\cdot 10^{-6}=10^{-4}$ (worst case; random errors may partially cancel—not guaranteed).

Example 7 — Relative product

$x=2.0$ with $1\%$ relative error, $y=5.0$ with $2\%$ relative error: product relative error $\approx 3\%$ rough bound.

Example 8 — Machine epsilon idea

If $\varepsilon=2^{-52}\approx 2.22\times 10^{-16}$ (IEEE double-ish order), then $1+\varepsilon/2$ may equal $1$ in floating-point, while $1+\varepsilon>1$.

Example 9 — Decimal fp model

$3$-digit decimal floating point: numbers like $d.dd\times 10^{e}$.
$1.00\times 10^{0}$ next number $1.01\times 10^{0}$; relative gap $0.01$.

Example 10 — Underflow / overflow idea

Too small $\to 0$ (underflow); too large $\to\infty$ flag (overflow). Not the same as rounding error in the normal range.

Example 11 — Rounding money

$\$1/3$ three ways: $0.33+0.33+0.34$ strategies matter; floating binary money is a design smell—prefer integer cents.

Example 12 — Forward error

Computed $\hat{y}$ vs true $y=f(x)$: $|\hat{y}-y|$ is forward absolute error.

Example 13 — Ill-conditioned subtraction

$f(x,y)=x-y$ with $x\approx y$: relative error in output can be huge even if $x,y$ accurate relatively.

Example 14 — Bound for round-to-nearest

Rounded to integer: absolute error $\le 1/2$. Rounded to $1$ decimal: $\le 0.05$.

Exercises

Easy

$x=50$, $\hat{x}=49$: absolute and relative error.
Round $2.71828$ to $3$ significant figures; to $2$ decimal places.
Truncate $2.71828$ to $2$ decimal places.
Which is more accurate relatively: $1000\pm 1$ or $10\pm 0.05$?
Define machine epsilon in one sentence.

Medium

True value $1/3$. Approximations $0.33$ and $0.333$: compare absolute and relative errors.
Explain catastrophic cancellation with a numeric decimal example of your own (state precision).
Rationalize $\sqrt{x+1}-\sqrt{x}$ as in Example 5; evaluate both forms at $x=10^{4}$ by hand with $6$ sig figs if possible.
If absolute errors add for a sum of $n$ terms each $\le\varepsilon$, state the bound.
Approximate relative error of a product given relative errors $r_1,r_2$.
In $4$-digit decimal floating point, represent $12.345$ and $0.0012345$ (normalized).
Why is relative error undefined or unhelpful at true value $0$? What do people use instead?
Compute percent error: claimed $256$ MB, true $256\times 10^{6}$ bytes vs $256\times 2^{20}$ bytes — different definitions (rough percent).

Hard

Prove $|\hat{x}-x|\le\varepsilon\Rightarrow$ relative error $\le\varepsilon/|x|$ for $x\neq 0$.
Show that if $\hat{x}=x(1+\delta)$ with $|\delta|$ small, then relative error is $|\delta|$.
Explain why $(1+x)^2-1$ is unstable for tiny $x$ but $2x+x^2$ is better.
Bound absolute error when rounding a real to $k$ decimal places (round-to-nearest).
Give an example where absolute error is small but relative error is large, and vice versa.
Propagation: if $\hat{\theta}=\theta+\delta$, estimate absolute error in $\sin\hat{\theta}$ using $|\sin'|\le 1$ (mean value idea: change $\le|\delta|$).
Discuss: is $0.1+0.2=0.3$ exact in binary floating point? Why related to Day 3 terminating decimals?

Challenge / CS-flavored

Why store currency as integer minor units instead of binary float?
Summing $1.0$ a million times vs summing many tiny values first—order can matter (qualitative associativity failure).
“Double has about $15$–$16$ decimal digits” — connect to $\log_{10}(2^{53})$.
Conditioning: solving $x+y=2$, $x+1.0001 y=2$ vs a well-separated system—what happens to solution if RHS perturbs slightly? (qualitative $2\times 2$ feel; Day 14)
Report an approximation of $\sqrt{2}$ with absolute error $<10^{-3}$; prove your bound by squaring.

Forward vs backward error (names)

Forward error: distance between computed output and true output for the same input.
Backward error: how little must you perturb the input for the computed output to be exact for the perturbed problem. Stable algorithms often have small backward error even if the problem is sensitive (ill-conditioned).

Rounding modes (IEEE awareness)

Round to nearest (ties to even), toward $0$, toward $+\infty$, toward $-\infty$. School “half up” is one of several policies. Exact ties are rare in binary but defined.

Decimal floating mental model

$t$ significant digits: represent numbers $\pm m\times 10^{e}$ with integer $m$ having at most $t$ digits. Relative spacing $\approx 10^{1-t}$. Mirrors binary with $2$ instead of $10$.

Propagation table (summary)

Operation	Absolute error (bound idea)	Relative error (approx)
$x\pm y$	add abs errors	can explode if cancellation
$xy$	—	add rel errors
$x/y$	—	add rel errors
$c x$	$\|c\|$ scales abs	rel unchanged

Extra exercises

True $e\approx 2.71828$, use $2.72$: abs and rel error.
Show that if two numbers agree in the first $k$ significant decimal digits, relative error is on the order $10^{-k}$ (heuristic).
Explain why summing many positive floats from smallest to largest can reduce error vs largest-first (qualitative).
Condition of $f(x)=\sqrt{x}$ near $0$ vs near $1$ (derivative intuition: $f'=\frac{1}{2\sqrt{x}}$ blows up near $0$).
Rewrite $1-\sqrt{1-x}$ for small $x$ by rationalizing to avoid cancellation.

CS connection

IEEE 754 float/double, GPU half-precision, and fixed-point DSP all trade range, precision, and speed. Numerical libraries rewrite formulas to avoid cancellation. Testing uses absolute tolerances and relative tolerances (atol, rtol). Hashing and crypto want exact integers, not floats. Logging “error bars” is relative-error thinking.

Common pitfalls

Pitfall	What to do instead
Only quoting absolute error	Also report relative when scale varies
Trusting all decimal digits of a float print	Know precision limits
Subtracting nearly equal floats	Rewrite algebraically when possible
Assuming $+$ is associative in fp	Order can change results
Using binary float for money	Integer cents / decimal types
Confusing truncation of series with chopping digits	Name the source of error
Machine epsilon as “smallest positive float”	It’s relative spacing near $1$, not underflow threshold

End-of-day synthesis problems

S1. True value $\sqrt{2}\approx 1.41421356$. Approx $1.414$: abs, rel, percent error.

S2. In $6$ significant digit decimal arithmetic, explain failure of $(1+10^{-7})-1$.

S3. Rationalize $\sqrt{x+1}-\sqrt{x}$; compare stability narrative for large $x$.

S4. Bound absolute error of sum of $50$ terms each rounded to $10^{-4}$ abs error.

S5. Define machine epsilon; distinguish from underflow threshold.

S6. Product relative error estimate: $2\%$ and $3\%$ inputs → about $5\%$ product.

S7. Why $0.1$ is problematic in binary floating point (link Day 3).

S8. Round $9.995$ to $3$ significant figures; to $2$ decimal places.

Checkpoint

Compute absolute, relative, and percent error
Round vs truncate a decimal expansion
Give a catastrophic cancellation example
Describe floating-point as $\pm m\times 2^{e}$ with finite $m$
State machine epsilon as an idea
Bound sum absolute errors; approximate product relative errors
Explain S2 and S5 in one sentence each

Write two takeaways in your own words.

Deep dive — error, cancellation, floating-point idea

D1 — Absolute vs relative side by side.
True $x=200$, $\hat x=201$: $E_{\mathrm{abs}}=1$, $E_{\mathrm{rel}}=0.005$.
True $x=2$, $\hat x=3$: same absolute error $1$, relative $0.5$.
Always ask “error relative to what scale?” before comparing approximations of different magnitudes.

D2 — Catastrophic cancellation in decimal $t$-digit arithmetic.
With $6$ significant digits, $1.00000$ and $1.00001$ may both round to $1.00000$ if the sixth digit cannot hold the distinction—difference becomes $0$ instead of $10^{-5}$.
Algebraic rewrite $(1+x)^2-1\to 2x+x^2$ keeps the small $x$ visible.

D3 — Rationalize for stability.
$f(x)=\sqrt{x+1}-\sqrt{x}=\dfrac{1}{\sqrt{x+1}+\sqrt{x}}$.
For large $x$, left subtracts nearly equal roots; right is a single well-scaled quotient.

D4 — Propagation table in action.
Two factors each with $2\%$ relative error: product relative error $\approx 4\%$ (first-order).
Sum of $50$ terms each with abs error $\le 10^{-4}$: worst-case abs bound $5\cdot 10^{-3}$ (errors need not cancel).

D5 — Machine epsilon vs underflow.
$\varepsilon_{\mathrm{mach}}$: relative spacing near $1$ (next number after $1$ is about $1+\varepsilon$).
Underflow threshold: smallest positive normal magnitude—far smaller than $\varepsilon$ as an absolute number near zero. Do not confuse the two.

D6 — Binary $0.1$ link to Day 3.
Decimal $0.1=\frac{1}{10}$ has denominator prime factor $5$; in binary only primes $2$ terminate. So $0.1$ is repeating in binary—hence classic 0.1+0.2≠0.3 stories in floating-point.

Extra practice set

True $\sqrt{2}\approx 1.41421356$, approx $1.41$: abs, rel, percent error.
Bound abs error rounding to $3$ decimal places (round-to-nearest).
Rewrite $1-\sqrt{1-x}$ for small $x$ by rationalizing.
Why currency as integer cents beats binary float: give a two-sentence argument.
If $\hat x=x(1+\delta)$, show relative error equals $|\delta|$.

Synthesis checklist (error day)

Syn-A. Compute abs/rel/percent for one approximation; state the true base.
Syn-B. Give one catastrophic cancellation numeric story at fixed digit precision.
Syn-C. Rationalize one difference of square roots (or $1-\sqrt{1-x}$).
Syn-D. Bound a sum of $n$ absolute errors; approximate a product’s relative error.
Syn-E. Define machine epsilon in one sentence and contrast it with underflow.
Syn-F. Link non-terminating binary decimals (Day 3 primes other than $2$) to float surprises.

Work Syn-A through Syn-D closed book before claiming the day done.

Tomorrow

Day 10 — Gate I. Mixed portfolio exam covering Days 1–9: integers through error analysis, with timed-feel sections and answer keys for half the items.