Linear Regression & Coefficient Uncertainty

⭐ The big idea: linear regression just draws the best straight line through your points, and the coefficients β₀ and β₁ tell you where that line starts and how steep it is. Everything here is "find that line, then measure how sure we are about it."

🟦 The Dataset (our 4 points)

We have four points: (0,1), (0,3), (10,10), (10,6).

x-values: 0, 0, 10, 10 → y-values: 1, 3, 10, 6

🎯 Two clusters: think of it like asking 4 people their age (x) and their shoe size (y). Two people are age 0, two are age 10. We want one straight line that "best fits" everyone.

🎨 How to read this: LEFT — the green line y = 2 + 0.6x is our "best fit" through the 4 blue dots. The dashed orange lines are how far each point is from the line (the "errors"). RIGHT — the same errors drawn separately: these are the "noise" ε. This is exactly the picture for Question 7.

⭐ The ONE formula you need first

Mean of x: x̄ = (0 + 0 + 10 + 10) / 4 = 5

The mean (x̄, read "x-bar") is just the average — add up all the x-values and divide by how many there are.

0 + 0 + 10 + 10 = 20 , 20 ÷ 4 = 5

1️⃣ Question 1 — What is x̄?

What is x̄ (the mean of the x-values)?

❌ 4 ✅ 5 ❌ 0.6 ❌ 2

✅ Answer: 5

List the x-values: 0, 0, 10, 10.

Add them: 0 + 0 + 10 + 10 = 20.

Divide by 4 points: 20 / 4 = 5.

🍎 Average analogy: if 4 friends have 0, 0, 10, and 10 apples, the average is 20 apples ÷ 4 friends = 5 apples each.

2️⃣ Question 2 — What is β̂₁?

In the same problem, what is β̂₁ (the slope)?

❌ 5 ✅ 0.6 ❌ 2

✅ Answer: 0.6

The slope β₁ measures "how much y goes up when x goes up by 1".

Formula: β₁ = Σ(x−x̄)(y−ȳ) / Σ(x−x̄)². It's the "best tilt" of the line.

Plugging in the numbers gives β₁ = 0.6. (Check: from x=0 to x=10, y rises from 2 to 8 — that's 6 up over 10 across = 0.6 per unit.)

⛰️ Slope analogy: if a road rises 6 meters over 10 meters of horizontal distance, its slope is 6/10 = 0.6. The line rises gently.

3️⃣ Question 3 — What is Mₓᵀ?

What is Mₓᵀ (the transpose of the design matrix)?

❌ [0 0 10 10] ✅ [[1 1 1 1],[0 0 10 10]] ❌ [[1 1 1 1],[10 10 0 0]] ❌ [[0 0 10 10],[1 1 1 1]]

✅ Answer: [[1 1 1 1],[0 0 10 10]] (the second option)

The design matrix Mₓ

The design matrix Mₓ has a column of 1s (for the intercept β₀) plus a column of the x-values:

Mₓ

Transpose = flip the matrix sideways (rows become columns, columns become rows).

First row of Mₓᵀ = the column of 1s: [1 1 1 1].

Second row of Mₓᵀ = the column of x-values: [0 0 10 10].

Result: [[1 1 1 1],[0 0 10 10]] ✅ (the 1s row comes first).

🔄 Flip analogy: transposing is like rotating a bookshelf sideways — the same books, just arranged differently. The 1s column and the x column swap which way they point.

4️⃣ Question 4 — What is MₓᵀMₓ?

What is MₓᵀMₓ (design matrix squared)?

✅ [[4, 20],[20, 200]] ❌ [[20, 4],[20, 200]] ❌ [[20, 4],[200, 20]] ❌ [[4, 20],[200, 20]]

✅ Answer: [[4, 20],[20, 200]] (the first option)

200

Top-left: count of points × 1 = 4 (each 1·1 summed: four of them).

Top-right & bottom-left: sum of x-values = 0+0+10+10 = 20.

Bottom-right: sum of x² = 0+0+100+100 = 200.

🔢 Summary table: MₓᵀMₓ packs the data into 3 numbers — how many points (4), the sum of x (20), and the sum of x² (200). It's the same idea as AᵀA from the last learning check.

5️⃣ Question 5 — What is (MₓᵀMₓ)⁻¹?

What is (MₓᵀMₓ)⁻¹ (the inverse)?

✅ [[0.5, −0.05],[−0.05, 0.01]] ❌ [[0.1, −0.05],[−0.01, 0.05]] ❌ [[1, 5],[0.1, 0.25]] ❌ [[−0.05, 0.01],[0.5, −0.05]]

✅ Answer: [[0.5, −0.05],[−0.05, 0.01]] (the first option)

The 2×2 inverse formula

⁻¹ = 1/(ad−bc) ×

−b

−c

Our matrix is [[4, 20],[20, 200]], so a=4, b=20, c=20, d=200.

Determinant = ad − bc = 4×200 − 20×20 = 800 − 400 = 400.

Swap d and a, put minus on b and c: [[200, −20],[−20, 4]].

Divide by 400: [[0.5, −0.05],[−0.05, 0.01]] ✅.

➗ Division analogy: the inverse is like "dividing" by a matrix. If MₓᵀMₓ is "times 400", then its inverse is "divide by 400" — but with the numbers rearranged too.

6️⃣ Question 6 — What is Mₓᵀy?

What is Mₓᵀy (design matrix times target)?

✅ [[20],[160]] ❌ [[160],[20]] ❌ [[400],[200]] ❌ [[20],[200]]

✅ Answer: [[20],[160]] (the first option)

160

Top row: sum of all y = 1+3+10+6 = 20.

Bottom row: sum of x·y = 0·1 + 0·3 + 10·10 + 10·6 = 0 + 0 + 100 + 60 = 160.

🧾 Receipt analogy: Mₓᵀy adds up "total y" (20) and "total x times y" (160) — the two numbers we need to solve for the line.

7️⃣ Question 7 — What does σ² mean?

In the equations below, what is the interpretation of σ²?

❌ Standard deviation of the β coefficients (which are normal) ❌ Standard deviation of the measurement noise ε ✅ Variance of the measurement noise ε in a linear additive model ❌ Variance of the β coefficients arising from matrix multiplication

✅ Answer: Variance of the measurement noise ε (the third option)

Every point isn't exactly on the line — it wobbles off by a bit. That wobble is the noise ε (the orange dashed lines in our picture).

σ² is how spread-out that wobble is — the "variance" of the noise. Small σ² = points hug the line tightly; big σ² = points scatter everywhere.

The two equations in the question use σ² to compute how "sure" we are about β₀ and β₁. More noise (bigger σ²) = less sure.

🎯 Darts analogy: imagine throwing darts at a board. The line is the bullseye. σ² is how spread-out your throws are. A tight cluster = small σ² (confident); scattered throws = big σ² (noisy, unsure).

🎮 Play with it yourself

📏 Regression Fitter

Watch how β₀ (intercept) and β₁ (slope) change the line. Can you eyeball the best fit?

Intercept β₀ (currently 2): Slope β₁ (currently 0.6):

—

🎯 Noise Simulator (Question 7)

Drag to change the noise level σ. Watch how much the points scatter around the line.

Noise σ (0 = perfect line, 3 = very noisy):

—

📋 Quick Answer Sheet

#	Question	Answer
1	x̄ (mean of x)	5
2	β̂₁ (slope)	0.6
3	Mₓᵀ	[[1 1 1 1],[0 0 10 10]]
4	MₓᵀMₓ	[[4, 20],[20, 200]]
5	(MₓᵀMₓ)⁻¹	[[0.5, −0.05],[−0.05, 0.01]]
6	Mₓᵀy	[[20],[160]]
7	Interpretation of σ²	Variance of noise ε

🧠 Checklist (did you get it?)

✅ Mean x̄ = average of x-values.
✅ β₁ is the slope (how much y rises per 1 unit of x).
✅ Mₓ has a column of 1s + the x column; Mₓᵀ flips it sideways.
✅ MₓᵀMₓ = [[n, Σx],[Σx, Σx²]].
✅ Inverse of [[a,b],[c,d]] = 1/(ad−bc) × [[d,−b],[−c,a]].
✅ Mₓᵀy = [[Σy],[Σxy]].
✅ σ² = variance of the noise (how spread-out the points are around the line).