ISS 2016 Statistics Paper-2 Solution: Question 5 (Correlation of Fitted Values and Residuals)
We continue our question-by-question walk through the ISS Statistics Paper-2 previous year papers, picking up right where the last post left off. Today's question sits inside the Gauss-Markov linear model chapter and tests whether you really understand what the hat matrix does to the fitted values and residuals, rather than just remembering a formula. Let's work through it slowly, with every algebraic step shown.
Quick Summary
Topic: Gauss-Markov linear model — correlation structure of fitted values and residuals
Question Reference: ISS 2016, Statistics Paper-2, Question 5
Correct Answer: Option (d) — Neither 1 nor 2
The Question
In the Gauss-Markov linear model, let ŷ denote the vector of fitted values and ε̂ denote the vector of residuals. Consider the following statements:
The components of ŷ are pairwise uncorrelated.
The components of ε̂ are pairwise uncorrelated.
Which of the above statements is/are correct?
(a) 1 only
(b) 2 only
(c) Both 1 and 2
(d) Neither 1 nor 2
Step 1: Set Up the Model Carefully
Start with the standard Gauss-Markov linear model in matrix form:
y = Xβ + ε
Here y is the n×1 vector of observations, X is the n×p design matrix (treated as fixed, known constants), β is the p×1 vector of unknown parameters, and ε is the n×1 error vector satisfying the Gauss-Markov assumptions: E(ε) = 0 and Var(ε) = σ²I, where I is the n×n identity matrix. Note that this is exactly where the assumptions stop — nothing is assumed about normality, and nothing is assumed about the columns of X being "nicely spread out." That gap is exactly what this question is probing.
Step 2: Write Down the Fitted Values and Residuals
The ordinary least squares estimator of β is β̂ = (X'X)⁻¹X'y. Using this, the fitted values are:
ŷ = Xβ̂ = X(X'X)⁻¹X'y
It is standard practice to give the matrix X(X'X)⁻¹X' its own name — the hat matrix, usually written H, because it "puts a hat on y": ŷ = Hy. The residual vector is then simply what's left over:
ε̂ = y − ŷ = y − Hy = (I − H)y
Two properties of H are used repeatedly in this derivation, and it's worth proving them rather than just quoting them:
H is symmetric:H' = [X(X'X)⁻¹X']' = X[(X'X)⁻¹]'X' = X(X'X)⁻¹X' = H, because (X'X)⁻¹ is itself symmetric.
H is idempotent:HH = X(X'X)⁻¹X'X(X'X)⁻¹X' = X(X'X)⁻¹X' = H, because the middle (X'X)⁻¹X'X cancels down to the identity.
Since I − H is also symmetric and idempotent (a short algebra check confirms (I−H)(I−H) = I − 2H + HH = I − 2H + H = I − H), both H and I − H behave like "projection matrices."
Step 3: Derive the Variance-Covariance Matrices
Now compute Var(ŷ). Since ŷ = Hy and Var(y) = Var(ε) = σ²I:
Var(ŷ) = H Var(y) H' = H(σ²I)H' = σ²HH' = σ²HH = σ²H
(using symmetry H' = H and idempotency HH = H). Similarly for the residuals:
Var(ε̂) = (I−H) Var(y) (I−H)' = σ²(I−H)(I−H) = σ²(I−H)
So the entire question boils down to one thing: are the off-diagonal entries of H (and of I−H) equal to zero, or not? The diagonal entries of σ²H give the variances of the individual ŷᵢ's — that part is never in doubt. What matters for "pairwise uncorrelated" is the off-diagonal entries, because Cov(ŷᵢ, ŷⱼ) = σ²Hᵢⱼ for i ≠ j, and Cov(ε̂ᵢ, ε̂ⱼ) = σ²(I−H)ᵢⱼ = −σ²Hᵢⱼ for i ≠ j (since I contributes nothing off the diagonal).
Step 4: Check Whether Hᵢⱼ Is Actually Zero — A Worked Numerical Example
This is the step most students skip, and it's exactly where the question is testing you. Let's not skip it. Take a tiny concrete example: simple linear regression with n = 3 observations at x = 0, 1, 2. The design matrix (with an intercept column) is:
X = [[1, 0], [1, 1], [1, 2]]
First compute X'X:
X'X = [[3, 3], [3, 5]] (since Σ1 = 3, Σxᵢ = 3, Σxᵢ² = 0+1+4 = 5)
Its determinant is 3×5 − 3×3 = 15 − 9 = 6, so:
(X'X)⁻¹ = (1/6)[[5, −3], [−3, 3]]
Now compute (X'X)⁻¹X'. With X' = [[1,1,1],[0,1,2]]:
(X'X)⁻¹X' = (1/6)[[5, 2, −1], [−3, 0, 3]]
Finally, H = X · (X'X)⁻¹X'. Multiplying each row of X into this result gives:
H = (1/6) × [[5, 2, −1], [2, 2, 2], [−1, 2, 5]]
Look at the off-diagonal entries: H₁₂ = 2/6 = 1/3, which is not zero. That single nonzero number is the whole answer to the question. It tells us Cov(ŷ₁, ŷ₂) = σ²(1/3) ≠ 0, so statement 1 is false in general. And since I − H has off-diagonal entries equal to −1/3, −1/6, and so on — also nonzero — Cov(ε̂₁, ε̂₂) ≠ 0 too, so statement 2 is false in general as well.
As a sanity check, notice trace(H) = (5+2+5)/6 = 12/6 = 2, which equals p, the number of parameters (intercept and slope) — exactly what idempotency guarantees, since trace(H) = rank(H) = rank(X) = p. This is a good habit to build into every hat-matrix calculation: if your trace doesn't equal the number of estimated parameters, you've made an arithmetic slip somewhere.
Pro Tip: The trap in this question is a false memory. Every student remembers the true and famous fact that fitted values and residuals are uncorrelated
Step 5: Conclusion and Answer Confirmation
Since H is generally a full matrix (not diagonal) unless the design happens to be orthogonal in a very specific way, its off-diagonal entries are nonzero in general — and the same goes for I − H. This means:
Statement 1 (components of ŷ pairwise uncorrelated) is not generally true.
Statement 2 (components of ε̂ pairwise uncorrelated) is not generally true.
So the correct answer is option (d) — Neither 1 nor 2. Checking against the official ISS 2016 Paper-2 answer key (Series A booklet), the answer listed for Question 5 is also D, so our independently derived answer matches the official key exactly.
Why This Question Matters
This question is a favourite in ISS-level papers because it can't be answered by rote memorisation of "fitted values and residuals are orthogonal" — a phrase most students learn in their first regression class without ever unpacking what it actually means. Questions built this way are designed to separate candidates who understand the hat matrix as an actual n×n matrix with real off-diagonal structure from those who have only memorised the trace and orthogonality results. Building the habit of writing out a small worked H matrix, as we did above, whenever a question mentions "pairwise" or "jointly" correlated observations pays off across the entire linear models portion of Paper-2, not just in this one question.
Frequently Asked Questions
What exactly is the hat matrix, in plain terms?
The hat matrix H = X(X'X)⁻¹X' is the matrix that transforms the observed y vector into the fitted ŷ vector, i.e. ŷ = Hy. Geometrically, it projects y orthogonally onto the column space spanned by X, which is why H is symmetric and idempotent.
Is Cov(ŷ, ε̂) = 0 always true?
Yes — this is a separate and always-true result under the Gauss-Markov assumptions, following directly from H(I−H) = 0. It says fitted values and residuals are uncorrelated with each other as two vectors. It does not say anything about correlation among the individual components within ŷ or within ε̂, which is what this particular question is actually asking.
Would the answer change under a special design matrix?
In principle, yes — if X were constructed so that H happened to be diagonal, the off-diagonal covariances would vanish and statement 1 (and correspondingly statement 2) could become true for that specific design. But the question asks about the Gauss-Markov linear model in general, without any such special restriction, so the general-case answer of "neither" is what's being tested.
Why does trace(H) equal the number of parameters?
Because H is idempotent, its eigenvalues are only 0 or 1, and the number of eigenvalues equal to 1 equals the rank of H, which equals rank(X) = p (the number of linearly independent columns, i.e. parameters, in the model). Since trace equals the sum of eigenvalues, trace(H) = p.
How often does this hat-matrix topic appear in ISS Paper-2?
Very often — properties of H, of I − H, degrees of freedom, and the distribution of quadratic forms built from these matrices are recurring themes across multiple years of the linear models and regression sections of Paper-2, so it's worth mastering the derivation once rather than memorising isolated facts.
What's the quickest way to avoid this mistake in the exam?
Before selecting an option, explicitly write down which covariance you are being asked about — within ŷ, within ε̂, or between ŷ and ε̂ — since these are three genuinely different quantities with three different answers, and the question is deliberately phrased to make you conflate them under exam time pressure.
If any step above felt rushed, drop your doubt in the comments and we'll walk through it further — and if this helped, do share it with a fellow ISS aspirant who's revising the same linear models chapter.
Comments