ISS 2016 Statistics Paper-2 Solution: Question 14 (Number of Solutions to Normal Equations)
Continuing our question-by-question walk through the ISS Statistics Paper-2 archive, today we pick up right where the last post left off: 2016, Question 14. If you've been following this series from Question 1 onward, you already know the drill — read the question carefully, work every line of algebra by hand, and only then look at what the official key says.
Quick Summary
Topic: Linear Models — existence and uniqueness of solutions to the normal equations
Question Reference: ISS 2016, Statistics Paper-2, Question 14
Correct Answer: (d) Either 1 or infinite
The Question, As Asked
In the Gauss-Markov linear model, what is the possible number of distinct solutions of the normal equations?
(a) Either 0 or 1 or infinite
(b) 1 only
(c) Infinite
(d) Either 1 or infinite
Step 1: Write Down the Gauss-Markov Setup
The Gauss-Markov linear model is written as
y = Xβ + ε
Here y is the n × 1 vector of observations, X is the known n × p design matrix (the columns are your explanatory variables), β is the p × 1 vector of unknown parameters we want to estimate, and ε is the n × 1 error vector with E(ε) = 0 and a constant-variance, uncorrelated structure (that's what "Gauss-Markov" assumptions mean — nothing about normality is required here).
To estimate β by the method of least squares, we minimise the sum of squared errors S(β) = (y − Xβ)′(y − Xβ) with respect to β. Differentiating S(β) with respect to β and setting the derivative to zero gives what are called the normal equations:
X′X β̂ = X′y
Every question about "properties of least squares estimation" in this exam eventually comes back to this one equation, so it is worth memorising exactly how it is derived, not just the final line.
Step 2: Ask — Does This System of Equations Even Have a Solution?
Before asking how many solutions there are, we should ask whether a solution exists at all. This is where most students get stuck, because X′X need not be invertible — if two explanatory variables are perfectly correlated (or if p > n), X does not have full column rank, and X′X is a singular (non-invertible) p × p matrix.
Despite that, here is the key linear-algebra fact that makes this question work: for any real matrix X, the column space of X′X is exactly the same as the column space of X′. That is,
Col(X′X) = Col(X′)
Now look at the right-hand side of the normal equations, X′y. By definition, X′y is a linear combination of the columns of X′ (each observation yi multiplies the i-th column of X′, and we add these up). So X′y always lies in Col(X′), which — by the fact above — is the same as Col(X′X).
This means X′y is always inside the column space of X′X, no matter what X or y are. A linear system Aβ = b has at least one solution precisely when b lies in the column space of A. So the normal equations X′X β̂ = X′y are always consistent — they never fail to have a solution, even when X′X is singular.
Pro Tip: The step students most often skip — and the one that actually earns you the mark on this question — is proving X′y ∈ Col(X′X) rather than just asserting "the normal equations are always solvable." The way this is usually taught is to first fix the general fact Col(A′A) = Col(A′) for any matrix A, and only then plug in A = X. Once that lemma is sitting in your toolbox, half a dozen "how many solutions" and "is β̂ unique" questions across ISS and ISI papers fall out of it almost for free.
Step 3: Now Count the Solutions, Given That At Least One Exists
A basic result from linear algebra tells us that any consistent linear system Aβ = b has exactly one of two possibilities for its solution set:
Exactly one solution — this happens when A has full column rank, i.e., the null space of A contains only the zero vector.
Infinitely many solutions — this happens when A does not have full column rank, i.e., the null space of A contains a nonzero vector. If β̂ is one solution and v is any nonzero vector in the null space of A, then β̂ + cv is also a solution for every scalar c, which immediately gives infinitely many solutions.
A consistent linear system can never have "0 solutions" (that only happens for an inconsistent system), and it can never have a finite number greater than 1 either — if two distinct solutions β̂1 and β̂2 exist, then every point on the line joining them is also a solution, which forces infinitely many.
Applying this to A = X′X: since we already showed the normal equations are always consistent,
If X has full column rank p (equivalently, X′X is nonsingular), the normal equations have exactly one solution: β̂ = (X′X)−1X′y.
If X does not have full column rank (X′X is singular), the normal equations have infinitely many solutions — this is exactly the situation that arises in models with multicollinearity or with more parameters than can be identified from the data.
Step 4: Match to the Options and Confirm Against the Key
We've shown two things hold simultaneously and always: (i) the normal equations are always consistent, so "0 solutions" is never possible, and (ii) being consistent, the count is either exactly 1 or infinite, never any other finite number. Putting these together, the correct description is "either 1 or infinite," which is option (d).
This matches the official ISS 2016 Statistics Paper-2 answer key, which also lists (d)as correct. Our derivation and the key are in full agreement.
Why This Question Matters
This question is really testing whether you understand least squares as a piece of linear algebra rather than as a memorised formula. The formula β̂ = (X′X)−1X′y that most students learn first only works when X′X is invertible — but real design matrices (especially in ANOVA-type models with dummy variables) are frequently rank-deficient, and the exam wants to know if you can reason about what happens then. Students who drill this kind of derivation — starting from the minimisation of S(β), through the normal equations, to the rank argument — until it becomes automatic tend to breeze through the "properties of OLS/GLS estimators" questions that show up repeatedly across ISS papers, because they all lean on the same handful of linear-algebra facts.
Frequently Asked Questions
What exactly are the "normal equations" in regression?
The normal equations are the system X′X β̂ = X′y obtained by setting the derivative of the sum-of-squared-errors function to zero. Solving this system for β̂ gives the least squares estimator of the regression coefficients.
Why can't the normal equations have zero solutions?
Because X′y is always contained in the column space of X′X, a fact that follows purely from how matrix multiplication works — it has nothing to do with whether X′X is invertible. A linear system only has zero solutions when the right-hand side falls outside the column space of the coefficient matrix, and that never happens here.
When does X′X fail to be invertible?
X′X is singular whenever the columns of X are linearly dependent — for example, when two explanatory variables are perfect linear combinations of each other, or when the model has more parameters than the data can support (p > n). In such cases X does not have full column rank.
If there are infinitely many solutions, which one do we actually use?
When X′X is singular, we typically use a generalized inverse (X′X)− to obtain one particular solution, often called the minimum-norm solution. Even though β̂ itself is not unique in this case, certain functions of β̂ — called estimable functions — do remain uniquely determined, and those are what applied work usually focuses on.
Is this the same idea as the rank of a matrix determining solvability of any linear system?
Yes. This question is simply the general theory of linear systems (consistency and uniqueness based on rank and column space) applied to the specific system that arises from least squares. If you're comfortable with that general theory, this question becomes a direct application rather than something new to learn.
Does this result depend on the errors being normally distributed?
No. Everything used here — the derivation of the normal equations and the argument about consistency and solution count — depends only on the Gauss-Markov assumptions (zero mean, constant variance, uncorrelated errors), not on normality. Normality only becomes necessary later, when we want exact distributional results for hypothesis tests.
That wraps up Question 14. If any step above needs more unpacking, or if you solved it a different way, drop a comment below — and if a fellow ISS aspirant is stuck on this exact topic, feel free to pass this post along to them.
Comments