ISS 2016 Statistics Paper-2 Solution: Question 16 (MLEs in Simple Linear Regression)
We're continuing our question-by-question walkthrough of the ISS Statistics Paper-2 previous year papers, and today we land on Question 16 from the 2016 paper. It's a question that packs three classic results about the normal linear regression model into a single "which statements are correct" format, which makes it a great one to slow down on. If you've been following along since Question 1, you'll see this one leans on ideas from least squares and hypothesis testing we've already built up; if you're joining fresh, every step below is written out in full so nothing is assumed.
Quick Summary
Topic:Maximum likelihood estimation in the normal simple linear regression model — sufficiency, completeness, independence, and the chi-square distribution of the residual variance estimator
Question Reference:ISS 2016, Statistics Paper-2, Question 16
Correct Answer:Option (d) — Statements 1, 2 and 3 are all correct
The Question, Exactly As It Appeared
If in a linear model yi = β0 + β1xi + εi for i = 1, 2, 3, ..., n where εi are normal and independently distributed random variables with mean 0 and variance σ², the maximum likelihood estimators of β0, β1 and σ² are denoted by b0, b1 and σ̂² respectively, then which of the following are correct for these estimators?
The estimators are jointly complete sufficient statistics.
(b0, b1) is independent of σ̂².
(n − 2)σ̂²/σ² is a Chi-square random variable with (n − 2) degrees of freedom.
Select the correct answer using the code given below:
(a) 1 and 2 only
(b) 2 and 3 only
(c) 1 and 3 only
(d) 1, 2 and 3
Step 1: Write Down the Model and What We're Estimating
We have n pairs (xi, yi), where the xi are fixed, known constants (not random) and
yi = β0 + β1xi + εi, εi ~ N(0, σ²), independently for i = 1, ..., n.
This means yi ~ N(β0 + β1xi, σ²), and the yi's are mutually independent (because the εi's are). We're told the MLEs of β0, β1, σ² are called b0, b1, σ̂². Before touching the three statements, it helps to actually write down what these estimators are, because everything else follows from their algebraic form.
The joint density (likelihood) of y1, ..., yn is
L(β0, β1, σ²) = (2πσ²)−n/2 exp[ −(1/2σ²) Σ(yi − β0 − β1xi)² ]
Maximising this over β0, β1 for any fixed σ² is the same as minimising Σ(yi − β0 − β1xi)² — which is exactly the least squares problem. So the MLEs of β0 and β1 are the familiar least squares estimators:
b1 = Σ(xi − x̄)(yi − ȳ) / Σ(xi − x̄)², b0 = ȳ − b1x̄
Differentiating the log-likelihood with respect to σ² and setting it to zero gives the MLE of σ² as SSE/n, where SSE = Σ(yi − b0 − b1xi)² is the residual sum of squares. In hypothesis-testing formulas — including the one this exam paper consistently uses (see the Pro Tip below) — this residual variance estimator is written σ̂² and paired with an (n − 2) divisor, since two parameters (β0 and β1) were estimated from the data before the residuals could be formed. With that notation fixed, we can now examine each of the three statements.
Step 2: Is Statement 1 True? Are (b0, b1, σ̂²) Jointly Complete Sufficient?
Expand the exponent of the likelihood:
Σ(yi − β0 − β1xi)² = Σyi² − 2β0Σyi − 2β1Σxiyi + nβ0² + 2β0β1Σxi + β1²Σxi²
Substituting this back into L(β0, β1, σ²), you can regroup every term that involves the data into just three pieces: Σyi², Σyi, and Σxiyi. Everything else in the exponent (nβ0², 2β0β1Σxi, β1²Σxi²) depends only on the parameters and the fixed xi's, not on the observed y-data. That means the likelihood factors as
L(β0, β1, σ²) = g(Σyi², Σyi, Σxiyi; β0, β1, σ²) · h(y1, ..., yn)
with h ≡ 1 here. By the Factorisation Theorem, T = (Σyi², Σyi, Σxiyi) is a sufficient statistic for (β0, β1, σ²). Because this exponent is linear in the parameters times the data (a "natural parameter" times "natural statistic" structure) with a parameter space that is a full three-dimensional open set, this is a full-rank exponential family — and a standard theorem for exponential families says the natural sufficient statistic of a full-rank family is automatically complete too. So T is jointly complete sufficient.
Now, b0 and b1 are both linear functions of Σyi and Σxiyi (look again at the formulas in Step 1), and SSE — hence σ̂² — can be written entirely in terms of Σyi², Σyi, Σxiyi, b0 and b1. Because the map from (Σyi², Σyi, Σxiyi) to (b0, b1, σ̂²) is one-to-one (you can algebraically invert it and recover Σyi², Σyi, Σxiyi from b0, b1, σ̂²), sufficiency and completeness both carry over unchanged. So (b0, b1, σ̂²) is also jointly complete and sufficient for (β0, β1, σ²).Statement 1 is correct.
Step 3: Is Statement 2 True? Is (b0, b1) Independent of σ̂²?
Write the model in matrix form: y = Xβ + ε, where X is the n×2 matrix whose columns are a column of 1's and the xi's, and ε ~ N(0, σ²In). The fitted values are ŷ = Xb = Hy, where H = X(X′X)−1X′ is the "hat matrix" (a projection matrix). The residual vector is e = y − ŷ = (I − H)y.
Two useful facts about H, which follow directly from its definition:
H is idempotent: H² = H (projecting twice does nothing new).
Since b = (X′X)−1X′y and X′H = X′, the vector b is completely determined once you know Hy — you don't need any more information from y.
Now compute the covariance between Hy and (I − H)y (using Cov(y) = σ²I):
Cov(Hy, (I − H)y) = H · σ²I · (I − H)′ = σ²(H − H²) = σ²(H − H) = 0
Since y is a normal vector, any two linear functions of y that have zero covariance are automatically independent (this is a special property of the multivariate normal — uncorrelated jointly normal random vectors are independent, which is not true in general for non-normal data). So Hy and (I − H)y are independent. Because b is a function of Hy alone, and SSE = e′e = [(I − H)y]′[(I − H)y] is a function of (I − H)y alone, it follows that b (i.e., b0 and b1 together) is independent of SSE, and therefore independent of σ̂² too, since σ̂² is just SSE divided by a constant.Statement 2 is correct.
Pro Tip: The one place students trip up on this question is the wording "maximum likelihood estimators... denoted by b0, b1 and σ̂²." Read literally, the MLE of σ² divides SSE by n, not by (n − 2), and (n − 2)·(SSE/n)/σ² is only a
Step 4: Is Statement 3 True? Does (n − 2)σ̂²/σ² Follow a Chi-Square Distribution?
Continue with the matrix set-up from Step 3. Since (I − H)X = 0 (residualising the columns of X against themselves leaves nothing), we get
e = (I − H)y = (I − H)(Xβ + ε) = (I − H)ε
So the residual vector e depends only on the error vector ε, not on the unknown β at all. Then
SSE = e′e = ε′(I − H)′(I − H)ε = ε′(I − H)ε (using that I − H is symmetric and idempotent, so (I−H)′(I−H) = I − H)
There is a standard theorem for normal quadratic forms: if ε ~ N(0, σ²In) and A is a symmetric idempotent matrix of rank r, then ε′Aε/σ² follows a Chi-square distribution with r degrees of freedom. Here A = I − H, which is idempotent (because H is), and its rank equals n − rank(H). Since X has 2 linearly independent columns (the intercept column and the xi column, assuming the xi's are not all identical), rank(H) = 2, so rank(I − H) = n − 2.
Applying the theorem: SSE/σ² = ε′(I − H)ε/σ² ~ χ²n−2. Writing σ̂² = SSE/(n − 2) (per the convention pinned down in the Pro Tip above), this is exactly the same statement as (n − 2)σ̂²/σ² ~ χ²n−2.Statement 3 is correct.
Step 5: Final Answer
All three statements — joint complete sufficiency, independence of (b0, b1) from σ̂², and the chi-square distribution of the scaled residual variance — check out. That means the correct option is:
(d) 1, 2 and 3
This matches the official answer key for this question, so our independently worked derivation and the key are in full agreement — no discrepancy to flag here.
Why This Question Matters
These three results aren't just abstract theory — they are the exact machinery behind the t-test and F-test formulas used to test hypotheses like H0: β1 = 0 in regression, which is one of the most frequently tested ideas across this exam's Paper-2 (you'll notice it resurfaces in several nearby questions in this very paper). The independence result (Statement 2) is what lets you treat the numerator and denominator of a t-statistic as independent, which is required for that ratio to actually follow a t-distribution rather than something more complicated. The chi-square result (Statement 3) is what gives you the degrees of freedom you plug into t-tables and F-tables. And completeness (Statement 1) is the deeper reason the Gauss-Markov and Cramer-Rao-type optimality results for these estimators hold at all — a complete sufficient statistic is what the Lehmann-Scheffé theorem needs to guarantee an estimator is the best unbiased one available. Once you see these three facts as one connected package rather than three separate things to memorise, a whole cluster of MCQs across the regression portion of this syllabus becomes much easier to reason through from first principles instead of from recall.
Frequently Asked Questions
What's the difference between "sufficient" and "complete sufficient"?
A sufficient statistic captures all the information in the sample that's relevant to estimating the parameter — once you know it, the raw data adds nothing more. Completeness is a stronger, separate property: it means there is no non-trivial function of that statistic whose expected value is zero for every value of the parameter. A statistic can be sufficient without being complete, but when it is both, powerful results like the Lehmann-Scheffé theorem become available for constructing best unbiased estimators.
Why does the residual sum of squares have (n − 2) degrees of freedom instead of n?
You start with n independent residuals, but fitting the regression line uses up two pieces of information from the data — it forces two linear constraints on the residuals (equivalent to the normal equations from least squares). Each constraint removes one degree of freedom, so n observations minus 2 estimated parameters leaves (n − 2) degrees of freedom for the residual sum of squares.
Is the maximum likelihood estimator of σ² the same as the one used in a t-test for β1?
Not exactly. The strict MLE divides the residual sum of squares by n and is slightly biased downward. The estimator used in t-tests and confidence intervals divides by (n − 2) instead, which makes it unbiased for σ². Exam questions (including this one) sometimes use "σ̂²" loosely for both, so always check which divisor a particular formula in front of you is actually implying.
What is Basu's theorem, and how does it relate to Statement 2?
Basu's theorem says that if a statistic is complete and sufficient, then it is independent of any ancillary statistic (one whose distribution doesn't depend on the parameter at all). The independence in Statement 2 can also be argued this way, though the direct matrix-algebra route shown above — using the fact that uncorrelated jointly normal vectors are independent — is usually the more concrete way to see it for a beginner.
Why is (b0, b1) independent of σ̂² such an important result in practice?
It's what makes the standard t-statistic for testing β1 = 0 actually follow a t-distribution. A t-statistic is built as a standard normal variable divided by the square root of an independent chi-square variable (divided by its degrees of freedom). If b1 and σ̂² weren't independent, the ratio used in the test wouldn't have a clean, tabulated distribution at all.
Does this result change if the errors aren't normally distributed?
Yes, significantly. The independence in Statement 2 relies specifically on the property that uncorrelated jointly normal variables are independent — this doesn't hold for most non-normal distributions. Without normality, (b0, b1) and σ̂² can still be uncorrelated by the same algebra, but they generally won't be independent, and the exact chi-square and t-distribution results in Statements 2 and 3 no longer hold exactly (though large-sample approximations often still work reasonably well).
That's Question 16 done and dusted. If any step above felt rushed or you'd solve a piece of it differently, drop a comment below — working through the disagreement is usually where the real learning happens. And if you know a fellow ISS aspirant who's stuck on this same question, feel free to pass this along; a second pair of eyes on a proof like this one rarely hurts.
Comments