Statistical Methods III: Assignment Week-1

Question:

Let \( X_1, X_2, \ldots, X_n \) be i.i.d. random variables with pdf \[ f(x \mid \theta) = \frac{x^{1/\theta - 1}}{\theta}, \quad 0 \le x \le 1,\ \theta > 0. \] Is the maximum likelihood estimator (MLE) of \( \theta \) unbiased?

Solution:

The likelihood is: \[ L(\theta) = \frac{1}{\theta^n} \left( \prod_{i=1}^n X_i \right)^{1/\theta - 1} \quad \Rightarrow \quad \ell(\theta) = -n \log \theta + \left( \frac{1}{\theta} - 1 \right) \sum \log X_i \] Differentiate: \[ \frac{d\ell}{d\theta} = -\frac{n}{\theta} - \frac{S}{\theta^2} \quad \Rightarrow \quad \hat{\theta}_{\text{MLE}} = -\frac{1}{n} \sum \log X_i \] To check unbiasedness: \[ \mathbb{E}[\hat{\theta}_{\text{MLE}}] = -\mathbb{E}[\log X] = - \frac{1}{\theta} \int_0^1 x^{1/\theta - 1} \log x\, dx = - \frac{1}{\theta} \cdot (-\theta^2) = \theta \] ✔️ The MLE is unbiased.

Related concepts / Other stuff:

• MLE (Maximum Likelihood Estimation): Estimator derived by maximizing the likelihood function.
• Unbiased Estimator: Satisfies \( \mathbb{E}[\hat{\theta}] = \theta \).
• Useful identity: \( \int_0^1 x^{a - 1} \log x\, dx = -1/a^2 \).
• Transformation Insight: \( -\log X \sim \text{Exp}(1/\theta) \) is key to understanding the unbiasedness.
• Rare Case: MLE here is unbiased for all \( n \), not just asymptotically.

Question:

In the previous problem, we found that the MLE of \( \theta \) is:

\[ \hat{\theta}_{\text{MLE}} = -\frac{1}{n} \sum_{i=1}^{n} \log X_i. \]

What is the variance of \( \hat{\theta}_{\text{MLE}} \)?
Does the variance go to zero as \( n \to \infty \)?

Solution:

Step 1: Recall the MLE expression

\[ \hat{\theta}_{\text{MLE}} = -\frac{1}{n} \sum_{i=1}^{n} \log X_i. \] Define \( Y_i = -\log X_i \). Then: \[ \hat{\theta}_{\text{MLE}} = \frac{1}{n} \sum_{i=1}^n Y_i. \] So, \[ \text{Var}(\hat{\theta}_{\text{MLE}}) = \frac{1}{n^2} \sum_{i=1}^n \text{Var}(Y_i) = \frac{1}{n} \text{Var}(Y_1). \]

Step 2: Distribution of \( Y = -\log X \)

Given: \( X \sim f(x|\theta) = \frac{x^{1/\theta - 1}}{\theta},\ 0 \le x \le 1 \)
Let \( X = e^{-Y} \), then \( dx = -e^{-Y} dY \) \[ f_Y(y) = f_X(x(y)) \cdot \left| \frac{dx}{dy} \right| = \frac{e^{-(1/\theta - 1)y}}{\theta} \cdot e^{-y} = \frac{e^{-y/\theta}}{\theta}, \quad y \ge 0. \] Thus, \( Y = -\log X \sim \text{Exponential}\left( \frac{1}{\theta} \right) \)

Step 3: Mean and Variance of Exponential Distribution

If \( Y \sim \text{Exp}(\lambda) \), then:

• \( \mathbb{E}[Y] = \frac{1}{\lambda} \)
• \( \text{Var}(Y) = \frac{1}{\lambda^2} \)

• \( \mathbb{E}[Y] = \theta \)
• \( \text{Var}(Y) = \theta^2 \)

Final Answer:

\[ \boxed{ \text{Var}(\hat{\theta}_{\text{MLE}}) = \frac{\theta^2}{n} } \]

Does variance → 0 as \( n \to \infty \)?

Yes — clearly: \[ \lim_{n \to \infty} \frac{\theta^2}{n} = 0 \] So, the MLE is consistent.

Related concepts / Other stuff:

• Variance of Sample Mean: If \( Y_1, \ldots, Y_n \) are i.i.d. with variance \( \sigma^2 \), then \( \text{Var}(\bar{Y}) = \frac{\sigma^2}{n} \)
• Consistency of Estimators: An estimator \( \hat{\theta}_n \) is consistent if \( \hat{\theta}_n \xrightarrow{P} \theta \). True if unbiased and variance → 0.

🧠 Visual Intuition:

As \( n \) increases, the sample mean of i.i.d. exponential variables concentrates around the true mean \( \theta \). The variance shrinks as \( 1/n \), leading to a tighter distribution around \( \theta \). Imagine a histogram of \( \hat{\theta}_{\text{MLE}} \) values from repeated sampling — it gets narrower around \( \theta \) as \( n \) grows.

💡 Worth Mentioning:

• This is a rare case where the MLE is unbiased, has finite variance, and is consistent — all in closed form.
• Transformation via \( -\log X \) is a classic estimation technique.
• This MLE is also asymptotically efficient (achieves CRLB).

Question:

Let \( X_1, X_2, \ldots, X_n \) be i.i.d. exponential random variables with mean \( \theta \), i.e.,

\[ f(x \mid \theta) = \frac{1}{\theta} e^{-x/\theta}, \quad x \ge 0. \]

Find an unbiased estimator of \( \theta \) that depends only on \( \min\{X_1, X_2, \ldots, X_n\} \).

Solution:

Step 1: Let’s define the statistic

Let: \[ Y = \min\{X_1, X_2, \ldots, X_n\} \] We aim to find a constant \( c_n \) such that: \[ \mathbb{E}[c_n Y] = \theta \quad \Rightarrow \quad \text{then } c_n Y \text{ is unbiased.} \] So we must find \( c_n = \frac{1}{\mathbb{E}[Y]} \cdot \theta \).

Step 2: Distribution of the Minimum

Let \( Y = X_{(1)} = \min\{X_1, \ldots, X_n\} \).
Then for exponential \( \text{Exp}(\theta) \), the minimum has the distribution: \[ Y \sim \text{Exponential}\left(\frac{1}{\theta} \cdot n\right) \] That is: \[ f_Y(y) = \frac{n}{\theta} e^{-ny/\theta}, \quad y \ge 0. \] So: \[ \mathbb{E}[Y] = \frac{\theta}{n} \]

Step 3: Construct Unbiased Estimator

We want: \[ \mathbb{E}[c_n Y] = \theta \Rightarrow c_n \cdot \frac{\theta}{n} = \theta \Rightarrow c_n = n \]

Final Answer:

\[ \boxed{ \hat{\theta} = n \cdot \min\{X_1, \ldots, X_n\} } \] This is an unbiased estimator of \( \theta \) using only the minimum.

Related concepts / Other stuff:

• Exponential Distribution and Order Statistics:
  • The minimum of \( n \) i.i.d. \( \text{Exp}(\theta) \) is again exponential, with rate \( \frac{n}{\theta} \).
  • This is because the exponential distribution has the memoryless property.
• Sufficiency and Completeness:
  • The full data \( X_1, \ldots, X_n \) gives the sufficient statistic \( \sum X_i \), but the minimum alone is not sufficient.
  • Still, one can construct an unbiased estimator from a function of a statistic, even if it's not sufficient.

🧠 Visual Intuition:

Imagine sampling 100 exponential lifetimes, and just recording the first failure time. Since failures happen randomly, and exponentially, the first failure tends to occur faster than the average lifetime.

Hence, the minimum is biased low, but scaling it up by \( n \) compensates for this — making it an unbiased estimate.

Think of \( \min(X_1, \dots, X_n) \approx \frac{\theta}{n} \), so multiplying by \( n \) gets you back to \( \theta \).

💡 Worth Mentioning:

• This estimator is not the most efficient (in terms of variance). The MLE \( \hat{\theta}_{\text{MLE}} = \bar{X} \) has lower variance.
• But this question is important in contexts where only partial information is available — like censoring or online streaming.
• The estimator \( n \cdot X_{(1)} \) is minimal sufficient in some practical settings, like reliability theory.

Question:

Let \( X_1, X_2, \ldots, X_n \) be i.i.d. with distribution \( \mathcal{N}(\theta, \theta^2) \), where \( \theta > 0 \).

Define:

• \( \bar{X} = \frac{1}{n} \sum_{i=1}^n X_i \) (sample mean)
• \( S^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X})^2 \) (sample variance)

For which value of \( c \) is \( cS \) an unbiased estimator of \( \theta \)?

Solution:

We are to find \( c \) such that:

\[ \mathbb{E}[cS] = \theta. \]

So, we need:

\[ c = \frac{\theta}{\mathbb{E}[S]} \]

We know:

• \( X_i \sim \mathcal{N}(\theta, \theta^2) \)
• \( \text{Var}(X_i) = \theta^2 \)

The sample variance:

\[ S^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X})^2 \]

is an unbiased estimator of \( \theta^2 \):

\[ \Rightarrow \mathbb{E}[S^2] = \theta^2 \]

But:

\[ S = \sqrt{S^2} \Rightarrow \mathbb{E}[S] \ne \sqrt{\mathbb{E}[S^2]} \Rightarrow \mathbb{E}[S] < \theta \]

So we must explicitly calculate \( \mathbb{E}[S] \) when \( X_i \sim \mathcal{N}(\theta, \theta^2) \).

🔹 Step 1: Use Chi-Square Distribution

We know that for \( X_i \sim \mathcal{N}(\mu, \sigma^2) \), the statistic:

\[ \frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1} \]

This holds even if \( \mu \ne 0 \).

Here, \( \sigma^2 = \theta^2 \), so:

\[ \frac{(n-1)S^2}{\theta^2} \sim \chi^2_{n-1} \Rightarrow S = \theta \cdot \sqrt{\frac{1}{n-1} \cdot \chi^2_{n-1}} \]

Let:

\[ Z = \chi^2_{n-1}, \quad \Rightarrow S = \theta \cdot \sqrt{\frac{Z}{n-1}} \]

So:

\[ \mathbb{E}[S] = \theta \cdot \mathbb{E}\left[\sqrt{\frac{Z}{n-1}}\right] = \theta \cdot \frac{1}{\sqrt{n-1}} \cdot \mathbb{E}\left[\sqrt{Z}\right] \]

We now need the expected value of \( \sqrt{Z} \), where \( Z \sim \chi^2_{n-1} \).

🔹 Step 2: Known Result: Expected value of square root of chi-square

For \( Z \sim \chi^2_k \), there’s a known result:

\[ \mathbb{E}[\sqrt{Z}] = \sqrt{2} \cdot \frac{\Gamma\left(\frac{k+1}{2}\right)}{\Gamma\left(\frac{k}{2}\right)} \]

Apply this to \( Z \sim \chi^2_{n-1} \):

\[ \mathbb{E}[S] = \theta \cdot \frac{1}{\sqrt{n-1}} \cdot \mathbb{E}[\sqrt{Z}] = \theta \cdot \frac{\sqrt{2}}{\sqrt{n-1}} \cdot \frac{\Gamma\left(\frac{n}{2}\right)}{\Gamma\left(\frac{n-1}{2}\right)} \]

🔹 Step 3: Solve for \( c \)

We want:

\[ \mathbb{E}[cS] = \theta \Rightarrow c \cdot \mathbb{E}[S] = \theta \Rightarrow c = \frac{\theta}{\mathbb{E}[S]} \]

Substitute:

\[ \mathbb{E}[S] = \theta \cdot \frac{\sqrt{2}}{\sqrt{n-1}} \cdot \frac{\Gamma\left(\frac{n}{2}\right)}{\Gamma\left(\frac{n-1}{2}\right)} \]

Cancel \( \theta \), we get:

\[ c = \boxed{ \frac{\sqrt{n-1}}{\sqrt{2}} \cdot \frac{\Gamma\left(\frac{n-1}{2}\right)}{\Gamma\left(\frac{n}{2}\right)} } \]

Final Answer:

\[ \boxed{ c = \frac{\sqrt{n - 1}}{\sqrt{2}} \cdot \frac{\Gamma\left( \frac{n - 1}{2} \right)}{\Gamma\left( \frac{n}{2} \right)} } \quad \text{makes } cS \text{ an unbiased estimator of } \theta. \]

Related concepts / Other stuff:

• Chi-square and Gamma functions:
  • If \( Z \sim \chi^2_k \), then it’s a special case of \( \text{Gamma}(k/2, 1/2) \)
  • \( \mathbb{E}[\sqrt{Z}] \) involves non-linear expectations → handled with gamma functions.
• Bias of the Sample Standard Deviation:
  • \( \mathbb{E}[S] < \sqrt{\text{Var}(X)} \)
  • This is because square root is concave → Jensen’s inequality implies bias downward.

🧠 Visual Intuition:

• The sample variance \( S^2 \) is unbiased for \( \theta^2 \)
• But taking the square root pulls the expectation down, because of concavity.
• To compensate, you scale up by \( c > 1 \), computed exactly via gamma ratios.

💡 Worth Mentioning:

• This is a rare exact correction for bias — many times people use approximate corrections (e.g., in small-sample t-tests).
• For large \( n \), \( c \to 1 \), so \( S \approx \theta \), and bias becomes negligible.
• This \( cS \) is not MVUE — \( \bar{X} \) is a better candidate due to sufficiency and completeness, but it doesn’t depend solely on \( S \).