Statistical Methods III: Week-2

🧠 Cramér–Rao Bound for Vector Parameters

Excellent — you’re now stepping into the multivariate (vector parameter) version of the Cramér–Rao Lower Bound (CRLB), which generalizes the scalar case using matrix calculus and Fisher information matrices.

Let’s break it down.

Let:

• \(\boldsymbol{\theta} \in \mathbb{R}^k\): vector parameter
• \(\mathbf{X} \sim f(\mathbf{x}|\boldsymbol{\theta})\)
• \(\mathbf{W}(\mathbf{X}) \in \mathbb{R}^m\): vector-valued statistic
• \(\mathbf{W}(\mathbf{X})\) has finite variance and differentiable expectation w.r.t. \(\boldsymbol{\theta}\)

📘 Multivariate CRLB (Matrix Form):

📌 Notation Breakdown:

• \(\operatorname{Var}_{\boldsymbol{\theta}}(\mathbf{W}) \in \mathbb{R}^{m \times m}\): Covariance matrix of the estimator
• \(\frac{d}{d\boldsymbol{\theta}} \mathbb{E}_{\boldsymbol{\theta}}[\mathbf{W}] \in \mathbb{R}^{m \times k}\): Jacobian matrix of expectation
• \(\mathcal{I}(\boldsymbol{\theta}) \in \mathbb{R}^{k \times k}\): Fisher Information matrix
• \(\succeq\): Matrix inequality meaning the difference is positive semi-definite

⚙️ Meaning:

This inequality tells you that the covariance matrix of any estimator of a vector parameter is bounded from below by a matrix formed from:

• The sensitivity of the estimator (Jacobian of expectation)
• The inverse Fisher Information matrix

✅ Special Case: Unbiased Estimator

If \(\mathbb{E}_{\boldsymbol{\theta}}[\mathbf{W}(\mathbf{X})] = \boldsymbol{\theta}\), then:

• Jacobian = Identity matrix \(I_k\)

So, the bound becomes:

Just like in the scalar case, but extended to full matrix structure.

📘 Fisher Information Matrix \(\mathcal{I}(\boldsymbol{\theta})\)

Let’s now define the Fisher Information Matrix \(\mathcal{I}(\boldsymbol{\theta})\) in the vector parameter case.

Let:

• \(\boldsymbol{\theta} \in \mathbb{R}^k\): vector parameter
• \(\mathbf{X} \sim f(\mathbf{x} | \boldsymbol{\theta})\)
• Assume \(f(\mathbf{x} | \boldsymbol{\theta})\) satisfies suitable regularity conditions (e.g., to allow differentiation under the integral)

✅ The Fisher Information Matrix is defined as:

🔍 Where:

• \(\nabla_{\boldsymbol{\theta}} \log f(\mathbf{X}|\boldsymbol{\theta}) \in \mathbb{R}^k\): the score vector \[ \nabla_{\boldsymbol{\theta}} \log f(\mathbf{X}|\boldsymbol{\theta}) = \begin{bmatrix} \frac{\partial}{\partial\theta_1} \log f(\mathbf{X}|\boldsymbol{\theta}) \\ \vdots \\ \frac{\partial}{\partial\theta_k} \log f(\mathbf{X}|\boldsymbol{\theta}) \end{bmatrix} \]
• The outer product produces a \(k \times k\) matrix
• So, the Fisher Information Matrix is a positive semi-definite matrix of size \(k \times k\)

🧠 Alternate Form (if second derivatives are easier to compute):

Under stronger regularity conditions, we can also write:

Here, \(\nabla_{\boldsymbol{\theta}}^2 \log f(\mathbf{X}|\boldsymbol{\theta}) \in \mathbb{R}^{k \times k}\) is the Hessian matrix of second-order partial derivatives.

🧠 Cramér–Rao Lower Bound: Vector Parameter Case

Here's the full form of the Cramér–Rao Lower Bound (CRLB) in the vector parameter case, complete with all definitions, assumptions, and mathematical structure.

Let:

• \(\boldsymbol{\theta} \in \mathbb{R}^k\): vector-valued parameter
• \(\mathbf{X} \in \mathbb{R}^n\): random sample with joint pdf (or pmf) \(f(\mathbf{x} | \boldsymbol{\theta})\)
• \(\mathbf{W}(\mathbf{X}) \in \mathbb{R}^m\): statistic estimating a function of \(\boldsymbol{\theta}\)
• All relevant expectations, gradients, and Fisher Information are well-defined and finite

📘 General Matrix Form of the CRLB:

🧩 Term Definitions:

• \(\operatorname{Cov}_{\boldsymbol{\theta}}(\mathbf{W}) \in \mathbb{R}^{m \times m}\): Covariance matrix of the estimator
• \(\frac{\partial}{\partial \boldsymbol{\theta}} \mathbb{E}_{\boldsymbol{\theta}}[\mathbf{W}] \in \mathbb{R}^{m \times k}\): Jacobian matrix of expected estimator wrt \(\boldsymbol{\theta}\)
• \(\mathcal{I}(\boldsymbol{\theta}) \in \mathbb{R}^{k \times k}\): Fisher Information Matrix

🎯 Fisher Information Matrix Definition:

Let the score vector be:

Then:

Or, under regularity conditions:

✅ Special Case: Unbiased Estimator

If \(\mathbb{E}_{\boldsymbol{\theta}}[\mathbf{W}(\mathbf{X})] = \boldsymbol{\theta}\), then the Jacobian is identity, and:

This matches the scalar case:

🧠 Setup: Vector CRLB with Fisher Info Substituted

• \(\boldsymbol{\theta} \in \mathbb{R}^p\): vector parameter
• \(\mathbf{X}\): data with PDF/PMF \(f(\mathbf{x}|\boldsymbol{\theta})\)
• \(\mathbf{W}(\mathbf{X}) \in \mathbb{R}^k\): unbiased estimator of \(\boldsymbol{\psi}(\boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}}[\mathbf{W}(\mathbf{X})]\)

✅ CRLB (Multivariate Version):

🔍 Explanation of Components:

• \(\frac{ \partial \boldsymbol{\psi}(\boldsymbol{\theta}) }{ \partial \boldsymbol{\theta} } \in \mathbb{R}^{k \times p}\): Jacobian of \(\boldsymbol{\psi}(\boldsymbol{\theta})\)
• Middle matrix is the Fisher Information Matrix: \[ \mathbf{I}(\boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}} \left[ \left( \frac{ \partial }{ \partial \boldsymbol{\theta} } \log f(\mathbf{X}|\boldsymbol{\theta}) \right) \left( \frac{ \partial }{ \partial \boldsymbol{\theta} } \log f(\mathbf{X}|\boldsymbol{\theta}) \right)^\top \right] \]

📌 Summary:

The multivariate CRLB states:

This is a matrix inequality: the LHS is the covariance matrix of the estimator, and the RHS is the lower bound (in the positive semi-definite sense).

✨ Information Equality and Fisher Information Identity

You're diving into the information equality result, which is foundational for the Cramér–Rao lower bound (CRLB). It links the variance of the score function with the expected curvature of the log-likelihood.

Let us go through each step in full clarity, with notation, interpretation, and a diagram at the end.

🔶 The Setup

• \( f(x \mid \theta) \) : parametric family of densities
• Regularity conditions assumed (interchanging integration and differentiation allowed)
• \( x \) is a random variable drawn from $f(x \mid \theta)$
• \( \theta \in \mathbb{R} \): scalar parameter

🧰 Step 1: Expectation of the Score Function is Zero

This is a standard identity under regularity conditions:

Conclusion: the expected score is zero.

🔹 Step 2: Variance of Score Function = Negative Expectation of Second Derivative

The claim is:

📌 LHS: Variance of the Score Function

📌 RHS: Negative Expectation of Second Derivative

Now, under regularity:

So the result becomes:

Conclusion:

📊 Visual Interpretation

Let \( \ell(\theta) = \log f(x \mid \theta) \). Then:

• \( \ell'(\theta) \): score function — slope of log-likelihood
• \( \ell''(\theta) \): curvature — measures peakness/flatness
• Taking expectation, the curvature = variance of slope (negated)

ℹ Fisher Information

This quantifies how much information one observation gives about \(\theta\).

📆 Summary

• This is the Fisher Information identity
• Backbone of the Cramér–Rao Inequality
• Relies on regularity conditions (to switch integral and derivative)
• Captures the connection between score and curvature

🔷 Step-by-Step Breakdown: The Score Function

Let’s walk through it clearly, with intuition, derivation, and a small example.

📘 Log-Likelihood Expression

You wrote:

This is the log-likelihood function for \( n \) i.i.d. observations from the model \( f(x \mid \theta) \).

🧠 Why Sum of Logs?

Because:

Taking the log simplifies multiplication into summation.

🔧 Score Function (Gradient of Log-Likelihood)

Now differentiate the log-likelihood with respect to \( \theta \):

Using the chain rule:

This is the score function, often denoted:

✅ Your symbolic expression is correct — it's the score function for the full sample.

✅ Summary

This is crucial for:

• Maximum Likelihood Estimation (MLE): Set score = 0 and solve for \( \theta \)
• Fisher Information for \( n \) samples: \( \mathcal{I}_n(\theta) = n \mathcal{I}(\theta) \)
• Asymptotic theory of MLEs (e.g., consistency, normality)

📊 Small Example: Normal Model with Known Variance

Suppose:

• \( X_1, \dots, X_n \sim \mathcal{N}(\mu, \sigma^2) \)
• \( \sigma^2 \) known; estimate \( \mu \)

Then:

Summing over \( i \):

Differentiating:

Set derivative = 0 for MLE:

✅ So the MLE for \( \mu \) is the sample mean \( \bar{x} \).

🔷 Newton–Raphson for MLE: Likelihood Maximization

You're now discussing numerical optimization of the likelihood using the Newton–Raphson method, one of the core techniques to compute Maximum Likelihood Estimators (MLEs).

We'll break this into four parts for clarity:

🔹 1. Understanding the Graph of \( L(\theta) \) vs \( \theta \)

The graph of the log-likelihood function \( L(\theta) \) often looks like a smooth, concave curve. The peak of this curve corresponds to the MLE \( \hat{\theta} \).

        ^
     L(θ)
        |          /\
        |         /  \
        |        /    \
        |_______/______\________>
                    θ̂
  

At the maximum, the slope \( \frac{d}{d\theta} L(\theta) \) is zero — this is the first-order optimality condition.

🔧 2. Gradient Ascent (Steepest Ascent)

The update:

This is a gradient ascent step, where:

• \( \eta_r \) is a step size (learning rate)
• \( \nabla_\theta L(\theta) \) is the gradient (score function)
• You’re moving in the direction of increasing likelihood

This is not Newton–Raphson yet — just basic gradient ascent.

🧠 3. Newton–Raphson Update

Newton–Raphson improves on gradient ascent by incorporating the second derivative:

• Gradient: \( \nabla_\theta L(\theta) \)
• Hessian: \( \nabla^2_\theta L(\theta) \)

The update rule becomes:

✅ The minus sign reflects that for maximizing a concave function, you move against the curvature direction.

📌 What's the Intuition?

• The first derivative tells you the direction to move
• The second derivative tells you how much to move (curvature)

   ^
L(θ)|         *
     |        ^
     |       /|\
     |      / | \    ← Newton step lands at the peak
     |_____/__|__\________>
               θ
  

💡 Newton vs Gradient Ascent

For well-behaved likelihoods (e.g., Normal), Newton–Raphson can converge in just a few steps.

🔢 Example: Normal Model with Known Variance

Suppose:

• \( X_1, \dots, X_n \sim \mathcal{N}(\mu, \sigma^2) \)
• \( \sigma^2 \) is known; find MLE for \( \mu \)

Log-likelihood:

Then:

• \( \nabla_\mu L = \frac{1}{\sigma^2} \sum (x_i - \mu) = \frac{n}{\sigma^2} (\bar{x} - \mu) \)
• \( \nabla^2_\mu L = -\frac{n}{\sigma^2} \)

Newton–Raphson update:

✅ It converges in one step — because the log-likelihood is quadratic!

✅ Summary

• You’re using gradient ascent to maximize \( L(\theta) \)
• Newton–Raphson is more efficient by using curvature (Hessian)
• It’s standard in MLE, logistic regression, Poisson models, etc.

On the Direction of Steepest Ascent:

🔷 1. Why the Direction of Steepest Ascent?

Imagine the log-likelihood function \( L(\theta) \) as a landscape — a smooth surface over the parameter space.
Your goal: reach the top (maximum likelihood estimate \( \hat{\theta} \)).

At any point \( \theta_r \), the gradient \( \nabla_\theta L(\theta_r) \) is a vector that:

• Points in the direction where \( L(\theta) \) increases most rapidly.
• Has a magnitude equal to the rate of increase in that direction.

So, to climb the hill toward the MLE, you step in the direction of the gradient:

Here, \( \eta_r > 0 \) is your step size — how far you move in that direction.

🔶 2. What Makes Gradient the Steepest Direction?

Let’s be precise:

Suppose you want to increase \( L(\theta) \) by moving a small distance \( \delta \theta \). The first-order change is:

We want to choose \( \delta \theta \) of unit length (i.e., \( \| \delta \theta \| = 1 \)) that gives maximum increase:

The solution to this maximization is:

i.e., move in the direction of the gradient. ✅

📊 Visual Intuition (2D Case)

Think of the log-likelihood surface like a hill in 3D.

• The gradient vector at your current point points uphill in the steepest direction.
• If you move orthogonally to the gradient, you stay at the same level (iso-likelihood curve).
• Moving against the gradient leads you downhill — which you want to avoid during maximization.

      ^
     /|\\         ← Gradient points this way
    / |
   /  |             Log-Likelihood surface
  *---→
Current θ_r
  

🔁 Comparison with Newton–Raphson

• Steepest ascent: always follows the direction of greatest immediate gain.
• Newton–Raphson: modifies the direction based on the curvature, sometimes avoiding overshooting or zigzagging.

So, in practice:

• Gradient ascent is more robust, especially when the curvature is poorly behaved or Hessians are hard to compute.
• Newton–Raphson is faster when curvature information is reliable and the function is well-behaved (e.g., Gaussian likelihood).

✅ Summary

• We prefer the direction of steepest ascent because it guarantees the fastest local increase in the log-likelihood.
• The gradient gives this direction.
• The step size \( \eta_r \) controls how aggressively we move — too small = slow; too large = unstable.
• This approach is the foundation for many modern learning algorithms, including stochastic gradient ascent in MLE for big data or machine learning.

🔷 Connecting Fisher Information with the Hessian Matrix

Now connecting Fisher’s Information Matrix with the Hessian matrix of the log-likelihood function.

Let’s unpack this carefully and precisely.

🔷 1. Fisher Information Matrix (Multivariate Version)

Suppose \( \theta \in \mathbb{R}^k \) is a vector parameter and \( f(x \mid \theta) \) is a probability density function satisfying standard regularity conditions.

Then, the Fisher Information Matrix is defined as:

• This is a \( k \times k \) positive semi-definite matrix.
• It measures how much information the data carries about each component of \( \theta \), and how the components interact.

🔶 2. Alternative Definition Using the Hessian (Second Derivatives)

Under regularity conditions, the Fisher Information Matrix can also be written as the negative expected Hessian of the log-likelihood function:

✅ This is what your note says:

\( \mathbb{E}\left( -\frac{ \partial^2 }{ \partial \theta \partial \theta^\top } \log f(x \mid \theta) \right) = \text{Fisher Information Matrix} \)

📌 3. What Is the Hessian Matrix?

The Hessian of a scalar-valued function \( L(\theta) \), where \( \theta \in \mathbb{R}^k \), is the matrix of second partial derivatives:

So for the log-likelihood function \( \log f(x \mid \theta) \), the Hessian is:

It’s a \( k \times k \) symmetric matrix, assuming smoothness.

🔷 4. The (i,j)-th Element of Fisher Info = Expected Negative Hessian Entry

You correctly stated:

(i, j)-th element of the Fisher Information matrix is the (i, j)-th element of the expected negative Hessian of \( \log f(x \mid \theta) \)

In symbols:

This captures the curvature of the likelihood in the \( (\theta_i, \theta_j) \) direction.

🧠 Why Two Equivalent Definitions?

Because of the Fisher identity:

Both are equal only under regularity conditions, where we’re allowed to exchange integration and differentiation.

✅ Summary

🔷 1. Background: Newton–Raphson in MLE

Recall that for Maximum Likelihood Estimation (MLE), we want to maximize the log-likelihood function:

$$L(\theta) = \sum_{i=1}^n \log f(x_i \mid \theta)$$

The Newton–Raphson update rule for finding the root of $\nabla_\theta L(\theta) = 0$ is:

$$\theta^{(r+1)} = \theta^{(r)} - \left[ H(\theta^{(r)}) \right]^{-1} \nabla_\theta L(\theta^{(r)})$$

• $\nabla_\theta L(\theta)$ is the score function
• $H(\theta) = \nabla_\theta^2 L(\theta)$ is the Hessian matrix

🔷 2. Problem with Newton–Raphson

In practice, $H(\theta)$ may:

• Be difficult to compute or invert
• Be non-invertible at some points
• Lead to unstable convergence

🔷 3. Fisher’s Scoring: Motivation

To solve these issues, we replace the observed Hessian \( H(\theta) \) with its expectation:

$$\mathcal{I}(\theta) = \mathbb{E}_\theta\left[-\nabla_\theta^2 L(\theta)\right]$$

This is the Fisher Information Matrix. It's often easier to compute and is positive semi-definite.

🔷 4. Fisher’s Scoring Update Rule

The update rule becomes:

$$\theta^{(r+1)} = \theta^{(r)} + \left[\mathcal{I}(\theta^{(r)})\right]^{-1} \nabla_\theta L(\theta^{(r)})$$

🔁 Notice:

• Update direction uses the gradient (score)
• Scaled by inverse Fisher Information, not Hessian

🔷 5. Visualization: Geometric Insight

Imagine you’re on a hill (log-likelihood surface):

• Gradient Ascent: step in steepest direction
• Newton–Raphson: adjusts for curvature (Hessian)
• Fisher Scoring: uses average curvature — smoother steps

🔷 6. Example: Bernoulli Distribution

Let \( X_1, \dots, X_n \sim \text{Bern}(p)\) where \( 0 < p < 1 \)

Log-likelihood:

$$L(p) = \sum_{i=1}^n \left[ x_i \log p + (1 - x_i)\log(1 - p) \right]$$

Score function:

$$\frac{dL}{dp} = \frac{n\bar{X}}{p} - \frac{n(1 - \bar{X})}{1 - p}$$

Observed Information:

$$H(p) = -\frac{n\bar{X}}{p^2} - \frac{n(1 - \bar{X})}{(1 - p)^2}$$

Fisher Information:

$$\mathcal{I}(p) = \frac{n}{p(1 - p)}$$

Fisher Scoring update:

$$p^{(r+1)} = p^{(r)} + \left[ \frac{n}{p^{(r)}(1 - p^{(r)})} \right]^{-1} \left( \frac{n\bar{X}}{p^{(r)}} - \frac{n(1 - \bar{X})}{1 - p^{(r)}} \right)$$

Which simplifies to:

$$p^{(r+1)} = \bar{X}$$

🎯 That’s the MLE! Fisher scoring converges to it in one step for Bernoulli.

🔷 7. Comparison Table

🔷 8. When to Use Fisher Scoring?

✅ Use it when:

• Likelihood function is well-behaved but observed Hessian is messy
• Working with GLMs — Fisher scoring is standard for IRLS
• You need stable convergence with positive curvature