Properties of Estimators — Unbiasedness, Efficiency, Consistency

What Are Properties of Estimators?

1. Unbiasedness

Worked Example: Bias of

For , the MLE is .

Compute the expectation:

Bias: .

Unbiased version: has .

2. Efficiency and the Cramér-Rao Lower Bound

Proof of the Cramér-Rao bound:

Let . For an unbiased estimator, so . Define the score function .

By the Cauchy-Schwarz inequality:

The left side equals (by the identity ). The right side has . Therefore .

Efficiency

3. Consistency

Consistency of the Sample Mean (Weak Law of Large Numbers)

4. Sufficiency and the Rao-Blackwell Theorem

Proof sketch: By the law of total variance: . The non-negative term vanishes if and only if is a function of alone.

5. Lehmann-Scheffé Theorem

Example: For , is complete sufficient. is unbiased for , so is the MVUE. (Note: is the MLE but is not unbiased.)

Python Simulation: Comparing Estimators

import numpy as np
from scipy import stats

np.random.seed(42)
n_values = [5, 10, 25, 50, 100, 500]
reps = 10000
true_mu = 5.0
true_sigma = 3.0

print("Comparing estimators for σ² (true = 9.0):")
print(f"{'n':>6} {'MLE (÷n)':>12} {'Unbiased (÷(n-1))':>18} {'MLE Bias':>10} {'Unbiased Bias':>15}")

for n in n_values:
    mle_vars = []
    unbiased_vars = []
    for _ in range(reps):
        data = np.random.normal(true_mu, true_sigma, n)
        mu_hat = np.mean(data)
        mle_vars.append(np.mean((data - mu_hat)**2))
        unbiased_vars.append(np.var(data, ddof=1))

    mle_mean = np.mean(mle_vars)
    unbiased_mean = np.mean(unbiased_vars)
    mle_bias = mle_mean - true_sigma**2
    unbiased_bias = unbiased_mean - true_sigma**2
    print(f"{n:6d} {mle_mean:12.4f} {unbiased_mean:18.4f} {mle_bias:10.4f} {unbiased_bias:15.4f}")

# Demonstrate WLLN
print("\nWLLN demonstration (convergence of X̄ to μ):")
for n in [10, 100, 1000, 10000]:
    data = np.random.normal(true_mu, true_sigma, n)
    xbar = np.mean(data)
    print(f"  n={n:5d}: X̄ = {xbar:.4f} (error = {abs(xbar - true_mu):.4f})")

Key Takeaways