\(\newcommand{\Cov}{\mathrm{Cov}}\) \(\newcommand{\Corr}{\mathrm{Corr}}\) \(\newcommand{\Sample}{X_{1},\dots,X_{n}}\)
Assume you have a normal distribution with unknown \(\mu\) but known \(\sigma\) (highly implausible). A sample is collected. The 95% confidence interval for \(\mu\) is \(\left(\bar{x}-1.96\frac{\sigma}{\sqrt{n}},\bar{x}+1.96\frac{\sigma}{\sqrt{n}}\right)\)
This is the classical confidence interval (CI) (i.e. the frequentist).
The \(100(1-\alpha)\) % CI for the normal is \(\left(\bar{x}-z_{\alpha/2}\frac{\sigma}{\sqrt{n}},\bar{x}+z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right)\)
Deriving a Confidence Interval
Let \(\Sample\) be the sample, and \(\theta\) the quantity to be estimated. If you can find a random variable \(Y\) that satisfies:
- The random variable depends on \(\Sample\) and \(\theta\).
- The pdf of the random variable does not depend on \(\theta\) or any unknown parameters
Then you can find \(a,b\) such that \(P(a<Y<b)=1-\alpha\) (for any \(\alpha\)). Manipulate the expression to get \(P(A<\theta<B)=1-\alpha\). Then the CI for \(100(1-\alpha)\) is \((A,B)\).
Normal
Use \(Z=\frac{X-\mu}{\sigma/\sqrt{n}}\)
Exponential
Use \(Y=2\lambda\sum X_{i}\) (you get the chi-squared distribution).
Bootstrap Confidence Intervals
You can use bootstrapping to estimate the confidence interval.
As an example, let \(B=1000\). Calculate \(\hat{\theta_{i}^{\ast}}\) for all of them, as well as the mean of all of them. Note the differences \(\hat{\theta^{\ast}}-\hat{\theta_{i}^{\ast}}\). Then look at the 25th largest and smallest values and you have your interval.