3. Distributions, Expectation, and Variance

What is a Random Variable?

A random variable is technically a mapping between outcomes in the sample space to a real number. The first random variable we examine is $X$, which is $1$ when heads is flipped and $0$ when tails is flipped. We have mapped both possible outcomes in our sample space to real numbers, so $X$ is a random variable.

We frequently categorize random variables into distributions. There are many families of distributions, each useful for describing different kinds of random variables. Our random variable $X$ belongs to the Bernoulli Distribution: the distribution of a “Bernoulli Trial,” an experiment with only two outcomes, success or failure, with some probability $p$. We formalize this as:

$X = \begin{cases}1 & \text{with probability } p \\ 0 & \text{with probability } 1-p\end{cases}$

Because we are focusing on coins, most of the random variables we work with will be discrete, meaning they take countably many values. For discrete random variables, we can define the probability mass function (pmf). Commonly denoted $f$, the pmf of a random variable $X$ taking values $\{x_1, x_2, x_3, \ldots\}$ is

$f_X(x)=\mathbb{P}(X=x)$

Applying this to our Bernoulli random variable gives $f_X(1) = p$ and $f_X(0) = 1-p$. Another random variable we have already encountered is the one that counts the number of heads flipped in $n$ coin flips. This belongs to the Binomial Distribution.

Question 1: What is the pmf of $X \sim \text{Binom}(n, p)$?

These two distributions, the Bernoulli and Binomial, are called parameterized because a set of parameters determines the shape of the distribution.

Question 2: How many parameters do the Bernoulli and Binomial distributions have?

We write $X \sim \text{Bern}(p)$ to denote that $X$ comes from the Bernoulli distribution with parameter $p$, and $X \sim \text{Binom}(n, p)$ for the Binomial with parameters $n$ and $p$. A collection of random variables drawn from the same distribution independently are called i.i.d. (independent and identically distributed).

Question 3: Given i.i.d. random variables $X_1, X_2, \ldots, X_n \sim \text{Bern}(p)$, prove that $\sum_{i=1}^{n} X_i \sim \text{Binom}(n, p)$.

Joint Distributions

Sometimes we look at the distribution of two random variables simultaneously using a joint distribution. A joint pmf describes the probability of two random variables taking specific values at the same time:

$f_{X,Y}(x,y) = \mathbb{P}(X=x, Y=y)$

The same rules that apply to ordinary distributions apply here: $f_{X,Y}(x,y) \geq 0$ and $\sum_{x} \sum_{y} f_{X,Y}(x,y) = 1$. Note, when I write $\sum_x$ or $\sum_{y}$, it means I want to sum over all possible outputs of the random variable $Y$

The explorer below lets you choose marginal distributions for $X$ and $Y$ independently and visualize the joint PMF that results when they are independent.

Question 4: Prove that random variables $X$ and $Y$ are independent if and only if $f_{X,Y}(x,y) = f_X(x) \cdot f_Y(y)$.

We have marginal distributions $f_X(x) = \mathbb{P}(X=x)$ and joint distributions $f_{X,Y}(x,y) = \mathbb{P}(X=x, Y=y)$. There is also the conditional probability mass function:

$f_{X|Y}(x, y) = \mathbb{P}(X=x \mid Y=y)$

Question 5: Write $f_{X,Y}(x,y)$ in terms of $f_Y(y)$ and $f_{X|Y}(x,y)$.

Question 6: Write $f_X(x)$ in terms of $f_{X,Y}(x,y)$.

Expected Value

The expected value of a discrete random variable is a weighted average of its possible values, where each value is weighted by the probability it occurs. Formally, for $X: \Omega \mapsto \mathbb{R}$:

$\mathbb{E}[X]=\sum_{x \in \mathbb{R}} x \cdot f(x) = \sum_{x \in \mathbb{R}} x \cdot \mathbb{P}(X=x)$

This connects to physics: the “first moment” of an object, its center of mass, is calculated in exactly the same way. The expected value is the point where, if you balanced the probability distribution on a fulcrum, it would sit perfectly level.

Applying this to a fair coin flip, with $X = 1$ for heads and $X = 0$ for tails:

Question 7: You go to a casino where two games each cost \5$. Game$A$flips a coin with$p = 1/100$of heads, paying out$$500$on heads. Game$B$flips a fair coin, paying out$$4$ on heads. What is the expected value of each game, and of playing both once?

Question 8: Prove that for any random variables $X$, $Y$ and constants $a$, $b$: $\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]$.

Question 9: Let $X \sim \text{Binom}(n,p)$. Compute $\mathbb{E}[X]$.

Question 10: Let $X \sim \text{Binom}(n,p)$ and let $f(x) = 2x$. (To be added.)

Question 11: What is the expected number of flips until you get your first heads, in terms of $p$?

LOTUS and Jensen’s Inequality

LOTUS

The humorously named Law of the Unconscious Statistician (LOTUS) states that for any function $g$:

$\mathbb{E}[g(X)] = \sum_{x} g(x) \cdot f(x)$

This feels like it should be obvious, but the reason it’s true is not. Let $Y = \{y : \exists\, x \in \Omega_X,\ g(x) = y\}$ be the range of $g$, and let $X_y = \{x \in \Omega_X \mid g(x) = y\}$. For any $y_i \neq y_j$, the sets $X_{y_i}$ and $X_{y_j}$ are disjoint, which enables:

Jensen’s Inequality

Given any convex function $\phi$, Jensen’s Inequality states that for any random variable $X$:

$\phi(\mathbb{E}[X]) \leq \mathbb{E}[\phi(X)]$

For a concave $\phi$, the inequality reverses. The interactive lab below lets you build geometric intuition: drag the control points to reshape $\phi$, click the $x$-axis to place probability mass, then click “Show Jensen’s” to compare $\phi(\mathbb{E}[X])$ and $\mathbb{E}[\phi(X)]$ directly.

Variance

The variance captures how spread out a distribution is around its mean:

$\mathbb{V}(X) = \mathbb{E}\!\left[(X - \mathbb{E}[X])^2\right]$

We center $X$ by subtracting its mean, square the result (so deviations above and below both contribute positively), then take the expected value of that squared distance. In physics, this is the centered second moment.

Question 12: Show that $\mathbb{V}(X) = \mathbb{E}[X^2] - \mathbb{E}[X]^2$.

For independent random variables $X_1, \ldots, X_n$ and constants $a_1, \ldots, a_n$:

$\mathbb{V}\!\left(\sum_{i=1}^{n} a_i X_i\right) = \sum_{i=1}^{n} a_i^2\, \mathbb{V}(X_i)$

Proof:

Question 13: What is the variance of $X \sim \text{Binom}(n, p)$?

Conditional Expected Value and Variance

Given random variables $X$ and $Y$, we can define conditional expectations $\mathbb{E}[Y \mid X]$ and conditional variance $\mathbb{V}(Y \mid X)$.

Question 14: You have two coins: coin 1 has $p = 0.25$ and coin 2 has $p = 0.50$. You pick one at random and flip it $100$ times. What is the expected number of heads?

Question 15: You flip a fair coin $100$ times, recording $h_1$ heads. You then build a weighted coin with $p = h_1/100$ and flip it $100$ more times. What is the expected number of heads from the second experiment?

Question 16: Prove that $\mathbb{E}[\mathbb{E}[Y \mid X]] = \mathbb{E}[Y]$. (Law of Total Expectation)

Question 17: Prove that $\mathbb{V}(Y) = \mathbb{E}[\mathbb{V}(Y \mid X)] + \mathbb{V}(\mathbb{E}[Y \mid X])$. (Law of Total Variance)

Covariance

(To be added.)

Normal and Poisson Distributions

We are already familiar with the Bernoulli and Binomial distributions. With small modifications to the Binomial, we get two more distributions that appear throughout science.

Normal Distribution

If we let $n \to \infty$ in $\text{Binom}(n, p)$ while keeping $p$ fixed, the distribution converges to the Normal distribution, parameterized by its mean $\mu$ and variance $\sigma^2$, written $\mathcal{N}(\mu, \sigma^2)$. Its importance cannot be overstated: the Normal distribution underlies much of modern statistical inference.

Poisson Distribution

If we let $n \to \infty$ while simultaneously letting $p \to 0$ such that $np = \lambda$ stays constant, we obtain the Poisson distribution, written $\text{Pois}(\lambda)$. It appears frequently in nature, often modeling the number of discrete events in a fixed time interval. The parameter $\lambda$ is both the mean and the variance.

Extra problem: Let $X \sim \text{Pois}(\lambda)$ and $Y \sim \text{Pois}(\mu)$ be independent. Show that the conditional distribution of $X$ given $X + Y = n$ is $\text{Binom}\!\left(n,\, \dfrac{\lambda}{\lambda + \mu}\right)$.