Expectation, variance and inequalities

The lesson "The Lebesgue integral" defined $\int f\,d\mu$ for any measurable function on any measure space, and "Random variables and their distributions" turned a random experiment into a number $X$ with a distribution $\mathbb{P}_X$ on the real line. This lesson joins the two. The expectation of a random variable is its Lebesgue integral under the probability measure, so everything proved about integrals earlier becomes a statement about averages of random quantities. Three questions drive the lesson: what a number such as $3.5$ for a die really says, how to compute it without going back to the underlying space $\Omega$, and how far a random variable can stray from its mean. The answers are the formula for $\mathbb{E}[g(X)]$, the variance, and three inequalities, Markov's, Chebyshev's and Jensen's, which turn a mean and a variance into guarantees about probabilities.

Expectation as an integral

Let $X$ be a random variable on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$. Its expectation, also called its mean or expected value, is the Lebesgue integral of $X$ against $\mathbb{P}$:

For a simple random variable $X = \sum_k a_k \mathbf{1}_{A_k}$, the integral of "The Lebesgue integral" is the sum $\sum_k a_k\,\mathbb{P}(A_k)$: each value $a_k$ is weighted by the probability of the event on which it is taken. A fair die is the model case. Take $\Omega = \{1, \ldots, 6\}$ with mass $\tfrac16$ on each face and $X(\omega) = \omega$. Then

The number $3.5$ can be read in two ways, and both are worth keeping. The first is the long-run average: roll the die many times and the average of the results settles near $3.5$. A later lesson of this track proves this as the law of large numbers; here it serves as the intuition. The second reading is a balance point. Put the distribution of $X$ on a beam as sand, a pile of mass $\tfrac16$ at each of $1, \ldots, 6$, and the beam balances on a pivot at $3.5$. The mean is the centre of mass of the distribution.

Neither reading says that $3.5$ is a likely or even a possible outcome, and a die never shows it. The mean describes where the distribution is centred, not what to expect on a single roll.

The integral of "The Lebesgue integral" does not always give a finite number, and the same care applies here. If $X \ge 0$, then $\mathbb{E}[X]$ always exists, possibly as $+\infty$. For a general $X$, split $X = X^+ - X^-$ into its positive and negative parts. The expectation is $\mathbb{E}[X^+] - \mathbb{E}[X^-]$ whenever at least one of the two terms is finite, and $X$ is integrable when both are, which is the same as $\mathbb{E}|X| < \infty$. Linearity, monotonicity and the fact that changing $X$ on a set of probability zero changes nothing all carry over from the integral, and so does $|\mathbb{E}[X]| \le \mathbb{E}|X|$. For a constant, $\mathbb{E}[c] = c$, because the total mass is $1$.

Computing expectations from the distribution

The definition integrates over $\Omega$, but $\Omega$ is often out of sight. For a dart thrown at a square it is the square, for an endless sequence of coin tosses it is a set of binary digit sequences, and the point of "Random variables and their distributions" was that the distribution $\mathbb{P}_X$ on the real line carries what matters about $X$. The expectation can be computed from the distribution alone. For a measurable function $g$,

Taking $g(x) = x$ gives the mean from the distribution, and $g(x) = x^2$ gives the second moment, which is needed below. The proof follows the pattern used for every statement about integrals: indicators, then simple functions, then non-negative functions.

1. Indicators. For $g = \mathbf{1}_B$ with $B$ a Borel set, $\mathbf{1}_B(X)$ equals 1 exactly when $X \in B$, so $\mathbb{E}[\mathbf{1}_B(X)] = \mathbb{P}(X \in B) = \mathbb{P}_X(B) = \int \mathbf{1}_B\,d\mathbb{P}_X$. This is the definition of the image measure from "Measurable functions".
2. Simple functions. A simple $g = \sum_k b_k \mathbf{1}_{B_k}$ is a finite combination of indicators, and both sides of the formula are linear in $g$, so the first stage gives the second.
3. Non-negative functions. Choose simple functions $g_n \uparrow g$, the dyadic staircases of "Measurable functions". Then $g_n(X) \uparrow g(X)$, and monotone convergence from "Convergence theorems" lets the limit pass inside the integral on both sides, so the formula holds for $g$.
4. General functions. Apply the result to $g^+$ and $g^-$ separately and subtract.

Two consequences follow. First, $\mathbb{E}[g(X)]$ depends on $X$ only through its distribution: two random variables with the same distribution, even on different probability spaces, have the same expectation. Second, the integral against $\mathbb{P}_X$ is one of the concrete integrals of "The Lebesgue integral", and which one depends on the shape of the distribution:

• A distribution made of atoms only (a discrete one): $\mathbb{E}[g(X)] = \sum_x g(x)\,\mathbb{P}(X = x)$, a sum over the possible values.
• A distribution with a density $f$: $\mathbb{E}[g(X)] = \int g(x) f(x)\,dx$.
• A mixed distribution: the sum for the atoms plus the integral for the density part.

The waiting time at the traffic light of "Random variables and their distributions" is a mixed case. With probability $0.7$ the light is green and $X = 0$; otherwise $X$ is exponential with mean $2$ minutes. The distribution is a point mass of size $0.7$ at $0$ plus a density part of total mass $0.3$, so

On average a driver waits $0.6$ minutes, although most drivers wait not at all and the rest wait about two minutes. The same distribution gives $\mathbb{E}[X^2] = 0.7 \cdot 0 + 0.3 \cdot 8 = 2.4$, since an exponential with mean $2$ has second moment $2 \cdot 2^2 = 8$ (the case of mean $1$ is computed in the section on variance).

The dart board of "The Lebesgue integral" is an example on a continuous space. A dart lands uniformly on the unit square, which has area 1, so probability equals area. The board has concentric rings around the centre of the square with outer radii $0.1$, $0.25$, $0.4$ and $0.5$, scoring $10$, $5$, $2$ and $1$ points, and the corners score $0$. The score is a simple function, so its expectation is each score times the probability of its ring, which is the ring's area $\pi(r_{\text{outer}}^2 - r_{\text{inner}}^2)$:

A throw scores $0$, $1$, $2$, $5$ or $10$, and no throw scores $2.0342$. The number is the long-run average score per throw.

The tail formula

For a non-negative random variable there is a second way to compute the mean, which uses only the probabilities $\mathbb{P}(X > t)$ of exceeding each level $t$. The layer-cake formula of "The Lebesgue integral", $\int f\,d\mu = \int_0^\infty \mu(f > t)\,dt$, applied with $\mu = \mathbb{P}$ and $f = X$, says

Think of the value of $X$ as a stack of thin layers, one for each height $t$ between $0$ and $X$. The layer at height $t$ is present exactly when $X > t$, which has probability $\mathbb{P}(X > t)$, so the average height of the stack is the sum of these probabilities over all heights. The function $t \mapsto \mathbb{P}(X > t)$ is the tail of the distribution. It starts at $1$ or a little below and decreases to $0$ when $X$ is finite, and the mean is the area under it.

For the exponential law with mean $1$, $\mathbb{P}(X > t) = e^{-t}$, and $\int_0^\infty e^{-t}\,dt = 1$ is the mean, found with no integration by parts. For the fair die the tail is a staircase: for $t$ in $[k, k+1)$ with $k = 0, \ldots, 5$, $\mathbb{P}(X > t) = \tfrac{6-k}{6}$, a step of width $1$, so

This is the same $3.5$ as before, reached by a different cut. The earlier computation sliced the average into columns, one per value of $X$; this one slices it into layers, one per level. Using $\mathbb{P}(X \ge t)$ instead of $\mathbb{P}(X > t)$ gives the same integral, because the two functions differ at only countably many $t$, the atoms of the distribution. The formula is useful whenever tail probabilities are easier to find than a density, and it prepares the inequalities below, which are statements about tails.

Linearity and the indicator trick

Linearity of the integral gives, for integrable $X$ and $Y$ on the same probability space and any constants $a$ and $b$,

No assumption about how $X$ and $Y$ are related is needed. They can be identical, opposite or tangled together in any way: a sum of two functions on $\Omega$ is integrated by integrating each and adding. This is unusual, since most statements about several random variables need a condition on how they interact. The tool that makes linearity useful is the indicator. For an event $A$,

A random count is a sum of indicators, one for each thing counted, and its mean is the sum of the probabilities, however tangled the things are. Roll a die ten times and let $N$ be the number of sixes. Then $N = \sum_{i=1}^{10} \mathbf{1}_{\{\text{roll } i \text{ is a six}\}}$, and

Here the rolls do not influence each other, but the computation never used that, and the next example shows why this matters. A secretary has $n$ letters and $n$ addressed envelopes and puts the letters into the envelopes in random order, every arrangement being equally likely. Let $M$ be the number of letters that land in their own envelope. Letter $i$ is equally likely to land in any of the $n$ envelopes, so it lands in its own with probability $\tfrac1n$, and

The events influence each other: if the first $n - 1$ letters are in their own envelopes, the last one must be too. The distribution of $M$ is complicated, and yet its mean is $1$ for three letters and for a hundred. For $n = 3$ the six arrangements can be listed: the identity has $3$ correct letters, each of the three swaps of two letters has $1$, and the two cyclic shifts have $0$, so the mean is $(3 + 1 + 1 + 1 + 0 + 0)/6 = 1$.

Linearity does not extend to products. In general $\mathbb{E}[XY] \ne \mathbb{E}[X]\,\mathbb{E}[Y]$: for the die, $\mathbb{E}[X \cdot X] = 91/6 \approx 15.17$, while $\mathbb{E}[X]^2 = 12.25$. The next lesson, "Independence and product measures", shows which extra condition makes the product rule true.

Infinite expectations and expectations that do not exist

A non-negative random variable always has an expectation, but it can be infinite, even when the variable is finite with probability one. The St Petersburg game shows how. A fair coin is tossed until the first head. If the first head comes on toss $k$, which has probability $2^{-k}$, the prize is $2^k$: a prize of $2$ with probability $\tfrac12$, of $4$ with probability $\tfrac14$, of $8$ with probability $\tfrac18$, and so on. Every outcome gives a finite prize, yet

Each possible number of tosses adds one unit to the mean. If the game is stopped after at most $m$ tosses, with prize $0$ when no head has appeared by then, the mean is exactly $m$: $20$ for $m = 20$, and a million for a million tosses. Without a cutoff the mean grows past every bound, because the rare huge prizes carry as much weight as the common small ones. No finite amount is a fair average.

A variable that takes both signs has a second way to fail. The Cauchy law has density $f(x) = \dfrac{1}{\pi(1 + x^2)}$ on the whole line. It is a proper density, with total mass $\tfrac1\pi\,[\arctan x]_{-\infty}^{\infty} = 1$, but

Both $\mathbb{E}[X^+]$ and $\mathbb{E}[X^-]$ are infinite, so $\mathbb{E}[X^+] - \mathbb{E}[X^-]$ would be $\infty - \infty$, and there is no mean. Symmetry suggests $0$, but symmetry cancels the two infinite halves only after one decides how to cut them, and a different cut gives a different answer; the integral refuses to choose. The practical symptom is that averages of more and more Cauchy values never settle down. This is why the definition asks for $\mathbb{E}|X| < \infty$ before it speaks of the mean of a variable with both signs.

Variance and moments

The mean locates a distribution but says nothing about its spread. A variable that equals $3.5$ with certainty and a fair die have the same mean and little else in common. The usual measure of spread is the average squared distance from the mean. For $X$ with $\mathbb{E}[X^2] < \infty$ and mean $\mu$, the variance and the standard deviation are

Squaring makes every deviation count positively, so deviations above and below the mean do not cancel, and it keeps the algebra clean. The square root undoes the squaring, so $\sigma$ has the same unit as $X$. The variance is an expectation like any other, $\int (x - \mu)^2\,d\mathbb{P}_X(x)$, and is computed from the distribution. Expanding the square and using linearity gives a shortcut. First,

Now take expectations of each term. The mean $\mu$ is a constant, so $\mathbb{E}[2\mu X] = 2\mu\,\mathbb{E}[X] = 2\mu^2$ and $\mathbb{E}[\mu^2] = \mu^2$:

For the die, $\mathbb{E}[X^2] = \tfrac{1 + 4 + 9 + 16 + 25 + 36}{6} = \tfrac{91}{6} \approx 15.17$ and $(\mathbb{E}X)^2 = 12.25$, so $\operatorname{Var}(X) = \tfrac{35}{12} \approx 2.917$ and $\sigma \approx 1.708$. For the exponential law with mean $1$, integration by parts twice gives $\mathbb{E}[X^2] = \int_0^\infty x^2 e^{-x}\,dx = 2$, so $\operatorname{Var}(X) = 2 - 1 = 1$ and $\sigma = 1$. For the traffic light, $\operatorname{Var}(X) = 2.4 - 0.6^2 = 2.04$ and $\sigma \approx 1.428$, more than twice the mean, which reflects many zeros and a few long waits.

The variance is zero only for a constant: $\mathbb{E}[(X - \mu)^2] = 0$ forces $(X - \mu)^2 = 0$ almost surely, by the rule from "The Lebesgue integral" that a non-negative function with integral zero vanishes almost everywhere. Shifting and scaling behave simply. If $Y = aX + b$, then $\mathbb{E}[Y] = a\mu + b$, so $Y - \mathbb{E}[Y] = a(X - \mu)$ and

Adding a constant slides the whole distribution without changing its spread, and multiplying by $a$ multiplies the standard deviation by $|a|$. For $Y = 2X + 1$ with $X$ a die, $\operatorname{Var}(Y) = 4 \cdot \tfrac{35}{12} = \tfrac{35}{3} \approx 11.67$. How variance behaves for a sum of two random variables is the subject of "Independence and product measures".

The numbers $\mathbb{E}[X^p]$ are the moments of $X$, and the mean and the second moment are the first two. The random variables with $\mathbb{E}|X|^p < \infty$ are said to lie in $L^p$; for $p = 1$ these are the integrable ones. A variable can have a mean and no variance, but on a probability space a finite higher moment always forces the lower ones to be finite. For $p = 2$ the reason is the inequality $|x| \le 1 + x^2$, valid for every real $x$: when $|x| \le 1$ the right side is at least $1$, and when $|x| > 1$ we have $|x| < x^2$. Taking expectations gives $\mathbb{E}|X| \le 1 + \mathbb{E}[X^2] < \infty$, where the $1$ is $\mathbb{E}[1]$, finite because the total mass is $1$. A finite variance therefore guarantees a mean. On the whole line with Lebesgue measure the constant $1$ is not integrable, and the implication fails.

Markov's inequality

How much probability can sit far from the mean? For a variable that is never negative, the mean alone already limits it. Markov's inequality says that for $X \ge 0$ and any $a > 0$,

If the average is $3$ and the values are never negative, at most $3/9 = \tfrac13$ of the probability can lie at $9$ or beyond. Picture the sand on the beam, which starts at $0$ and extends to the right. Sand beyond $a$ sits at least $a$ away from the left end, so every unit of it pushes the balance point at least $a$ to the right; with too much of it out there, the balance point could not stay at $\mathbb{E}[X]$.

The proof is a comparison of two functions. For $X \ge 0$ and $a > 0$,

holds at every outcome: where $X \ge a$ the left side is $a \le X$, and elsewhere it is $0 \le X$ because $X$ is not negative. Expectation is monotone and $\mathbb{E}[\mathbf{1}_{\{X \ge a\}}] = \mathbb{P}(X \ge a)$, so $a\,\mathbb{P}(X \ge a) \le \mathbb{E}[X]$, and dividing by $a$ finishes the proof.

The inequality uses nothing but the mean, so it has to hold for every distribution with that mean, and it is often far from the truth. For the exponential law with mean $1$ and $a = 3$, the true tail is $\mathbb{P}(X \ge 3) = e^{-3} \approx 0.0498$, while Markov gives $\mathbb{E}[X]/a = 1/3 \approx 0.3333$, nearly seven times larger. It is not wasteful for every law. The two-point law with mass $\tfrac23$ at $0$ and $\tfrac13$ at $3$ has mean $1$ and $\mathbb{P}(X \ge 3) = \tfrac13$, exactly the bound, so no better bound can be had from the mean alone. For $a \le \mathbb{E}[X]$ the bound is at least $1$ and says nothing.

Chebyshev's inequality

Markov's inequality controls one side of a non-negative variable. To control the distance from the mean on both sides, apply it to the non-negative variable $(X - \mu)^2$, whose mean is the variance, at the level $a = k^2\sigma^2$. The event $|X - \mu| \ge k\sigma$ is the same event as $(X - \mu)^2 \ge k^2\sigma^2$, so for $\sigma > 0$ and $k > 0$,

This is Chebyshev's inequality. Whatever the distribution, at most $\tfrac14$ of the probability lies two standard deviations or more from the mean, at most $\tfrac19$ three or more, and at most $\tfrac1{16}$ four or more. The standard deviation becomes a ruler for rare events, and the only ingredients are the mean and the variance.

For the die, $\mu = 3.5$ and $\sigma \approx 1.708$. The event $|X - 3.5| \ge 2.5$ consists of the faces $1$ and $6$, so its probability is $\tfrac13$, while Chebyshev, with $k = 2.5/\sigma$, gives $\sigma^2/2.5^2 \approx 0.4667$. For the exponential law with mean $1$ and standard deviation $1$, take $k = 2$: the event $|X - 1| \ge 2$ is $X \ge 3$, because $X \le -1$ is impossible, with probability $e^{-3} \approx 0.0498$ against the bound $\tfrac14$. Even for the normal law the bound is generous, since two standard deviations leave about $0.0455$ in both tails together.

Is the bound as good as it can be when only the mean and variance are known? It is. For $k \ge 1$, take the three-point law with masses $\tfrac{1}{2k^2}$, $1 - \tfrac{1}{k^2}$ and $\tfrac{1}{2k^2}$ at $\mu - k\sigma$, $\mu$ and $\mu + k\sigma$. By symmetry its mean is $\mu$. Its variance is $(k\sigma)^2 \cdot 2 \cdot \tfrac{1}{2k^2} = \sigma^2$, and $\mathbb{P}(|X - \mu| \ge k\sigma) = \tfrac{1}{k^2}$, exactly the bound. For each $k$ some law with that mean and variance reaches it, so no smaller number can replace $1/k^2$ in a statement that uses only $\mu$ and $\sigma$.

The figure compares the true tail with the bound. Choose a law and a bound, then move the slider: $a$ for Markov, where the plot shows $\mathbb{P}(X \ge a)$ against $\mathbb{E}[X]/a$, or $k$ for Chebyshev, where it shows $\mathbb{P}(|X - \mu| \ge k\sigma)$ against $1/k^2$. The solid curve always stays below the dashed one. With the exponential law and Markov at the default $a = 3$, the readouts show $0.0498$ against $0.3333$, a ratio of about $6.70$; switching to Chebyshev at $k = 2$ gives $0.0498$ against $0.25$, a ratio of about $5.02$. The die draws a staircase with a jump at every face. The last law changes with the slider: it is the two-point law for Markov or the three-point law for Chebyshev that attains the bound at the marked value, where the solid curve touches the dashed one and the ratio is exactly $1$.

Jensen's inequality

The last inequality compares the average of a function with the function of the average. A function $\varphi$ is convex if every chord between two points of its graph lies above the graph:

Read this as an expectation. Let $X$ equal $x$ with probability $t$ and $y$ with probability $1 - t$. Then $tx + (1-t)y$ is $\mathbb{E}[X]$ and the right side is $\mathbb{E}[\varphi(X)]$, so the statement is $\varphi(\mathbb{E}[X]) \le \mathbb{E}[\varphi(X)]$. Jensen's inequality says the same for every integrable random variable $X$ with $\varphi(X)$ integrable and every convex $\varphi$. In a picture: the average of points on a convex curve lies above the curve, at the average position.

The proof for a general $X$ replaces the chord by a tangent. A convex function lies above its tangent lines, so at $m = \mathbb{E}[X]$ there is a slope $c$ (at a corner of the graph, any slope between the left and right slopes) with

Put $x = X$ and take expectations. The term $\mathbb{E}[X - m]$ vanishes, and what remains is Jensen's inequality:

Three consequences come back constantly. With $\varphi(x) = x^2$ the inequality reads $\mathbb{E}[X^2] \ge (\mathbb{E}X)^2$, which says that the variance is not negative; the variance is exactly the gap in Jensen's inequality for $x^2$. For the die, $\mathbb{E}[X^2] = 91/6 \approx 15.17 \ge 12.25 = (\mathbb{E}X)^2$, and the gap is $35/12$. With $\varphi(x) = |x|$ it gives $|\mathbb{E}X| \le \mathbb{E}|X|$. With $\varphi(x) = e^x$ it gives $\mathbb{E}[e^X] \ge e^{\mathbb{E}X}$: for the die, $\mathbb{E}[e^X] = \tfrac{e + e^2 + \cdots + e^6}{6} \approx 106.1$, far above $e^{3.5} \approx 33.12$.

For a concave function the inequality turns around, and it is a warning against a common mistake: in general $\mathbb{E}[g(X)] \ne g(\mathbb{E}[X])$. For the concave $g(x) = \sqrt{x}$ and a die, $\mathbb{E}\sqrt{X} \approx 1.805$, below $\sqrt{3.5} \approx 1.871$. The two agree only for affine $g(x) = ax + b$, which is linearity again.

Summary

• The expectation $\mathbb{E}[X] = \int X\,d\mathbb{P}$ is the Lebesgue integral under a probability measure: the long-run average, and the balance point of the distribution. It exists for $X \ge 0$ (possibly infinite) and for integrable $X$.
• $\mathbb{E}[g(X)] = \int g\,d\mathbb{P}_X$ computes it from the distribution: a sum for atoms, $\int g f\,dx$ for a density, both for a mixed law. For $X \ge 0$, $\mathbb{E}[X] = \int_0^\infty \mathbb{P}(X > t)\,dt$.
• Expectation is linear with no condition on $X$ and $Y$, and $\mathbb{E}[\mathbf{1}_A] = \mathbb{P}(A)$, so counts are averaged by adding probabilities. The mean can be infinite (St Petersburg) or not exist at all (Cauchy).
• $\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}X)^2$ measures spread, $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$, and a finite second moment implies a finite first moment.
• Markov: $\mathbb{P}(X \ge a) \le \mathbb{E}[X]/a$. Chebyshev: $\mathbb{P}(|X - \mu| \ge k\sigma) \le 1/k^2$, sharp for a three-point law. Jensen: $\varphi(\mathbb{E}X) \le \mathbb{E}\varphi(X)$ for convex $\varphi$.