Probability spaces and Kolmogorov's axioms

The lesson "Measuring the size of sets" called a measure of total mass 1 a probability measure, "σ-algebras and Borel sets" explained which sets such a measure may measure, and "Lebesgue measure and null sets" built length on the real line and showed that a set can have length zero without being empty. Everything needed to talk about chance is therefore already in place, except the vocabulary. This lesson supplies it. It reads a measure space of total mass 1 as a model of an experiment with an uncertain result, translates each word of measure theory into a word of probability, and writes down Kolmogorov's axioms. It then builds the models that the rest of the track keeps returning to: two dice, a dart thrown at a square board, a number drawn uniformly from $[0, 1]$, and a coin tossed forever.

Outcomes, events and probabilities

Probability starts with an experiment whose result is not known in advance: roll two dice, throw a dart at a board, toss a coin. A probability space is a mathematical description of such an experiment. It has three ingredients, and they are exactly the three ingredients of a measure space.

The first is the set $\Omega$ of outcomes. An outcome is a complete description of one possible result, detailed enough that once it is known, every question about the experiment has an answer. For two dice an outcome is an ordered pair such as $(3, 5)$, meaning that the first die shows 3 and the second shows 5; there are $6 \cdot 6 = 36$ of them. For a dart thrown at a square board of side 1, an outcome is the point $(x, y)$ where the tip lands, and $\Omega$ is the unit square. When the experiment is run, exactly one outcome occurs.

The second is the collection $\mathcal{F}$ of events. An event is a statement about the result that is either true or false once the outcome is known, such as "the sum of the dice is 7" or "the dart lands in the left half". Each statement is identified with the set of outcomes that make it true: "the sum is 7" is the set $\{(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)\}$. As "σ-algebras and Borel sets" explained, $\mathcal{F}$ is a σ-algebra. It lists the questions that may be asked, and it is closed under the operations that combine questions: not, or, and their countable versions. For two dice, $\mathcal{F}$ can safely contain every subset of $\Omega$.

The third is the probability $\mathbb{P}$, a measure on $(\Omega, \mathcal{F})$ that gives each event $A$ a number $\mathbb{P}(A)$ between 0 and 1 saying how likely it is. The triple $(\Omega, \mathcal{F}, \mathbb{P})$ is the probability space.

Because events are sets, the words of everyday logic turn into the words of set theory that the earlier lessons already use. The dictionary is short:

• an outcome is a point $\omega \in \Omega$;
• an event is a set $A \in \mathcal{F}$, and it occurs when the outcome $\omega$ lies in $A$;
• "$A$ or $B$" is the union $A \cup B$, and "$A$ and $B$" is the intersection $A \cap B$;
• "not $A$" is the complement $A^c = \Omega \setminus A$;
• the impossible event is the empty set $\emptyset$, and the certain event is $\Omega$;
• "$A$ and $B$ cannot happen together" means $A \cap B = \emptyset$, that is, the events are disjoint;
• "if $A$ occurs then $B$ occurs" means $A \subseteq B$.

Take two dice, and let $A$ be "at least one die shows a six" and $B$ be "the sum is 7". The event $A$ has 11 outcomes: six with the first die a six, six with the second die a six, and the pair $(6, 6)$ counted twice, so $6 + 6 - 1 = 11$. The event $B$ has the 6 outcomes listed above. "$A$ and $B$" holds only for $(1, 6)$ and $(6, 1)$, so $A \cap B$ has 2 outcomes, and "$A$ or $B$" has $11 + 6 - 2 = 15$. The events "the sum is 7" and "the sum is 8" are disjoint, since no pair has both sums.

Kolmogorov's axioms

A probability has to give sensible numbers, and Kolmogorov's three axioms say what that means. A function $\mathbb{P}$ defined on the events of a σ-algebra $\mathcal{F}$ is a probability measure if:

1. Non-negativity: $\mathbb{P}(A) \ge 0$ for every event $A$. A chance is never negative.
2. Normalisation: $\mathbb{P}(\Omega) = 1$. Something certainly happens, and the scale is fixed so that certainty is 1, not 100 or 6.
3. Countable additivity: $\mathbb{P}\big(\bigcup_{k} A_k\big) = \sum_{k} \mathbb{P}(A_k)$ for every finite or infinite sequence of pairwise disjoint events $A_1, A_2, \ldots$. If events cannot happen together, the chance that one of them happens is the sum of their chances.

These are the rules of a measure from "Measuring the size of sets" (sizes are non-negative and add over disjoint pieces, now countably many), together with one extra requirement, a total mass of 1. A probability space is therefore nothing other than a measure space $(\Omega, \mathcal{F}, \mathbb{P})$ with $\mathbb{P}(\Omega) = 1$. Every fact about measures proved in the earlier lessons is a fact about probabilities, and probability contributes a new vocabulary and new questions, not new axioms.

Each axiom has a job. Without non-negativity, additivity could cancel one chance against another. Without normalisation, the numbers would be sizes, not probabilities: under counting measure a fair die would give "a six" the number 1 and "a number from 1 to 6" the number 6. Without countable additivity, infinite experiments would behave badly. "Measuring the size of sets" showed this with the natural density of the positive integers, a finitely additive size that gives every single positive integer size 0 and yet gives the set of all positive integers size 1. A probability like that would describe a lottery that has a winner among the integers although no integer has any chance of winning.

The axioms are named after Andrey Kolmogorov, who published them in 1933 in his book "Foundations of the Theory of Probability". Before then probability had been in use for almost three centuries without a settled definition: counting equally likely cases breaks down for infinitely many cases, and limits of frequencies were hard to make precise. Measure theory, developed by Borel and Lebesgue in the first years of the twentieth century, supplied what was missing. Kolmogorov's observation was that a probability is a measure of total mass one.

What probabilities mean

The axioms say how probabilities combine. They do not say what a probability is, and three readings are in use. Each one obeys the axioms.

• Symmetry. When outcomes are interchangeable, they get equal shares. A fair die has no preferred face, so each face gets $1/6$, and the probability of an event is the share of outcomes in it.
• Long-run frequency. Repeat the experiment many times: the fraction of repetitions in which $A$ occurs settles near $\mathbb{P}(A)$. Roll a fair die 6,000 times and about 1,000 sixes appear.
• Degree of belief. $\mathbb{P}(A)$ measures how strongly one is convinced that $A$ is true. This reading also covers experiments that cannot be repeated, such as whether tomorrow will be rainy.

The readings agree on the rules, and the rules are all that this track uses. For beliefs the axioms are a consistency check. Suppose someone is 60 % sure that a die will show an even number and 60 % sure that it will show a 1. These events are disjoint, so their chances would have to add up to $120\ \%$ for an event that cannot be more than certain. No probability measure produces such beliefs, so they are incoherent.

Rules that follow from the axioms

Every property of measures in "Measuring the size of sets" holds for $\mathbb{P}$, and the total mass 1 makes several of them simpler. Each rule comes with its short reason.

Complement. The events $A$ and $A^c$ are disjoint and their union is $\Omega$, so $\mathbb{P}(A) + \mathbb{P}(A^c) = \mathbb{P}(\Omega) = 1$:

The chance that something does not happen is one minus the chance that it does. This is the main tool for events of the form "at least one …", whose complement is "none", which is often easier to count. Two consequences: $\mathbb{P}(\emptyset) = 1 - \mathbb{P}(\Omega) = 0$, and every probability is at most 1.

Monotonicity. If $A \subseteq B$, then $B$ is the disjoint union of $A$ and $B \setminus A$, so

Inclusion–exclusion. Adding $\mathbb{P}(A)$ and $\mathbb{P}(B)$ counts the overlap $A \cap B$ twice, so one copy has to be taken away:

Union bound. For any events, disjoint or not, the chance that at least one occurs is at most the sum of their chances:

Continuity. If $A_1 \subseteq A_2 \subseteq \cdots$ grow to $A = \bigcup_n A_n$, then $\mathbb{P}(A_n) \to \mathbb{P}(A)$. If $B_1 \supseteq B_2 \supseteq \cdots$ shrink to $B = \bigcap_n B_n$, then $\mathbb{P}(B_n) \to \mathbb{P}(B)$. The second statement is continuity from above, and it holds here because the total mass is finite. Both are used below, when infinitely many coin tosses appear.

Here is how the rules work on two dice. The chance of "at least one six" is easiest through the complement: "no six" has $5 \cdot 5 = 25$ of the 36 outcomes, so

Inclusion–exclusion gives the same number. Let $A_1$ be "the first die shows six" and $A_2$ "the second die shows six"; then $\mathbb{P}(A_1 \cup A_2) = \frac16 + \frac16 - \frac1{36} = \frac{11}{36}$. The union bound would give only $\frac16 + \frac16 = \frac13 \approx 0.3333$. It overshoots by $\frac1{36}$, exactly the probability of $A_1 \cap A_2$, the pair $(6, 6)$ that the sum counts twice.

Discrete models

The simplest probability spaces have countably many outcomes, finitely many or listed in a sequence $\omega_1, \omega_2, \ldots$. Such a model is fixed by a mass function $p(\omega) \ge 0$ that gives each outcome a share, with the shares adding up to 1:

Here $\mathcal{F}$ is the collection of all subsets of $\Omega$, since every subset of a countable set is a countable union of single points. The three axioms hold. Sums of non-negative numbers are non-negative, $\mathbb{P}(\Omega) = 1$ is the first equation, and the sum over a countable disjoint union can be split into the sums over the pieces because the terms are non-negative, so the order of adding does not matter.

For two fair dice every one of the 36 outcomes gets $p = 1/36$. The sum is 7 for six outcomes, so $\mathbb{P}(\text{sum} = 7) = 6/36 = 1/6 \approx 0.1667$. The sum is even for 18 outcomes, so it has probability $1/2$.

An infinite example: toss a fair coin until the first head appears and record the number of the toss. The outcomes are $1, 2, 3, \ldots$. The first head comes on toss $k$ only if the first $k - 1$ tosses are tails and toss $k$ is a head. Each toss has two equally likely results, so the pattern of $k$ tosses has probability $(1/2)^k$ and $p(k) = 2^{-k}$. The shares add up to $\frac12 + \frac14 + \frac18 + \cdots = 1$, as a mass function requires. The model has no outcome "no head ever", and needs none: the shares already use up the total mass, so such an outcome would get probability 0.

Countable additivity now matters. How likely is it that the first head comes on an even toss? The event is $\{2, 4, 6, \ldots\}$, a countable disjoint union of single outcomes, so its probability is an infinite sum:

Finite additivity could not even state this probability. The worked example evaluates it.

Continuous models

Not every experiment has countably many outcomes. When the result is a number on a scale or a position on a board, $\Omega$ is uncountable, and the probability has to be spread out like sand rather than placed in lumps.

The first continuous model is a number $U$ chosen uniformly from $[0, 1]$. Here $\Omega = [0, 1]$, $\mathcal{F}$ is the Borel σ-algebra and $\mathbb{P} = \lambda$ is Lebesgue measure, which is a probability measure because $\lambda([0, 1]) = 1$. The probability of an event is its length, so intervals of equal length are equally likely wherever they sit:

Whether an endpoint is included makes no difference, because a single point has length 0.

The second model is a dart that lands at a uniformly chosen point of the unit square. Here $\Omega = [0, 1]^2$ and $\mathbb{P}$ is area, the two-dimensional Lebesgue measure of "Lebesgue measure and null sets". The dart lands in the left half with probability $0.5$. It lands within $0.1$ of the centre of the board with the probability that equals the area of a disc of radius $0.1$, which fits inside the square, so $\pi \cdot 0.1^2 \approx 0.0314$. The largest disc that fits has radius $0.5$ and area $\pi/4 \approx 0.7854$. In both models the probability of a region depends only on its size, which is what "uniform" means.

Two features differ from the discrete case. First, there is no mass function: every single outcome has probability zero. $\mathbb{P}(U = 0.5) = \lambda(\{0.5\}) = 0$, and the same holds for every point of the board and, by the rule that countable sets are null, for every countable set of points, such as all the rational numbers in $[0, 1]$. Second, $\mathcal{F}$ cannot be the collection of all subsets. Vitali's set from "σ-algebras and Borel sets" has no length, so events are the Borel sets, and any event one can actually describe is among them.

This looks like a paradox. Every outcome has probability 0, yet the experiment produces an outcome, so an event of probability 0 has happened. The resolution is that countable additivity adds countably many probabilities and says nothing about uncountably many. The interval $[0, 1]$ is the union of uncountably many points, each of probability 0, and the sum of uncountably many zeros is not defined and need not be 0. Probability behaves like mass in a solid rod: a single point of the rod weighs nothing, and the rod still weighs something.

Infinitely many coin tosses

Many experiments are repeated without a bound: toss a coin again and again, forever. A probability space for this needs an outcome that records an entire infinite sequence of heads and tails, and a probability under which the tosses are fair and separate. The requirement is concrete: any particular result of the first $k$ tosses, say head, tail, head, has probability $(1/2)^k$, in this case $1/8$. There is a short way to build such a space from the uniform number of the previous section, with nothing new to construct.

Every number $\omega \in [0, 1)$ has a binary expansion $\omega = 0.b_1 b_2 b_3 \ldots_2$, which means

For example $0.6875 = \tfrac12 + \tfrac18 + \tfrac1{16} = 0.1011_2$, and $\tfrac13 = 0.010101\ldots_2$. The digit $b_1$ tells which half of $[0, 1)$ the number lies in, 0 for the left half and 1 for the right, and each further digit tells which half of the remaining piece it lies in. The coin model reads the digits as tosses: draw $\omega$ uniformly from $[0, 1)$, and toss $k$ is a head when the $k$-th binary digit of $\omega$ is 1 and a tail when it is 0. So $\omega = 0.6875$ gives head, tail, head, head, and then tails forever, and $\omega = 1/3$ gives tail, head, tail, head, and so on, alternating.

To see that these tosses are fair, look at what a pattern of the first $k$ tosses says about $\omega$. A first head means $\omega \in [\tfrac12, 1)$, a set of length $\tfrac12$. Fixing the second toss as well cuts that half in two: head then tail is $[\tfrac12, \tfrac34)$ and head then head is $[\tfrac34, 1)$, each of length $\tfrac14$. Every further toss halves each piece again. After $k$ tosses the unit interval is cut into $2^k$ dyadic intervals of length $2^{-k}$, one for each pattern, and the pattern with binary value $m$ corresponds to $[m/2^k, (m+1)/2^k)$. For head, tail, head the value is $m = 101_2 = 5$, the interval is $[\tfrac58, \tfrac68) = [0.625, 0.75)$, and its length, hence its probability, is $\tfrac18$ as required:

The figure shows the construction. Move the number $\omega$ and watch its first twelve tosses change; then press the tosses of the pattern to flip them, and change the number $k$ of tosses in the pattern. The highlighted interval is always the set of numbers whose first tosses equal the pattern, and its length is always $2^{-k}$. With the defaults, $\omega = 0.6875$ starts H T H H followed by tails, the pattern H T H occupies $[0.625, 0.75)$ with probability $0.125$, and $\omega$ lies inside it.

One technical point. Some numbers have two binary expansions: $0.5 = 0.1000\ldots_2 = 0.0111\ldots_2$. These are the numbers $m/2^n$, a countable set, which is null, so the choice affects no event with positive probability. The convention is to use the expansion that does not end in endless 1s, and that is what the figure and the rest of the track do.

The space now answers questions about the whole infinite sequence at once. Take "a head appears at some point". Its complement, "all tails", needs every digit to be 0, which only the number $\omega = 0$ achieves. So the probability is $1 - 0 = 1$. Continuity gives the same from the other side. The event "no head in the first $n$ tosses" is the interval $[0, 2^{-n})$, with probability $2^{-n}$, and these events shrink to $\{0\}$ as $n$ grows. For $n = 10$ the probability is $1/1024 \approx 0.0009766$, for $n = 20$ it is about $9.537 \cdot 10^{-7}$, and the limit is 0. "Every toss is a tail" is the single outcome $\omega = 0$: possible, yet of probability zero.

The first-head model of the previous section sits inside this one. The first head comes on toss $k$ exactly when the digits are $k - 1$ zeros followed by a 1, which is $\omega \in [2^{-k}, 2^{1-k})$, an interval of length $2^{-k} = p(k)$. The first head on an even toss is the union of $[\tfrac14, \tfrac12)$, $[\tfrac1{16}, \tfrac18)$, $[\tfrac1{64}, \tfrac1{32})$ and so on, with lengths $\tfrac14, \tfrac1{16}, \tfrac1{64}, \ldots$, which add up to $\tfrac13$ as in the worked example. Both models give the same answers. That the tosses are "separate" has only been used informally so far; a later lesson of this track makes it precise and shows that the binary digits of a uniform number behave as separate fair coins.

Almost sure events

An event $A$ with $\mathbb{P}(A) = 1$ occurs almost surely (a.s.). This is the probability name for what "Lebesgue measure and null sets" called "almost everywhere": a property that fails only on a null set, which in a probability space is an event of probability 0. A number drawn uniformly from $[0, 1]$ is almost surely irrational, almost surely outside the Cantor set, and almost surely has a 7 somewhere in its decimal expansion. In the coin model a head appears almost surely.

An event of probability 0 can still happen. Whatever number the experiment produces, the event "$U$ equals that number" had probability 0 before the draw and occurred. Probability zero describes how thin an event is: it would be hit with frequency zero in a long series of repetitions, not that it has no outcomes.

Three different statements: an event is impossible if it is the empty set; it has probability zero if it is null, which allows outcomes (any single point of $[0, 1]$); it occurs almost surely if its complement is null. Impossible implies probability zero, and probability zero does not imply impossible.

Almost sure events combine well. If $A_1, A_2, \ldots$ each occur almost surely, so does their intersection: the complements are null, a countable union of null events is null by the union bound, $\mathbb{P}\big(\bigcup_n A_n^c\big) \le \sum_n 0 = 0$, and the complement of that union is the intersection. For the coin model, fix $n$ and consider "a head occurs after toss $n$". Its complement says that all digits after the $n$-th are 0, so $\omega$ is one of the $2^n$ numbers $m/2^n$, a finite set, which is null. The event is therefore almost sure for each $n$, and so is the intersection over all $n$: almost surely there are infinitely many heads.

A last statement, quoted without proof: the fraction of heads among the first $n$ tosses tends to $1/2$ almost surely. A later lesson of this track proves it.

Choosing a model

The same experiment can have several probability spaces. A single fair die can be modelled by $\Omega = \{1, \ldots, 6\}$ with $p = 1/6$ for each face. It can also be modelled by $\Omega = [0, 1)$ with Lebesgue measure, reading the face $k$ when $\tfrac{k-1}{6} \le \omega < \tfrac{k}{6}$. Both give the face 4 probability $1/6$ and the even faces probability $3/6 = 1/2$, and every other question about the die gets the same answer in both. What matters is that the events of interest receive the right probabilities, and a smaller space is often easier.

The model also has to be chosen with care, because "pick a number at random" is not a complete description. If $U$ is uniform on $[0, 1]$ then the number $U^2$ is not. By "Measurable functions", $\lambda(x^2 \le c) = \sqrt{c}$, so $\mathbb{P}(U^2 \le 0.25) = \mathbb{P}(U \le 0.5) = 0.5$, while a number uniform on $[0, 1]$ would fall below $0.25$ with probability $0.25$. Squaring moves probability towards 0. Whenever an experiment is described in words, the first task is to say which $\mathbb{P}$ is meant. The next lesson, "Random variables and their distributions", studies exactly this: how a quantity computed from the outcome, such as $U^2$ or the sum of two dice, has a probability measure of its own.

Summary

• A probability space $(\Omega, \mathcal{F}, \mathbb{P})$ is a measure space of total mass 1: $\Omega$ holds the outcomes, the events in $\mathcal{F}$ are the answerable questions, and $\mathbb{P}(A)$ is the chance of the event $A$.
• Kolmogorov's axioms are non-negativity, $\mathbb{P}(\Omega) = 1$ and countable additivity. They fit symmetry, frequencies and degrees of belief alike, and they give the complement rule, inclusion–exclusion, the union bound and continuity.
• A discrete model is a mass function with total 1 on a countable set; a continuous model spreads probability, like Lebesgue measure, so that every single outcome has probability zero.
• Infinitely many fair coin tosses are the binary digits of one uniform number: a pattern of $k$ tosses is a dyadic interval of length $2^{-k}$.
• An event of probability 1 occurs almost surely, the probability name for almost everywhere; probability zero does not mean impossible, and countably many almost sure events hold together.
• One experiment can have several probability spaces, and "at random" is incomplete until $\mathbb{P}$ is specified.