Probability: Bayes, Base Rates, and Tests

Probability is the language AI uses to reason under uncertainty. A spam filter asks how likely a message is junk given the words it contains. A medical system asks how likely a patient is sick given a test result. A model evaluation asks whether an observed improvement is real or just noise. Each of those is a probability question wearing different clothes.

Many developers use probability through a library and never derive it, which works until a result looks wrong and there is no intuition to check it against. This chapter builds the machinery from scratch in pure Scala 3 so the formulas stay visible. It covers the two main schools of thought. Bayesian reasoning treats probability as a degree of belief and revises that belief when evidence arrives. Frequentist reasoning treats probability as a long-run frequency and asks how surprising the data would be if a null claim were true. It closes with correlation, the most used and most abused single number in data work.

All code is in source-code/probability. The project has no dependencies beyond the Scala standard library, so every formula is ours to read.

Before the formulas, fix the vocabulary. A sample space is the set of all possible outcomes, such as the two sides of a coin or the six faces of a die. An event is a subset of . A probability assigns a number in to each event, with and, for disjoint events, . A random variable is a function from outcomes to numbers, so “number of heads in ten tosses” is a random variable even though the outcomes are sequences of coin faces.

The first useful idea is conditional probability, the chance of given that happened:

We divide by because once we know occurred, the only outcomes left are those inside , and becomes our new universe. Rearranging gives the product rule , and applying it both ways gives the equation that powers this whole chapter.

The same numbers carry two different meanings. A frequentist reads as a statement about a long run of tosses: about half come up heads. A Bayesian reads it as a degree of belief, something that can be updated as evidence arrives. The arithmetic is identical. Only the interpretation and the willingness to state a prior differ, and that difference drives the split between the next two sections.

Conditional probability is symmetric in a specific way. The chance of given and the chance of given are linked by the product rule:

Solving for gives Bayes theorem:

Read as a hypothesis and as evidence and the four named parts appear:

• is the prior, what we believed before seeing the evidence.
• is the likelihood, how well the hypothesis predicts the evidence.
• is the marginal likelihood (or evidence), the total chance of seeing under every hypothesis.
• is the posterior, what we believe after the evidence.

When there are several competing hypotheses, the marginal is the sum over all of them, which is exactly how the code normalizes:

That sum is the key. It is what forces the posteriors to add up to one, and it is why a strong test can still produce a weak posterior when the alternatives are numerous or likely. The denominator is where base rates do their work.

The implementation splits cleanly into a value type and a companion object. The value type holds a normalized map from hypothesis name to probability. The object holds the two operations, constructing a model from priors and updating it with evidence. Here is the complete file:

Walk through it line by line.

fromPriors divides every prior by their sum. This means the caller can pass raw counts, rates, or anything proportional and still get a valid distribution. A prior map of Map("disease" -> 1, "healthy" -> 999) normalizes to the same model as Map("disease" -> 0.001, "healthy" -> 0.999). Accepting unnormalized input is a small kindness that removes a whole class of caller mistakes.

update is the heart of the file. It maps over the current posteriors, multiplies each prior by likelihood(h, evidence), and collects the results in unnorm. Those numbers are proportional to the true posteriors but do not sum to one, so the next line sums them into marginal and divides. The map is rebuilt with BayesModel(...) so a new immutable value is returned and the old model is untouched. Updating twice is just calling update twice, which is exactly what a Bayesian agent does when evidence arrives in sequence.

The require calls mark the two ways Bayes can fail. If every prior is zero, the prior map was empty or nonsense and there is nothing to normalize. If the marginal is zero, the evidence is impossible under every hypothesis, so no amount of updating can explain it. Both are caller errors rather than data problems, so the code fails loudly with a clear message instead of returning a map of NaN values that would silently poison everything downstream.

The posterior method reads one hypothesis by name and throws a named exception if it is missing, which turns a typo into an immediate error. The map method returns the maximum a posteriori hypothesis, the single most likely one, which is the natural output when a system must commit to a decision. Note that the most likely hypothesis is not the same as a confident one. As the medical example shows, the MAP can be “healthy” while the probability of disease is still far from zero.

This is the most famous counterintuitive result in applied probability, and the code demo exists to make it concrete. A rare disease affects 0.1% of a population. A screening test detects 99% of true cases (its sensitivity), but it also returns a false positive 5% of the time. A patient tests positive. What is the chance she actually has the disease?

Almost everyone, including many physicians asked in real studies, answers “about 99%.” The correct answer is about 1.9%. The reason is that the healthy group is enormous, so even a small false-positive rate produces far more false positives than the tiny sick group produces true positives.

Do the arithmetic by hand first. Take a population of 100,000 people.

Out of 100,000 people, about 99 + 4,995 = 5,094 test positive, and only 99 of those actually have the disease. The share is , or 1.94%. The false positives outnumber the true positives roughly fifty to one. This is the base rate fallacy: people anchor on the test’s accuracy and ignore how rare the condition is.

The same numbers fall straight out of Bayes. The unnormalized products are for disease and for healthy. Their sum, the marginal, is . Dividing gives for disease and for healthy. The likelihood ratio of 99 to 5, about 20 to 1, is strong, but the prior ratio of 1 to 999 is far stronger in the other direction, and the prior wins.

Here is the demo that runs these numbers, in full:

The likelihood function is the one piece the caller supplies. It ignores the evidence string here because there is only one kind of evidence in this demo, a positive test, but the signature (String, String) => Double is general enough to score any evidence against any hypothesis. The pattern match ends with a wildcard case that throws, so a misspelled hypothesis name is caught at the moment of use rather than silently treated as zero. The f string interpolator with %.4f keeps the printed columns aligned, and %% prints a literal percent sign.

One update is a good start, but the interesting behavior appears with a second test. A second positive result is treated as independent evidence, so we call update again on the already updated model. The prior for the second update is the posterior from the first, which is the entire point of Bayesian updating: yesterday’s belief becomes today’s starting point. After two positive tests the chance of disease rises from about 1.9% to about 28%. That is still not a diagnosis, but it changes the decision completely, which is why a good doctor orders a confirmatory test instead of celebrating or panicking after one result. The test file checks exactly this monotonic rise.

Bayes gives a clean answer but demands a prior, and sometimes no honest prior exists. The frequentist toolkit takes a different route. It does not assign probabilities to hypotheses at all. Instead it states a null hypothesis, a default claim such as “the coin is fair,” and asks how likely the observed data would be if that claim were true. If the data would be very unlikely, we reject the null. The output of a test is a p-value, the chance of seeing data at least this extreme under the null.

The subtle point, repeated in the code comments because it is so often gotten wrong, is what a p-value is not. A small p-value says the data would be rare if the null were true. It does not say the null has a small probability of being true. Those are different statements, and swapping them is the most common error in applied statistics. With that warning in place, here is the complete file:

Everything here rests on the standard normal distribution, the bell curve with mean 0 and variance 1. Its cumulative distribution function, written , gives the probability that a normal random variable is at most . There is no closed form for , so the code uses a rational approximation from Abramowitz and Stegun (formula 26.2.17) that is accurate to about . The approximation is written for the upper tail of the positive side, so phiApprox computes the value for and then mirrors it with 1 - cdf for negative to respect the symmetry of the bell curve. The five constants through are the coefficients of the fitted polynomial in , a change of variable that keeps the fit well behaved far out in the tails.

chiSquaredCdf reuses phiApprox through the Wilson-Hilferty transform, which says a chi-squared variable with degrees of freedom behaves roughly like a cube root of a normal. The transform maps the chi-squared value to a z-score, then feeds it to phiApprox. This is why one small normal routine supports two very different tests.

zTestProportion is the coin-toss check. For a proportion, the standard error under the null is , and the z-score measures how many standard errors the observed rate sits from the hypothesized rate:

The p-value is two-tailed, , because a coin that is biased in either direction is equally interesting, and it is capped at 1.0 so rounding can never produce a probability above one.

chiSquaredTest handles more than two categories. It compares each observed count to its expected count, sums the squared relative gaps, and sets the degrees of freedom to one less than the number of categories:

The guard if e == 0.0 then 0.0 avoids dividing by zero. The final p-value is the upper tail , clamped at zero for the same reason the z-test p-value is clamped at one.

inversePhi goes the other way, from a probability to a z-score, using Acklam’s approximation with three polynomial regions and a Horner-style poly fold. We need it for confidence intervals, where the multiplier is the z-score that leaves a given tail area. For 95% confidence the code calls inversePhi(0.975) and gets , the number every statistics course makes you memorize.

The Wilson interval is the payoff. A naive confidence interval for a proportion uses the same standard error as the z-test, . That is the Wald interval, and it fails badly near 0 and 1, sometimes producing bounds outside or a zero-width interval when you observe no successes. The Wilson interval solves the quadratic that the Wald interval linearizes, which pulls the center toward 0.5 and adds inside the square root. The result is always inside , and the code enforces that with math.max and math.min. For 60 heads in 100 tosses the Wilson interval is about , a range you can read with no test lore: the data are consistent with a fair coin, but also with a genuine bias up to about 69%.

The last tool is the one most likely to mislead. Correlation measures how strongly two series move together. It begins with covariance, the average product of each pair’s deviations from their own means:

If and tend to be above their means at the same time, the products are mostly positive and the covariance is positive. If one is high while the other is low, it is negative. The problem is that covariance carries the units of the data, so a covariance of 500 means nothing on its own. Dividing by both spreads fixes that and produces the Pearson correlation coefficient:

Here is the complete file:

The design is a small chain of reuse. mean guards against an empty series. covariance checks that the two series are the same non-empty length, computes both means, then zips the pairs and averages the products of deviations. stdDev calls covariance with the same series twice, because the covariance of a variable with itself is its variance, and takes the square root. pearson divides the covariance by the product of the two standard deviations.

Note that the variance here divides by rather than . That is the maximum likelihood form. For correlation it makes no difference at all, because the same appears in the numerator and both denominators and cancels. The choice only matters when you report a variance or standard deviation as a standalone estimate.

The pearson guard is the interesting line. If either series is constant, its standard deviation is zero and the division would produce , a value that spreads silently through any computation that touches it. Rather than propagate a poison value, the function returns 0.0. That is a defensible convention, because a flat line carries no trend to track, so it has no linear relationship with anything. The test file pins both cases: a perfect line gives and a constant series gives 0.0.

The interpretation is where caution is required. near 1 means the two series move together almost linearly, near -1 means they move in opposite directions, and near 0 means no linear relationship. It never means that one series causes the other. The medical example in this chapter is the proof. Test results and disease status are strongly correlated, yet a single positive test still implies only a small chance of disease, because the base rate is tiny. Correlation measures co-movement, not importance and not causation. A second trap is that captures only straight-line relationships: a perfect parabola can have while is completely determined by . Always plot the data before trusting the number.

The project includes a self-checking test file that runs without any test framework, using plain assert calls and a small close helper for floating point comparisons. Reading it is the fastest way to see the expected behavior of every function:

The close helper compares two doubles within a tolerance, which is the correct way to test floating point math. Comparing exact equality would fail on the rounding error that is unavoidable in a polynomial approximation. The tolerance is chosen per check: where the arithmetic is exact, where an approximation is involved.

The checks are chosen to pin down the interesting behavior rather than to chase coverage. The prior check confirms normalization. The medical posterior check pins the 1.94% result. The MAP assertion confirms that a single positive test does not flip the decision. The second-update assertion confirms that evidence accumulates. The normal checks pin , , and the inverse round trip. The biased coin case uses 700 heads in 1000 tosses, which is so far from fair that the z-score exceeds 10 and the p-value underflows past , an unambiguous rejection. The dice case is the opposite: near-uniform counts produce a high p-value, so the fair null is kept. The Wilson check confirms the interval brackets the observed rate and stays inside the legal range. The two correlation checks pin the perfect line and the constant-series guard.

The project is driven by scala-cli, which compiles and runs a directory of Scala sources with no build file. Run the interactive demo and the test suite from the example directory:

The demo prints the Bayesian screen followed by the two frequentist checks:

The test suite prints a single line when every assertion holds:

The first three lines of the demo show the prior. Disease is rare at and health is nearly certain at . These are the numbers that a frequentist is unwilling to state and a Bayesian insists on, and they dominate everything that follows.

After one positive test, the posterior for disease is , or 1.94%, and the MAP hypothesis is still healthy. The test’s 99% sensitivity feels decisive, but the prior is so heavily stacked against disease that a single result barely moves the needle. The healthy posterior is , so the model is not confused, it is appropriately cautious. This is the correct behavior, not a bug in the test.

The coin line reports and for 60 heads in 100 tosses. The observed rate of 0.6 sits exactly two standard errors above 0.5, and the two-tailed p-value of about 0.0455 is just under the conventional 0.05 threshold. That is a borderline result, and the honest reading is “suggestive but not overwhelming.” A larger sample with the same proportion would produce a smaller p-value, because the standard error shrinks with the square root of .

The Wilson interval is the more informative output. It says that the data are consistent with a true bias anywhere from essentially fair to about 69% heads, and it stays inside where the naive Wald interval might not. Reporting the interval alongside the p-value is a habit worth keeping, because the interval carries the uncertainty that a single p-value hides.

This chapter built three tools in a few dozen lines of dependency-free Scala.

• The Bayesian model, a normalized map plus a two-line update, shows how a prior and a likelihood combine into a posterior and how evidence accumulates across updates. The medical example is the case to remember: a 99% sensitive test yields a 1.94% posterior when the condition is rare, because base rates set the scale.
• The frequentist toolkit builds the normal CDF once and reuses it for z-tests, chi-squared tests, and Wilson intervals. It answers a narrower question than Bayes, requires no prior, and outputs a p-value that says how surprising the data are under a null claim, not how likely the claim is.
• Correlation compresses the co-movement of two series into a single number in and guards against the degenerate constant case. It measures association, never causation, and only linear association at that.

The recurring theme is that the denominator decides the answer. In Bayes it is the marginal likelihood that lets the base rate dominate. In the z-test it is the standard error that sets the scale of the evidence. In correlation it is the product of the spreads that puts on a fixed scale. When a probability result looks wrong, find the denominator and the intuition usually returns.

The code also shows a style that later chapters reuse. Small immutable value types, companion objects for the operations, pure functions with require guards on the preconditions, and a test file that uses plain assertions instead of a framework. That shape scales from a seven-line Bayes model to a full neural network.

1. A better test. Modify probabilityDemo so the screen test has a sensitivity of 0.99 and a false-positive rate of 0.01 instead of 0.05. Compute the posterior for disease after one positive test and compare it to the 1.94% baseline. Explain in one sentence why lowering the false-positive rate helps so much.

2. Three hypotheses. Extend the medical example to three hypotheses: disease, healthy, and carrier, with priors , , and . Give carriers a positive-test likelihood of . Run one update and check that the three posteriors sum to one. Which hypothesis is the MAP?

3. Sequential evidence. Starting from the original two-hypothesis model, update with a positive test three times in a row and print the disease posterior after each step. Confirm the sequence is strictly increasing and report the value after the third test. Is the MAP still healthy?

4. Odds form of Bayes. Add a function oddsRatio(likelihood: (String, String) => Double, h1: String, h2: String, evidence: String): Double that returns P(E|h1)/P(E|h2). Use it to show that the posterior odds equal the prior odds times the likelihood ratio, and verify the identity on the medical example.

5. Rejecting the null. Add a function to Frequentist that takes a p-value and a threshold and returns "reject" or "keep". Run it on the coin case with 60 heads in 100 and again with 700 heads in 1000. Report both decisions and the p-values.

6. A confidence interval for a proportion of zero. Call Frequentist.wilsonInterval(0, 20) and print the result. Then compute the naive Wald interval by hand and explain why the Wilson version is the one to trust.

7. Correlation versus causation. Construct two series with near that have an obvious common cause, for example the number of umbrellas sold and the number of puddles in a month. Write a short paragraph explaining why high does not license a causal claim.

8. Non-linear dependence. Create two series where for in . Compute Correlation.pearson and observe that it is near zero even though is a deterministic function of . Explain what this says about the limits of .