Anomaly Detection

Anomaly detection is a technique for finding data points that don’t fit the normal pattern. Unlike standard classification where we have roughly equal numbers of examples for each class, anomaly detection is designed for problems where “normal” examples vastly outnumber “abnormal” ones. Think of credit card fraud (millions of legitimate transactions vs. a handful of fraudulent ones), network intrusion detection, manufacturing defect spotting, or medical diagnosis.

The core idea is simple: learn what “normal” looks like, then flag anything that deviates too far from that model.

In this chapter we build a Gaussian anomaly detector from scratch in Common Lisp and apply it to the Wisconsin Diagnostic Breast Cancer dataset. The dataset contains 647 samples of cell measurements, each labeled as either benign (normal) or malignant (anomalous). Our goal is to train a model on mostly normal data and then use it to identify anomalies — samples that the model considers unusual.

The source code for this chapter is in the directory src/anomaly_detection.

Before diving into the code, let’s understand the key idea behind our detector. A Gaussian distribution (also called a “bell curve” or “normal distribution”) describes how values of a measurement tend to cluster around an average value. Most measurements fall close to the average, and very few fall far away.

Every Gaussian distribution is characterized by two numbers:

• Mean (μ) — the average value. This is the center of the bell curve.
• Variance (σ²) — how spread out the values are. A small variance means values cluster tightly around the mean; a large variance means they are more spread out.

The Gaussian probability density function (PDF) tells us how likely a particular value is, given our distribution:

If we compute p(x) and get a high number, the value x fits well with our model of “normal.” If p(x) is very low, then x is far from the average — it’s an anomaly.

Our detector works in four steps:

1. Split the data into three groups: training (60%), cross-validation (28%), and test (12%). The training set is mostly normal examples. This mirrors how you would use the system in practice: train on data you believe to be normal, then test on new data.

2. Fit a Gaussian model to the training data. For each feature (measurement), we compute the mean μ and variance σ². This tells us what “normal” looks like for each feature independently.

3. Tune the epsilon threshold. For each data sample, we compute its average probability across all features. If this probability is below a threshold called epsilon, we classify it as an anomaly. But what value of epsilon works best? We try 200 different values and pick the one that makes the fewest mistakes on the cross-validation set. This is called hyperparameter tuning.

4. Evaluate on test data. With the best epsilon selected, we run the detector on previously unseen test data and report precision, recall, and the F1 score.

The dataset contains 647 samples with 9 numeric features each (such as “Clump Thickness,” “Cell Size Uniformity,” and “Bare Nuclei”) scored on a 1–10 scale, plus a target label: 2 for benign and 4 for malignant. In our preprocessing we remap the target to 0 (benign/normal) and 1 (malignant/anomaly).

The raw features are integer-valued and skewed, which is not ideal for a Gaussian model. We apply two preprocessing steps (matching the original Java implementation):

• Log transform — applying log(x + 1.2) to each feature makes the distribution more bell-shaped.
• Min-max normalization — scaling each row’s features to the range [0, 1] so that all features contribute equally.

The code is split into two files:

• anomaly-detection.lisp — the core detection algorithm (package, data structures, training, evaluation)
• wisconsin.lisp — CSV loading, preprocessing, histograms, and the run-wisconsin entry point

We use a defstruct to hold the model state:

The mu slot holds the per-feature means, sigma-sq holds the per-feature variances, and best-eps is the learned epsilon threshold. The three data splits — training, cross-validation, and testing — are stored so we can use them during the training process.

The compute-mu function calculates the average value of each feature across all training examples:

This is straightforward: sum all values for each feature, then divide by the number of examples. The inner loop iterates over features and accumulates into the mu array.

The compute-sigma-sq function calculates the variance that is how much each feature’s values deviate from the mean:

For each feature, we compute the sum of squared differences from the mean, then divide by n. The (max ... 1.0d-10) guard prevents division by zero later when a feature has identical values across all examples. Notice that we skip the last column ((1- num-features)) because that column is the target label, not a measurement.

The heart of the algorithm is the gaussian-p function. For a given data sample, it computes the Gaussian probability for each feature and returns their average:

Let’s trace through what happens for one feature:

1. We retrieve the variance s2 and compute the standard deviation sigma = √s2.
2. We compute diff — how far this sample’s value is from the mean.
3. The exponent is -(diff²) / (2 · σ²). When diff is large (the value is far from the mean), this exponent becomes a large negative number, making exp(exponent) very small.
4. We multiply by the normalization constant 1 / (√(2π) · σ) to get a proper probability density.
5. We sum these per-feature probabilities and divide by the number of features to get an average.

A normal sample will have values close to the mean across most features, producing a relatively high average probability. An anomalous sample will have unusual values in several features, producing a low average probability.

The train function searches for the best epsilon threshold:

This tries 200 epsilon values from 0.001 to 1.0 in steps of 0.005. For each value, train-helper counts how many cross-validation examples are misclassified. The epsilon with the fewest errors wins. The <= comparison (rather than <) ensures that when multiple epsilon values tie, we pick the highest one — this gives the decision boundary more margin and tends to generalize better.

Note the sb-int:with-float-traps-masked wrapper. SBCL, unlike Java, traps floating-point exceptions by default. The log transform and Gaussian PDF computations can occasionally produce extreme values (especially during the first few epsilon trials), so we mask these traps to match Java’s behavior of silently propagating special float values.

The test-model function evaluates the trained detector on held-out test data and computes three standard metrics:

• Precision — of the samples flagged as anomalies, what fraction actually are? (Measures false alarm rate.)
• Recall — of the actual anomalies, what fraction did we catch? (Measures miss rate.)
• F1 score — the harmonic mean of precision and recall. An F1 of 1.0 means perfect detection with no false alarms.

The preprocess-wisconsin function in wisconsin.lisp transforms the raw CSV data to make it suitable for Gaussian modeling:

Each row goes through three transformations:

1. Scale by 0.1 — the raw values are integers 1–10; dividing by 10 puts them in [0.1, 1.0].
2. Log transform — log(x + 1.2) pushes the distribution toward a bell shape. The offset 1.2 ensures we never take the log of zero.
3. Per-row min-max normalization — maps values to [0, 1] so that all features contribute equally. The guard for range < 1e-10 handles the rare case where all features in a single row have the same value (which would cause a division by zero).

Finally, the target label is remapped from {2, 4} to {0, 1}.

To run the full pipeline:

Here is typical output (the exact numbers vary due to the random data split):

The detector achieves an F1 score above 0.85 — meaning it correctly identifies most malignant samples while producing few false alarms. This is a strong result for such a simple model, and demonstrates that the Gaussian approach works well when the normal data is roughly bell-curve shaped (which our log transform helps ensure).

You can also use the anomaly detection module interactively from the REPL:

The anomaly-p function returns T if the detector considers the input vector anomalous and NIL otherwise. This makes it easy to integrate into a larger pipeline — for example, flagging suspicious transactions in a stream of financial data.

If you are new to machine learning, the evaluation output deserves a closer look:

• True positives (TP) — anomalies correctly identified as anomalies.
• True negatives (TN) — normal samples correctly identified as normal.
• False positives (FP) — normal samples incorrectly flagged as anomalies (false alarms).
• False negatives (FN) — anomalies that slipped through undetected (misses).

In medical diagnosis, false negatives are especially dangerous — a malignant tumor classified as benign could delay treatment. Our detector’s high recall (above 0.95) means it catches nearly all anomalies, at the cost of a few false alarms. This is generally the right tradeoff for safety-critical applications.

We built a Gaussian anomaly detector from scratch, using nothing but Common Lisp’s built-in math and array operations. The approach is simple yet effective:

1. Model each feature as a Gaussian distribution (mean + variance).
2. Score new samples by how well they fit the model.
3. Use cross-validation to find the best decision threshold.

This technique works best when you have many normal examples and relatively few anomalies — exactly the scenario where traditional classification struggles because there aren’t enough positive examples to learn from.

The code in this chapter was ported from a Java implementation and corrected to use the proper Gaussian PDF formula. The same algorithm can be applied to any numeric dataset where you need to detect outliers or unusual patterns.