Regression and Clustering

In the previous chapter we used a K-Nearest Neighbors classifier with scikit-learn. Here we expand our coverage of “classic” machine learning with two more fundamental techniques: regression for predicting continuous values, and clustering for discovering groups in data without labeled targets.

The requirements for this chapter are:

The examples for this chapter are in the directory source-code/regression_and_clustering.

Regression is a type of supervised machine learning where the goal is to predict a continuous numerical value rather than a class label. While classification asks “which category?”, regression asks “how much?” or “how many?”

Common regression algorithms include:

• Linear Regression: models the relationship between features and target as a straight line (or hyperplane in multiple dimensions).
• Polynomial Regression: extends linear regression by adding polynomial terms, allowing the model to capture non-linear relationships.
• Ridge and Lasso Regression: variants of linear regression that add regularization to prevent overfitting.

We will use the California Housing dataset, which is built into scikit-learn. It contains 20,640 samples with 8 features describing California census block groups (median income, average number of rooms, location, etc.), with the target being the median house value in units of $100,000s.

Our first example fits a linear regression model to all 8 features:

We use train_test_split to hold out 20% of the data for evaluation, scale the features with StandardScaler, and train the model:

To evaluate a regression model, we use different metrics than classification:

• MAE (Mean Absolute Error): average magnitude of prediction errors in the same units as the target.
• RMSE (Root Mean Squared Error): like MAE but penalizes larger errors more heavily.
• R² (R-squared): the proportion of variance in the target explained by the model (1.0 = perfect, 0.0 = no better than predicting the mean).

Here is the output of running the full regression.py script:

Because we scaled the features, the coefficients tell us the relative importance of each feature. MedInc (median income) has the largest positive coefficient, confirming that income is the strongest predictor of housing prices. The geographic features (Latitude and Longitude) are also very influential — location matters!

An R² of 0.58 means our simple linear model explains about 58% of the variance in housing prices. Not bad for a single line of model code, but there is room for improvement.

We can capture non-linear relationships by creating polynomial features. For interpretability, we use just two features (MedInc and AveRooms) and generate all degree-2 combinations:

The PolynomialFeatures transformer with degree=2 creates five features from our original two: x0, x1, x0², x0·x1, and x1². The interaction term x0·x1 lets the model learn that the relationship between income and price depends on the number of rooms.

With only 2 features, the polynomial model achieves R²=0.46 compared to 0.58 with all 8 features linearly. This illustrates an important tradeoff: polynomial features add expressive power but using too few input features limits the information available to the model.

Clustering is an unsupervised learning technique — unlike classification and regression, we do not provide labeled targets. Instead, the algorithm discovers natural groupings in the data.

K-Means is the most widely used clustering algorithm. It works by:

1. Choosing K initial cluster centers (centroids).
2. Assigning each data point to the nearest centroid.
3. Recomputing each centroid as the mean of its assigned points.
4. Repeating steps 2-3 until convergence.

We will use the classic Iris dataset (150 samples, 4 features measuring sepal and petal dimensions, with 3 known species). This lets us evaluate how well K-Means recovers the true species groupings.

A common question is: how many clusters should we use? The silhouette score measures how similar each point is to its own cluster compared to other clusters (ranging from -1 to +1, higher is better):

Output from running clustering.py:

The silhouette score is highest at k=2, which suggests that the data most naturally splits into two groups. This makes biological sense: setosa is very distinct from the other two species, while versicolor and virginica overlap considerably.

Since we happen to know the true species labels, we can use a cross-tabulation to see how well K-Means (with k=3) recovers them:

K-Means perfectly separates all 50 setosa samples into cluster 1. However, it struggles to distinguish versicolor from virginica — 14 virginica samples are assigned to the versicolor-dominant cluster 0, and 11 versicolor samples end up in the virginica-dominant cluster 2. This confirms what the silhouette scores suggested.

The cluster centers (transformed back to the original scale) reveal the “typical” flower in each group:

Cluster 1 (setosa) has much smaller petals, while cluster 2 (mostly virginica) has the largest sepals and petals.

In this chapter we covered two fundamental machine learning techniques beyond classification:

• Regression lets us predict continuous values. We saw that even simple linear regression can be informative, and that feature coefficients reveal which inputs drive predictions.
• Clustering discovers structure in unlabeled data. We used silhouette scores to evaluate cluster quality and cross-tabulations to compare clusters against known labels.

Together with the classification example in the previous chapter, you now have hands-on experience with the three core tasks of classic machine learning. In the next chapter we will explore the data preparation work that comes before training any model.

To solidify your understanding of regression and clustering, try these practical exercises. They build upon the code examples in the source-code/regression_and_clustering directory.

Distance-based algorithms like K-Means are highly sensitive to the scale of the input features.

• Task: Modify clustering.py to run K-Means on the original, unscaled dataset X (instead of X_scaled).
• Requirements:
  1. Calculate and print the silhouette scores for $k=2$ through $6$ using the unscaled data.
  2. Generate a cross-tabulation table comparing clusters ($k=3$) to the true species labels for the unscaled model.
• Questions for Reflection: How do the silhouette scores change when scaling is omitted? Looking at the features, why does sepal length or petal length dominate the distance calculations when unscaled?

Standard Linear Regression can overfit or behave erratically when features are correlated. Regularization adds a penalty term to shrink coefficients.

• Task: Extend regression.py to train and evaluate Ridge and Lasso models from sklearn.linear_model on the California Housing dataset.
• Requirements:
  1. Train Ridge and Lasso models using all 8 features (using scaled training data).
  2. Compare their metrics (MAE, RMSE, $R^2$) to the standard LinearRegression model.
  3. Print the feature coefficients for all three models side-by-side. Experiment with different regularization strengths (alpha=0.1, alpha=1.0, alpha=10.0).
• Questions for Reflection: What is the key difference between Ridge (L2 penalty) and Lasso (L1 penalty) regarding the sparsity of the coefficients? Which coefficients shrink to exactly zero first under Lasso?

While the silhouette score is a powerful metric, the “Elbow Method” using the sum of squared distances to the nearest centroid (inertia) is also widely used.

• Task: Write a short script or modify clustering.py to implement the Elbow Method.
• Requirements:
  1. Run K-Means on the scaled Iris dataset for $k$ values from $1$ to $8$.
  2. Record the value of the inertia_ attribute for each $k$.
  3. Print or plot the relationship between $k$ and inertia.
• Questions for Reflection: Identify the “elbow” point where the rate of decrease in inertia slows down significantly. Does this elbow match the optimal cluster count suggested by the silhouette score?

Evaluating hyperparameter tuning and preprocessing choices using a single train/test split can lead to data leakage and overly optimistic metrics. A better approach is to wrap preprocessing and modeling steps in a pipeline and use cross-validation.

• Task: Create a new script, housing_pipeline.py, in your source code directory.
• Requirements:
  1. Define a scikit-learn Pipeline containing StandardScaler, PolynomialFeatures(include_bias=False), and Ridge.
  2. Use GridSearchCV with 5-fold cross-validation to search over a grid of:
     • Polynomial degree: [1, 2]
     • Ridge parameter alpha: [0.1, 1.0, 10.0, 100.0]
  3. Print the best parameters found by the grid search.
  4. Evaluate the best estimator on your held-out test set and report the final MAE, RMSE, and $R^2$.
• Questions for Reflection: Why is it important to put StandardScaler inside the pipeline rather than scaling the entire dataset before cross-validation? How did the performance of this optimized model compare to the simple OLS model in the chapter?