Lecture 16 - (23/04/2026)

Today’s Topics:

• Taxomony of Machine Learning

• K-Means Clustering

• Hierarchical Agglomerative Clustering

• Choosing a Clustering Algorithm

Last time, we began our journey into unsupervised learning by discussing Principal Component Analysis (PCA).

In this lecture, we will explore another very popular unsupervised learning concept: clustering. Clustering allows us to “group” similar datapoints together without being given labels of what “class” or where each point explicitly comes from.

Taxomony of Machine Learning¶

Supervised Learning¶

In supervised learning, our goal is to create a function that maps inputs to outputs. Each model is learned from example input/output pairs (training set), validated using input/output pairs, and eventually tested on more input/output pairs. Each pair consists of:

• Input vector (features)

• Output value (label)

In regression, our output value is quantitative, and in classification, our output value is categorical.

Unsupervised Learning¶

In unsupervised learning, our goal is to identify patterns in unlabeled data. In this type of learning, we do not have input/output pairs. Sometimes, we may have labels but choose to ignore them (e.g. PCA on labeled data). Instead, we are more interested in the inherent structure of the data we have rather than trying to simply predict a label using that structure of data. For example, if we are interested in dimensionality reduction, we can use PCA to reduce our data to a lower dimension.

Now, let’s consider a new problem: clustering.

Clustering Examples¶

First off, let’s see some use cases for clustering

Example 1 - Image Compression¶

Clustering can be used for image compression. Why? Digital images consume significant storage and bandwidth, so reducing the number of colors simplifies images while retaining their visual appeal.

• Clustering can group similar colors (pixels) in an image into clusters.

• Each cluster represents a color centroid (mean color of the group).

• Replace each pixel in the image with the color of its cluster centroid, reducing the number of unique colors.

In the example below, notice that using more clusters will make the compressed image look more similar to the original image, while using fewer clusters will look more like a silhouette of the image. Finding a good balance can help resemble the original image while storing less information.

Example 2 - Clustering in Social Networks¶

In social network clustering, we identify groups (clusters or communities) of individuals who interact more frequently with each other. This can help analyze network behavior, predict relationships, and recommend connections. Some applications include:

• Detecting communities on social network platforms.

• Understanding collaboration in organizations.

• Optimizing targeted marketing strategies.

Shown above is a synthetic graph representing interactions in a photography club, where each Node represents a member of the club and each Edge is an interaction between members (or Nodes). Members can be clustered based on the people they interact the most with.

Example 3 - Clustering in Climate Sciences¶

Clustering in climate science helps to identify patterns in large, complex datasets and provide insights into global and regional climate trends. Some examples include:

• Climate zone identification

• Weather pattern analysis

• Environmental monitoring

In this example, we grouped regions based on average temperatures to understand climate zones.

There are many types of clustering algorithms, and they all have strengths, inherent weaknesses, and different use cases. There are two main groups of clustering algorithms we will focus on: Agglomerative approaches to clustering, and Partitional approaches to clustering.

We will first focus on a partitional approach: K-Means clustering.

K-Means Clustering¶

The most popular clustering approach is K-Means. The algorithm itself entails the following:

1. Pick an arbitrary $k$, and randomly place $k$ “centers”, each a different color.

2. Repeat until convergence:

   1. Color points according to the closest center (also called centroids). b. Move the center for each color to the center of points with that color.

Note: K-Means is a completely different algorithm than K-Nearest Neighbors. K-means is used for clustering, where each point is assigned to one of $K$ clusters. On the other hand, K-Nearest Neighbors is used for classification (or, less often, regression), and the predicted value is typically the most common class among the $K$-nearest data points in the training set.

One major difference between these two is that $K$-Nearest Neighbors is used for supervised learning, where we have labeled input and output pairs, and $K$-means is used for unsupervised learning, where we only have features, and no label that we’re trying to predict.

The names may be similar, but there isn’t really anything in common.

Due to the randomness of where the $K$ centers initialize/start, you will get a different output/clustering every time you run $K$-Means. Consider three possible $K$-Means outputs; the algorithm has converged, and the colors denote the final cluster they are clustered as.

Which clustering output is the best? To evaluate different clustering results, we need a loss function.

The two common loss functions are:

• Inertia: Sum of squared distances from each data point to its center.

• Distortion: Weighted sum of squared distances from each data point to its center.

In the example above:

• Calculated Inertia: $0.47^2 + 0.19^2 + 0.34^2 + 0.25^2 + 0.58^2 + 0.36^2 + 0.44^2$

• Calculated Distortion: $\frac{0.47^2 + 0.19^2 + 0.34^2}{3} + \frac{0.25^2 + 0.58^2 + 0.36^2 + 0.44^2}{4}$

It turns out that the function K-Means is trying to minimize is inertia, but often fails to find global optimum. Why does this happen? We can think of K-means as a pair of optimizers that take turns. The first optimizer holds center positions constant and optimizes data colors. The second optimizer holds data colors constant and optimizes center positions.

This is a hard problem: give an algorithm that optimizes inertia FOR A GIVEN $K$; $K$ is picked in advance. Your algorithm should return the EXACT best centers and colors, but you don’t need to worry about runtime.

We won’t dwell too much on this problem as it delves deep into material from CSCI 350/353/761.

• For all possible $k^n$ colorings:

  • Compute the $k$ centers for that coloring

  • Compute the inertia for the $k$ centers

    • If the current inertia is better than the best known, write down the current centers and colorng and call that the new best known

No better algorithm has been found for solving the problem of minimizing inertia exactly.

Hierarchical Agglomerative Clustering¶

Now, let us introduce Hierarchical Agglomerative Clustering! We start with every data point in a separate cluster, and we’ll keep merging the most similar pairs of data points/clusters until we have one big cluster left. This is called a bottom-up or agglomerative method.

There are various ways to decide the order of combining clusters called Linkage Criterion:

• Single linkage (similarity of the most similar): the distance between two clusters as the minimum distance between a point in the first cluster and a point in the second.

• Average linkage: the distance between two clusters as the average of all pairwise distances between points in the first cluster and points in the second.

• Complete linkage (similarity of the least similar): the distance between two clusters as the maximum distance between a point in the first cluster and a point in the second.

The linkage criterion decides how we measure the “distance” between two clusters.

When the algorithm starts, every data point is in its own cluster. In the plot below, there are 12 data points, so the algorithm starts with 12 clusters. As the clustering begins, it assesses which clusters are the closest together.

Clustering, Dendrograms, and Intuition¶

Agglomerative clustering is one form of “hierarchical clustering.” It is interpretable because we can keep track of when two clusters got merged (each cluster is a tree), and we can visualize the merging hierarchy, resulting in a “dendrogram.” Won’t discuss this in detail for this course, but you might see these in the wild. Here are some examples:

Applying Clustering¶

Population Data¶

The algorithms we’ve discussed require us to pick a $K$ before we start. But how do we pick $K$? Often, the best $K$ is subjective. For example, consider the state plot below.