TDM 19000: Image Processing

Objectives

Motivation: Computers cannot interpret images the way humans do. Instead, images must be converted into numerical representations of themselves that capture import visual features. Learning how image embeddings are created and compared allows us to analyze them, and discover patterns in large collections of images.

Context: Students will use the CLIP model to process images and generate image embeddings, compare images using cosine similarity, and use clustering to group visually similar images based on their extracted features

Scope: Image Processing, CLIP, image embedding, clustering

Learning Objectives

  • Understand how computers represent visual information using numerical feature vectors.

  • Generate image embeddings using a pre-trained CLIP model.

  • Compare images based on their similarity scores

  • Group images based on visually similar characteristics to create clusters

Dataset

  • /anvil/projects/tdm/data/images/Snails/*

Snails

The Snails folder contains 30 different images of snails - sourced from this Kaggle Agricultural Pests Image Dataset. The original dataset had 500 images of snails, as well as folders focusing on other animals. This subset was selected because it is a nice starting amount of images to work with without getting overwhelming results at the beginning.

The purpose of this dataset is to give us a sampling of images to break down into vectors, so that we can begin learning about embedding images - putting them into a form the computer will understand. Any dataset of images is fine to use; this was one we found interesting because the snails themselves vary in shape, color, species, and size, as well as the backgrounds of the images can be leaves, dirt, conrete, and other surface, while all still focusing on a singular snail to create consistency across the 30 images.

Humans are naturally good at finding patterns and understanding context that causes something to look how it does - humans instantly process visual information. It’s how we make sense of the world.

Computers excel at taking text, and searching, sorting, and analyzing it at high speeds. Much faster than humans can do. But a computer cannot technically "see" an image - it sees a collection of pixels (or RGB values) that make up any image it encounters.

Image processing allows computers to analyze and interpret visual data. This process is done by taking raw visual data, and transforming it into representations of the image that can be analyzed by a computer. These techniques are now commonly combined with machine learning models, in order to identify patterns across large quantities of images at once.

We will apply image processing and feature extraction techniques to convert a set of images into numeric embeddings, and then create groups based on similarity.

We will work with the CLIP Model and Processor. There are numerous other models that are more modern now, including (for instance) Apple DFN, BLIP-2, EVA-CLIP, Long-CLIP, MetaCLIP, OpenCLIP, SigLIP, etc.

First, we need to setup the CLIP Model and Processor: the-examples-book.com/projects/summer2026/19000/settingupclip

Now we can actually begin working with the model, and have it process some images.

From the 'Snails' folder, choose one image to read in (using Image.open) and view, as done below.

import os

folder_path = "/anvil/projects/tdm/data/images/Snails"

# print the names of the images
for filename in os.listdir(folder_path):
    print(filename)
from PIL import Image

# change the selected image name as needed
my_image = Image.open("/anvil/projects/tdm/data/images/Snails/snail (179).jpg")

my_image
Chosen snail image

Think of how you might describe the image you’ve just displayed. Some common themes amongst the images in this dataset are:

  • Very green and leafy background

  • Snail is fully inside / extended out of its shell

  • Clear view of shell spiral

  • Snail on concrete / road

Just as we can think of images by a set of key features, so will the computer…​just a bit differently.

The CLIP Model and Processor will convert the image into vectors using embeddings: the-examples-book.com/projects/summer2026/19000/imageembeddings

One of the most natural concepts to learn is how to compare images.

Again, just as with large language models, one way to make comparisons is to use cosign similarity.

We give some examples here: the-examples-book.com/projects/summer2026/19000/imagesimilarity

We can even make a heatmap of how each of the 30 images compares to all of the other images: the-examples-book.com/projects/summer2026/19000/heatmap

We have found which pairs of images are the most similar. This is useful for direct comparison. But what if we want to find a lot of images that are similar? Or if we would like to divide up an entire folder of images based on similarity? We can use clustering: the-examples-book.com/projects/summer2026/19000/clustering