TDM 19000: Image Similarity

If we wanted to compare a group of images and decide which were the most similar, sometimes it is easy to do. In this group, it would be the two images of snails:

snail (179) bike 207 snail (103) cat.1

But in a group with very visually similar images, it is a bit harder for us to just look and know:

snail (261) snail (304) snail (262)

All three snails look very similar.

The CLIP model is able to compare how similar images are based on the values in their feature vectors. We will build a loop to read in every image of the Snails folder. For each image, it will be important to make sure that it goes through processor and model so the key description features get stored correctly.

First we specify the location of the files, so that we are prepared to loop through them:

# lists to contain the names of the files and the image vectors
filenames = []
features = []

folder = "/anvil/projects/tdm/data/images/Snails"

We need torch in order to convert the images into tensors.

import torch

Now we loop through the files, running the processor on each image. The no.grad means that we will do this in a light way, with no gradient calculations.

for file in os.listdir(folder):
    if file.endswith(".jpg"):
        path = os.path.join(folder, file)

        with Image.open(path) as image:
            # Define 'inputs' to be the result of running processor on 'image'
            # (so processor will be run on each image)
            # Return the images in a format that CLIP can process
            inputs = processor(images=image, return_tensors="pt")


        with torch.no_grad():
            # 'image_features' should take the processed images and use
            # model to convert them into embeddings
            # to store what each images "means"
            image_features = model.get_image_features(**inputs)

        # We normalize the image features just as we did before
        normalized_img_feats = image_features / image_features.norm(dim=-1, keepdim=True)

        features.append(normalized_img_feats)
        filenames.append(file)

        print(file)

Print the length of features to ensure that it contains all 30 snail images.

We can use:

features = torch.cat(features, dim=0)

to take the list of individual image vectors (from each image) and combine them into one big tensor. Each row of combined_features represents one image embedding.

Now we will use cosine similarity, to compare how similar the images are.

To start very simply, compare the just two images to find their similarity.

# compare image #0 to image #1
first_similarity = torch.matmul(features[0], features[1])

print(first_similarity)

In cosine similarity, there are three score categories:

  • -1 - items are dissimilar

  • 0 - items are neutral

  • 1 - items are identical

The similarity scores between the images will follow these patterns.