TDM 20100: Project 5 — Fall 2026

  • Image Processing (Part 2)

Motivation: Computers cannot interpret images the way humans do. Instead, images must be converted into numerical representations of themselves that capture import visual features. Learning how image embeddings are created and compared allows us to analyze them, and discover patterns in large collections of images.

Context: In this project, students will use the CLIP model to process images and generate image embeddings, compare images using cosine similarity, and use clustering to group visually similar images based on their extracted features

Scope: Image Processing, CLIP, image embedding, clustering

Learning Objectives

  • Understand how computers represent visual information using numerical feature vectors.

  • Generate image embeddings using a pre-trained CLIP model.

  • Compare images based on their similarity scores

  • Group images based on visually similar characteristics to create clusters

Dataset

  • /anvil/projects/tdm/data/images/Moths/*

We continue the topics from Project 4 on image processing.

Questions 1, 2, 3 are an extention of the topics already learned in Project 4.

Questions 4 and 5 introduce you to YOLO.

Question 1

The Moths images are stored at:

/anvil/projects/tdm/data/images/Moths

1.1 Pick two Moths (of your own choosing) and find the similarity score for these two Moths, as you did in Project 3, Question 3.4, for the two snails.

1.2 Show the image pairings of Moths with the top 5 similarity scores, as you did in Project 3, Question 4.2 for the pairs of snails.

Question 2

You can upload pictures of your own choosing to Anvil for this question:

2.1 Find two images of your own choosing (neither snails nor moths) which are very similar, according to their similarity scores.

2.2 Find two images of your own choosing (neither snails nor moths) which are very different, according to their similarity scores.

Question 3

You can continue to upload pictures of your own choosing to Anvil for this question:

3.1 Upload several images (at least 10 images) of your own choosing.

3.2 Split these images (at least 10 images) into two clusters. It is OK, for instance, to upload 5 pictures of your dog, and 5 pictures of balloons, and see if they get classified into two clusters in the correct way.

Introduction/Background to Question 4 and Question 5

The image screenshot:

/anvil/projects/tdm/data/images/seq_000014.jpg

is from a shopping mall security camera video clip. There are some people scattered throughout the scene. The camera footage is not particularly clear. The computer will still recognize (most of) these figures as people when we go to classify the objects in the image.

Object detection is a computer vision technique in image processing that is used to identify, label, and locate specific objects from within images or videos. This method uses bounding boxes to show what items it has detected, and labels them accordingly. This is commonly done with cars, people, or animals in many different contexts.

One of the faster approaches to perform object detection is through YOLO.

Read these notes about YOLO: the-examples-book.com/projects/fall2026/20100/introductiontoyolo

and then answer Question 4 and Question 5.

Question 4

4.1 Explain (2-3 sentences) what YOLO is and how it is used in image detection.

4.2 Run the model on the image for the shopping mall. What objects is it predicting?

Question 5

5.1 Print the results of 'counts', and explain (in 1 to 2 sentences) how the model is indicating that there are people in the image.

5.2 Display the original image, with the bounding boxes, object labels, and confidence scores from yolo_results overlaid.

Submitting your Work

Please make sure that you added comments for each question, which explain your thinking about your method of solving each question. Please also make sure that your work is your own work, and that any outside sources (people, internet pages, generative AI, etc.) are cited properly in the project template.

Prior to submitting your work, you need to put your work into the project template, and re-run all of the code in Jupyter Lab and make sure that the results of running that code is visible in your template. Please check the detailed instructions on how to ensure that your submission is formatted correctly. To download your completed project, you can right-click on the file in the file explorer and click 'download'.

Items to submit

  • firstname_lastname_project5.ipynb

It is necessary to document your work, with comments about each solution. All of your work needs to be your own work, with citations to any source that you used. Please make sure that your work is your own work, and that any outside sources (people, internet pages, generative AI, etc.) are cited properly in the project template.

You must double check your .ipynb after submitting it in gradescope. A very common mistake is to assume that your .ipynb file has been rendered properly and contains your code, markdown, and code output even though it may not.

Please take the time to double check your work. See here for instructions on how to double check this.

You will not receive full credit if your .ipynb file does not contain all of the information you expect it to, or if it does not render properly in Gradescope. Please ask a TA if you need help with this.