TDM 19000: Week 2 Questions — Summer 2026

1.1 Explain (in 1 to 2 sentences) how computers "understand" visual content; your answer can be at a high level, but please explain what you know so far!

1.2 Direct the CLIP model to store downloads in the SCRATCH directory

1.3 Load the model and processor to prepare for image processing

2.1 Read in an image from the Snails dataset folder, and display it. What are some key features you notice about the image?

2.2 Convert the snail image into a dictionary of tensors

2.3 Unpack the dictionary (inputs) to find keyword arguments

2.4 Explain (in 2 to 3 sentences) how we are using processor and model to make the snail image model-friendly

3.1 How might the CLIP model decide that the three images of snails (shown above: snail (261).jpg, and snail (304).jpg, and snail (262).jpg) are more similar to each other than to an image of a bike or a cat?

3.2 Read in the 30 snail images, and extract a feature vector for each

3.3 Combine the vectors into one total list of vectors

3.4 Show the similarity score of image #0 to image #1. Are these two images dissimilar, neutral, or similar?

4.1 Which snail images (from the initial cosine similarity scores heatmap) are similar? Which are dissimilar?

4.2 Show the image pairings of snails with the top 5 similarity scores.

5.1 Create clusters to sort the snail images into

5.2 Show the names of the images within each cluster group (there are three groups: please do this for cluster 0, cluster 1, and cluster 2)

5.3 Show the images within each cluster group of snails (three clusters of images)

5.4 In English (in 2 to 3 sentences), what main characteristics are you noticing of the images in each cluster?

6.1 Pick two Moths (of your own choosing) and find the similarity score for these two Moths, as you did in question 3.4 for the two snails.

6.2 Show the image pairings of Moths with the top 5 similarity scores, as you did in question 4.2 for the pairs of snails.

7.1 Find two images of your own choosing (neither snails nor moths) which are very similar, according to their similarity scores.

7.2 Find two images of your own choosing (neither snails nor moths) which are very different, according to their similarity scores.

8.1 Upload several images (at least 10 images) of your own choosing.

8.2 Split these images (at least 10 images) into two clusters. It is OK, for instance, to upload 5 pictures of your dog, and 5 pictures of balloons, and see if they get classified into two clusters in the correct way.

9.1 Explain (2-3 sentences) what YOLO is and how it is used in image detection.

9.2 Run the model on the image for the shopping mall. What objects is it predicting?

9.3 Print the results of 'counts', and explain (in 1 to 2 sentences) how the model is indicating that there are people in the image.

9.4 Display the original image, with the bounding boxes, object labels, and confidence scores from yolo_results overlaid.

10.1 Explain (in 1 to 2 sentences) what SAM is, and what it is used for in image processing

10.2 Define and predict masks for each object in the shopping mall image, using SAM

10.3 Assign, resize, and overlay the mask for one object on the original image.

11.1 Use Method 1 to display an image for each object mask in the shopping mall image. Display the result.

11.2 Use Method 2 to take the object masks and put them all onto in the shopping mall image at once. Display the result.

12.1 Upload an image of your own, and use YOLO to print the results of 'counts', indicating how many people and objects are in the image. Give 1 to 2 sentences explaining what YOLO discovered.

12.2 Display the image you selected, with the bounding boxes, object labels, and confidence scores from yolo_results overlaid.

12.3 Now use SAM to define and predict masks for the image of your own choosing (same image from questions 12.1 and 12.2), using SAM. You already did this in question 10.2 for the shopping mall image; do the same thing with SAM for your image.