TDM 20100: Project 6 — Fall 2026

  • Image Processing (Part 3)

Motivation: The Segment Anything Model (SAM) is a AI foundational model that is designed for image segmentation. This model is able to identify and isolate any object within an image or video with zero-shot generalization.

Context: In this project, students will use the SAM model for masking and removing objects.

Scope: SAM, masks, removing objects

Learning Objectives

  • Understand the SAM model

  • Understand how masks in images work

  • Remove objects from images

We continue the topics from Projects 4 and 5 on image processing.

Question 1

The Segment Anything Model (SAM) is a AI foundational model that is designed for image segmentation. This model is able to identify and isolate any object within an image or video with zero-shot generalization.

We want to introduce you to SAM and to masks, and use it to further our work on the image processing.

Read our notes about SAM here:

the-examples-book.com/projects/fall2026/20100/introductiontosam

1.1 Explain (in 1 to 2 sentences) what SAM is, and what it is used for in image processing

1.2 Define and predict masks for each object in the shopping mall image, using SAM

1.3 Assign, resize, and overlay the mask for one object on the original image.

Question 2

Now that we know how to make masks, we want to make the masks systematically: the-examples-book.com/projects/fall2026/20100/systematicmasks

2.1 Use Method 1 to display an image for each object mask in the shopping mall image. Display the result.

Question 3

3.1 Use Method 2 (from the notes in the previous question) to take the object masks and put them all onto in the shopping mall image at once. Display the result.

Question 4

We can also remove objects from an image, either one object at a time:

the-examples-book.com/projects/fall2026/20100/removingoneobject

or systematically removing multiple objects all at once:

the-examples-book.com/projects/fall2026/20100/removingmultipleobjects

4.1 Upload an image of your own, and use YOLO to print the results of 'counts', indicating how many people and objects are in the image. Give 1 to 2 sentences explaining what YOLO discovered.

4.2 Display the image you selected, with the bounding boxes, object labels, and confidence scores from yolo_results overlaid.

Question 5

5.1 Now use SAM to define and predict masks for the image of your own choosing (same image from question 4), using SAM. You already did this in question 1.2 for the shopping mall image; now do the same thing with SAM for your image.

Submitting your Work

Please make sure that you added comments for each question, which explain your thinking about your method of solving each question. Please also make sure that your work is your own work, and that any outside sources (people, internet pages, generative AI, etc.) are cited properly in the project template.

Prior to submitting your work, you need to put your work into the project template, and re-run all of the code in Jupyter Lab and make sure that the results of running that code is visible in your template. Please check the detailed instructions on how to ensure that your submission is formatted correctly. To download your completed project, you can right-click on the file in the file explorer and click 'download'.

Items to submit

  • firstname_lastname_project6.ipynb

It is necessary to document your work, with comments about each solution. All of your work needs to be your own work, with citations to any source that you used. Please make sure that your work is your own work, and that any outside sources (people, internet pages, generative AI, etc.) are cited properly in the project template.

You must double check your .ipynb after submitting it in gradescope. A very common mistake is to assume that your .ipynb file has been rendered properly and contains your code, markdown, and code output even though it may not.

Please take the time to double check your work. See here for instructions on how to double check this.

You will not receive full credit if your .ipynb file does not contain all of the information you expect it to, or if it does not render properly in Gradescope. Please ask a TA if you need help with this.