TDM 40200: Project 02 - Image Preprocessing

Firstly, let’s review what we learned from last project. Please load the /anvil/projects/tdm/data/icecream/hd/images/56_hd.png image into a variable called image_opencv, print out its dimensions, and display it using matplotlib and OpenCV. Additionally, please explain any techniques you had to do to ensure the image was displayed correctly.

Next, let’s learn about cropping and resizing images. Firstly, we will look at cropping images. There are many reasons to crop an image, but a major one is removing unwanted parts of the image such as background. Typically, images are cropped by specifying the coordinates of the top left and bottom right corners of the desired region. For example, if we had a 1000x1000 pixel image and wanted to crop the center 500x500 pixels, we would specify the top left corner as (250, 250) and the bottom right corner as (750, 750).

Please review the below function to crop an image based on the input coordinates. This function will take in an image and the cropping coordinates (tuples) and return the cropped image. Make sure you understand the underlying mechanics here (slicing an image), as it will be used in future projects.

To test your function, please run the below test cases. This should crop the center 200x200 pixels of the image, the top left 60x60 pixels, and the bottom right 150x35 pixels.

Even if we crop an image down to just its most important region, the image may still be too large for our model to handle. This is where resizing or downscaling an image comes in. Although decreasing the size of an image inherently results in a loss of information, which sounds bad for our model, it is actually often very beneficial. Think back to last semester where we discussed the curse of dimensionality. In a 500x500 pixel RGB image, there are 750,000 values. Resizing this image to 100x100 pixels shrinks that down to 30,000 values. This is a much more manageable number of inputs for our model to handle.

There are many different methods for downsizing (and upsizing) images, but one of the most common and simple is bilinear interpolation. Bilinear interpolation is a method for estimating the value of a function at a point by averaging the values of its closest known points. For example, if we know that f(1) = 2 and f(3) = 4, we can use bilinear interpolation to estimate f(2) = 3. When it comes to images, bilinear interpolation is used on the pixel values/color channels to estimate the new pixel values.

OpenCV has a built-in function for resizing images that defaults to bilinear interpolation, cv2.resize. In addition to bilinear interpolation, OpenCV also has support for nearest neighbor interpolation, bicubic interpolation, area interpolation, and Lanczos interpolation. Each of these methods has their own strengths and weaknesses, and their methodology is explained briefly below:

To start, let’s try to resize the image to 100x100 pixels using bilinear interpolation. Please run the below code to resize the image and display it.

As you can see, the image is quite pixelated, yet we can still clearly see the important features of the image. However, some of you may notice that the ice cream looks a little stretched out. This is because we actually changed the image’s aspect ratio with the last operation. This can lead to distortion in the image, including stretching or squishing. An images aspect ratio is simply its width divided by its height. To maintain the aspect ratio of the image, the output width and height should also have the same aspect ratio (see for more details: learnopencv.com/image-resizing-with-opencv/). This may be challenging to do when resizing an image, but luckily OpenCV also supports scaling images down while maintaining or adjusting the aspect ratio. The optional parameters fx and fy can be used to scale the image by a factor in the x and y directions, respectively.

To test this out, please run the below code to resize the image to 1/5th of its original size while maintaining the aspect ratio.

Now that you know how to resize images, please resize the original image to a smaller size and compare the nearest neighbor, bilinear interpolation, and area interpolation methods. Please display the images and point out any differences you see between the methods. Which resulting image do you think looks the best?

Now that we understand cropping and resizing, another important preprocessing technique is filtering images. Filters are a wide range of operations that can be applied to blur images, sharpen images, detect edges, and much more. The mathematics behind these filters is quite complex at times, but the general idea is that a matrix is convolved across the image to produce a new image. That matrix is called a kernel, and the size and values of said kernel determine its effects.

OpenCV provides a generic cv2.filter2D function that can be used to apply any kernel to an image (see for more details: learnopencv.com/image-filtering-using-convolution-in-opencv/). Run the code below to apply a simple 7x7 averaging filter to the image.

You should be able to see that the image is now somewhat blurry. This is because the averaging filter works by simply taking the average of (in this case) the 49 pixels surrounding the pixel in question. This has the effect of smoothing out the image.

Arguably the most common filter is the Gaussian filter, which is a low-pass filter used to blur images. This filter is very often used as a preprocessing step before applying other filters such as edge detection or running a model. OpenCV provides a cv2.GaussianBlur function that can be used to apply a Gaussian filter to an image. This function takes in an image along with the desired kernel size, computes the Gaussian kernel, and convolves it across the image. Run the code below to apply a Gaussian filter to the image.

You should see that the image is now blurrier than before, but the edges are more preserved than the averaging filter.

Please select 3 different kernel sizes for the gaussian filter and display the resulting images. What is the correlation between the kernel size and the blurriness of the image?

In addition to blurring images, filters can also be used to sharpen images by enhancing the edges. One of the most common sharpening filters is Laplacian filter, which is a high-pass filter (in contrast to the low-pass Gaussian filter, e.g.: www.geeksforgeeks.org/difference-between-low-pass-filter-and-high-pass-filter/). This filter works by taking the second derivative of the image, which highlights the edges. OpenCV provides a cv2.Laplacian function that can be used to apply a Laplacian filter to an image. Run the code below to apply a Laplacian filter to the image.

You should see the image has very enhanced edges, but also a lot of noise. This is because the Laplacian filter is very sensitive to noise. To reduce the noise, the image can be blurred before applying the Laplacian filter. This is a common technique called edge detection.

Another popular filter is the Sobel filter which is used to detect edges in the images. This filter is more complex, and actually involves two filters: one in the x direction and one in the y direction. The two resulting images are then combined to produce the final image. OpenCV provides a cv2.Sobel function that can be used to apply a Sobel filter to an image. Run the code below to apply a Sobel filter to the image.

In this, you should see the more vertically aligned edges in the image have become more bold. Please modify the code to apply the Sobel filter in the y direction and display the resulting image. What do you see?

Another less common filter is the Scharr filter, which works similarly to the Sobel. OpenCV, again, provides a cv2.Scharr function that can be used to apply a Scharr filter to an image. Run the code below to apply a Scharr filter to the image.

What similarities and differences do you see between the Sobel and Scharr filters?