TDM 40200: Project 2 - Convolutional Neural Network

Let’s start by loading in the MNIST dataset.

The output of head() should look like:

We will start by creating a class to load the MNSIT data from the csv file.

torchvision is a library used for computer vision tasks and provides different models, transformation functions, datasets, etc.

torch.utils.data is a pytorch module that allows more efficient data handling for training and evaluation.

transforms.ToTensor() takes in an image or a Numpy array and converts it into PyTorch tensors. The data now ranges from [0,255] to [0.0, 1.0].

transforms.Normalize0.5,), (0.5, (first argument = mean, second argument = standard deviation) uses (input - 0.5 / 0.5) and rescales to [-1,1].

DataLoader() provided by PyTorch is used for loading and processing data efficiently when training models. It works with another object called Dataset, allowing different features such as batching and shuffling be available.

Now, we will create a bar plot to visualize the distribution of the digits (0-9) in the training dataset.

We can convert the list into a Pandas series using pd.Series(). It will also be helpful to use value_counts() to count the occurences of unique values, and sort_index() to sort based on the indices (for digits 0-9). The x axis should be digits 0 through 9 and the y axis should be the number of examples per digit.

Lastly, we will create another visualization to view some sample images.

fig, axes = plt.subplots(1, 5, figsize=(10, 5)): Creates 5 plots in one row. We then get the five sample images, where image is the pytorch tensor representing the digit, and label is the true digit (0-9). train_dataset[i] will return one sample. We then convert the tensor into numpy, and use matplotlib to show the visualization.

Convolutional Neural Networks have three main types of layers, where Convolutional layer is the first, followed by the Pooling Layer and the Fully Connected Layer.

Most of the computation occurs at this stage. An activation function is applied to the output of the convolution operation.

Kernels (also called filters) are small matrices that slide across the image. Convolution is the process of applying these kernels to detect specific features within the image. Convolution is defined with:

$S(i,j) = (I * K)(i,j) = \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} K(m,n)\, I(i-m, j-n)$

where $I$ is the image and $K$ is the kernel. $m$ and $n$ specifies the position (coordinates inside the kernel), and $K(m,n)$ represents the respective weight at row $m$, column $n$.

As another example, in images (2D grid) we compare pixel values around edges and adjacent elements. Mathematically, we can express 2D convolution as:

$S(i,j) = \text{feature map at} (i,j) = (I * K)(i,j) = \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} I(i+m, j+n) \, K(m,n)$

High value in a feature map suggests existence of a pattern in an input region. Convolutional layer incorporates multiple filters to detect different features. Identification of more complex features occurs as we go deeper into the network.

You can see that there is a slight difference in the representation. This is because the first formula uses convolution, while second one is technically a cross correlation. While we just perform multiplication through aligned entries with the filter placed over an area and shifting it throughout, in cross correlation, we flip the filter by 180 degrees before performing element wise multiplication/summation in convolution. Flipping can be thought of as reversing both rows and columns.

However, symmetric kernels produce the same result for both methods. Even if this is not the case, the overall result regardless of the method, is not significantly impacted as the training process allows the learning of the kernel weights, so CNN adapts to it either way.

After convolution, an activation function (like ReLU) is applied to identify complex patterns linear models do not have the capabilities of doing.

Below imports are needed to build our CNN model.

We will be using torch.nn to create our convolutional layers. It is used for classes for neural network components. It has pre built layers and different functions needed for convolution, loss, activation, etc. (docs.pytorch.org/docs/stable/nn.html) nn.Conv2d implements 2D convolutional layer. torch.nn.functional has functions that work only with input data and perform direct operations without working with parameter that gets learned.

See below starter code. self.conv1 is given to you:

self.conv1 = nn.Conv2d(1,8,kernel_size=3, stride=1, padding=1) means output convolution layer with 8 output feature maps after taking in 1-channel input. kernel_size is for defining the size of the filter. Larger the kernel size, more spatial feature we can keep, however the output dimension will be smaller. Stride defines how much the kernel move across the input. Here, saying stride=1 means we are moving the kernel one pixel per movement. padding=1 adds one layer of zeros to the boundary of the input.

The fully connected layer part should reflect the fact that after the second convolutional layer and pooling, the image size becomes (16,7,7) where 16 is the number of channels coming from conv2, and 7x7 is the spatial size. We will use nn.Linear() for this part. The second argument to nn.Linear will be the number of classes in the outcome (corresponds to the number of digits we have).

We also have the forward function which implements the forward pass. We use ReLU as the activation function and max pooling.

Also known as downsampling, this operation reduces the spatial size of feature maps. 2D filter is applied over channels of feature map and obtains the most important features. This operation also leads to a more efficient model as the amount of computation and the number of parameters are reduced.

Two main types of pooling are:

All neurons here are connected to all other neurons in all previous layers. The previously learned features in the layers are used to perform final classification at this layer.

$z_{j} = \sum_{i} (w_{ij}x_{i}) + b_{j}$.

Above describes the way inputs are given to each neuron at this layer. All previous neurons have individual weights with bias. Note that $w_{ij}$ would represent the weight of neuron $i$ at a previous layer, going to neuron $j$. $x_{i}$ is the input.

$a_{j} = f(z_{j})$.

Above denotes the non linear output obtained by passing weighted sum through an activation function.

Now continue with other setups:

nn.CrossEntropyLoss():

Cross Entropy Loss is a common loss function used in classification ML tasks. It measures how closely the model predicted to the real value.

We recommend you to read more about it. Some example sources:
docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html,
www.geeksforgeeks.org/machine-learning/what-is-cross-entropy-loss-function/

optim.Adam(model.parameters(), lr=0.001):

Adam (Adaptive Moment Estimation) is an optimization algorithm that uses Momentum and RMSprop (Root Mean Square Propagation). Momentum makes gradient descent faster through convergence by taking past gradients into consdieration. RMSprop computes the moving average of past squared gradient and prevent the learning rate from decreasing more than it needs to. Our example is not very complicated but there are some very complex models that have heavily varying gradients leading to either explosive or vanishing gradients, and this method tries to counteract that through adaptive learning rate (changes over time).

Convolution stage can lead to loss of information as the output feature map can shrink as any convolution involving filters going out of bounds of the input image are ignored.

To prevent losing information at the border of the image, there is a step (Padding) that adds extra pixel layer on the border of the image. This preserves the original size and maintain the same dimensions throughout the layers for their inputs.

We are going to try out something that lets us see this ourselves. First take a sample image.

image.unsqueeze(0) will add a new dimension of size 1 at index 0. Standard input shape of pytorch cnn is 4D where (batch size, channel, height, width).

But at the end of the day CNN works well. In addition to pooling, different filters (and multiple of them) capture different features and even if certain pixel information is lost, some other filter going over the same region might still use it. Also, CNN models in general, as they go deeper into the layers they end up capturing more sophisticated features although they might start off with detecting more simple features.