TDM 20200: Project 5 - Introduction to PyTorch & Tensors

PyTorch is a very popular machine learning framework, as many of you might already have heard of it. It supports training and inference in neural networks. Understanding how PyTorch does this and how it can be used requires us to understand Tensors, which is the core data structure in PyTorch. Without tensors, storing inputs and parameters, or the fast computations would not be possible. So for now, we will focus on understanding what tensors are, operations we can do with them, and relating it to other tools and concepts we already know. We can think of tensors (PyTorch’s Tensor Documentation) as multidimensional arrays (NumPy’s ndarrays).

These are the necessary imports:

We are already very familiar with pandas. import torch loads the PyTorch library into our program, and this is where Tensors are provided from. In this project, we will mostly work with those alongside torch’s mathematical operations, but later once we get deeper into machine learning, torch allows us to perform tasks such as automatic differentiation, or obtain libraries to build and train neural networks.

Let’s first load in our dataset.

Print the head and shape of the dataset. What does this tell us?

In the MNIST dataset, the first column has the digit labels (0-9), and all other columns after that have the pixel values. Each row is where we get the 784 pixel values from. Use .iloc() to select only the first column for 'labels', and all rows and columns starting from the second column for 'pixels'. You can use to_numpy() to convert into a 2D NumPy array.

You should see below shapes:

Let’s check the data types as well.

Running above should output <class 'pandas.core.frame.DataFrame'> and <class 'numpy.ndarray'>. As expected, it tells us that 'data' is a Pandas Data Frame, and 'pixels' is a NumPy array (after all, we did the conversion). PyTorch does not work directly on DataFrames and it can convert NumPy arrays into tensors. This is exactly what we are going to do now.

We can easily do the conversion like this:

Understanding tensor’s basic attributes is necessary before using it further. They describe characteristics of tensor objects, such as shape, datatypes, and other relevant information such as what device they are stored.

Shape

Similarly as before, we use .shape; this returns the same thing as if we were to use tensor.size. They both return torch.Size object which shows us each dimension’s length of the tensor. In tensor.size, you can specify the dimensions as well.

Data Type of Elements .dtype attribute represents the data type of torch.Tensor.

Number of Dimensions .ndim attribute is an alias for dim(), and they both return the number of dimensions.

Below is what you can expect to see.

Device

Although not directly relevant to the work you will do in this project, know that all tensors have a .device attribute as well. It indicates where in memory tensors are located and where operations will occur. Most commonly, they will be "cpu" (system RAM) or "cuda" (GPU).

We mention this, because having the ability to move tensors to a CUDA device allows great acceleration of operations such as matrix multiplications and convolutions, since GPU has a lot more cores optimal for parallel processing. This is one of the key advantages that can be obtained through PyTorch.

We can indicate using torch.device. For example,

A common usage includes selecting a GPU if it is available, otherwise use the CPU.

We can move tensors or models to a device through .device.

We also ensure that all tensors in an operation done are on the same device. PyTorch gives us convenient ways to move tensors or models to different devices. We can explicitly manage the device.

We are changing the shape of the tensor through X.reshape(-1, 1, 28, 28). The returned tensor of .reshape(input, shape) has the same data and number of elements as the input.

If we were to work with other models, such as convolutional neural networks, this would be the expected input structure.

Let’s take a look at a sample image!

Element Wise Operations

Element wise operations is implemented on tensors with same dimensions. As the name suggests, the operations as element by element:

$z_{i,j} = x_{i,j} \ (*) \ y_{i,j}$,

where z is the resulting element at the same indices as x and y. (*) is the operation.

Also, although we are showing examples through using a dataset, we can create our own tensors from scratch too!

Output:

Common operations include:

1) Addition (torch.add()) and Subtraction (torch.sub())

Here is an example.

Output:

2) Multiplication (torch.mul()) and Division (torch.div())

print("Scalar Multiplication:\n", x * 2)

Output:

3) Comparisons and Logical operation

We can use standard python operators for element wise comparisons amongst tensors. For example, > (greater than), >= (greater than or equal to), < (less than), == (equal).

Here is an example.

Output:

The first example has the condition x>3; thus, it checks every element in x to see if the value is greater than 3. The first three results are False, since 1,2,3 are not greater than 3, and the last three results are True, since 4,5,6 are greater than 3. The result is a Boolean tensor, where either True or False is stored in each position.

The second example compares each element at the same position for matching values: 1 == 7 is false, 2 == 9 is false, …​, 6 == 12 is false. Since no values are equal, we obtain all False results.

Below functions return a tensor of Boolean values after computing element wise logical operations between input and other tensors.

Example:

First, given case1 = x > 2, every value in x is compared to 2. Since 1,2 are less than 2, they are False. On the other hand, 3-6 are greater than 2, so they are True.

Similarly, given case2 = y > 25, all values in y are compared against 25. It will be all False, as no values are greater than 25.

You can print them to check yourself, as well.

Now, torch.logical_and(case1, case2) performs AND operations between each of these True/False values. Notice all values in case2 is False, hence the result will be all False.

Similarly for torch.logical_or(case1, case2), the computation becomes 'False OR False', 'False OR False', …​, 'True OR False', and 'True OR False'.

Output of the print statements:

Broadcasting

But what if our tensors do not have the same shape? Broadcasting is the technique that allows us to still perform element wise operations between tensors with different shapes. An intuitive way to think about this is matching the smaller tensor to the size of the larger tensor; we are not copying over data but extending it (advantageous for memory).

We will explore this concept through an example where we calculate the average brightness of each image.

We can calculate across the horizontal axis by setting 'dim=1'.

Please calculate the same using NumPy as well. You should see both outputs like:

Note that we set unbiased=False when calculating the standard deviation using PyTorch. This is because sometimes we add in Bessel’s Correction (dividing by N-1 instead of N); however, here we will just use N which is also NumPy’s default setting.

Calculating these actually had broadcasting implicitly! We can go step by step, by manually calculating mean and standard deviation, and explicitly see it.

Please write code to calculate sd_ver2 and norm yourself, and print the shapes for all four of them! Then, you should see:

There are two rules PyTorch follows in order to broadcast:

1) Comparing dimensions right to left, the dimensions must either be equal or one of them must be 1 (if one of them is missing, the '1' is added).
2) If one of the dimension is 1, that is the one that gets expanded.

If sizes of both tensors do not match and there is no '1' dimension, then they can not be broadcasted, and you will encounter a RunTimeError.

The first broadcast occurs when X_centered is calculated. Take a look at the shape of X and mean_ver2 from earlier. It was [60000, 784] and [60000, 1]. Print the shape of X_centered; it should be [60000, 784].

PyTorch automatically expanded mean_ver2 from [60000, 1] to [60000, 784] to be able to subtract each row element wise.

Broadcasting worked because comparing 784 and 1, 1 is allowed to be expanded to 784, and comparing 60000 and 60000, they are the same size, so again this meets the broadcasting requirements.

Two more broadcasts occurs when X_norm is calculated. Please write code to calculate this manually by following the formula, and print the shape of X_norm.

Printing the shape should give you: X_norm shape: torch.Size([60000, 784]). Please explain how and why broadcasting was applied here; please include consideration of the shape of sd_ver2, as well as both scalar addition and division that takes place that contributes to broadcastings.

Matrix Multiplication

The main difference between NumPy and PyTorch, especially in performing matrix multiplication is that PyTorch tensors can use GPUs to accelerate computation if it is available (we do not have that option here, so we will continue with CPU). The GPU acceleration is provided through parallelized hardware, where numerous cores lead to parallel computation. NumPy performs CPU based operations, which is efficient for the datasets we so far have been using, alongside the general numerical computations, but is not as optimal for further deep learning tasks.

Let’s take a look at some basic operation options. There are a few different ways to perform matrix multiplication.

Below is a simple example.

Each resulting element is a weighted sum of the row and column elements. This is very important when it comes to linear transformations.

We can try above in an example using our dataset. Suppose we want to map 784 inputs into 10 outputs, one per digit.

This represents 784 input features and 10 output, with random weights. We can perform matrix multiplication here, because the 784 dimension between the two matches.

$(m, n) @ (n, p) = (m, p) \rightarrow (60000, 784) @ (784, 10) = (60000, 10)$

Basically, each pixel is multiplied by a weight and the weighted pixels are summed. If you think about this by only looking at one image,

$x_{i} @ r \rightarrow (1, 784) @ (784, 10) = (1, 10)$

Which gives us: [number for 0, number for 1, …​, number for 9]. We get 10 computed numbers for one image, each corresponding to a digit (0-9).

This was a simple case of linear transformation, but the underlying concept is relevant to neural networks, because they also compute:

$z = Wx + b$,

where W is the weight matrix, x is the input vector, and b is the bias vector. Each neuron in a layer computes weighted sum as well, and although we did not see it here, PyTorch allows computation for all images simultaneously through matrix multiplication, which is significantly more efficient especially if GPU was available.