unit 0.2 - Introduction to PyTorch and Tensors

Note: This notebook has input from PyTorch tutorials!

Tensors

Tensors is the way for PyTorch to represent complex sets of numbers. Tensors are multi-dimensional vectors of data, can be 1D, 2D, 3D, 4D. 5D etc. They are the equivalent of numpy arrays. They are the fundamental data used by PyTorch for all neural network operations.

import torch

A vector:

a = torch.Tensor(10)
a

tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

A more complex tensor - seems like an image of 4x4 pixels:

a = torch.Tensor(3,4,4)
a
# a[0:2]

tensor([[[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]],

        [[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]],

        [[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]]])

But what is the first dimension? RGB?

Here is a black/white image - why so?

a = torch.Tensor(5,5)
a

tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])

Tensors can be created directly from data. The data type is automatically inferred.

data = [[1, 2],[3, 4]]
x_data = torch.tensor(data)
print(x_data)

tensor([[1, 2],
        [3, 4]])

creating a tensor from numpy and vice-versa

import numpy as np
np_array = np.array(data)
x_np = torch.from_numpy(np_array)
print(x_np)

tensor([[1, 2],
        [3, 4]])

we can also copy tensors and make similar ones with same dims:

x_ones = torch.ones_like(x_data) # retains the properties of x_data
print(f"Ones Tensor: \n {x_ones} \n")

x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")

Ones Tensor:
 tensor([[1, 1],
        [1, 1]])

Random Tensor:
 tensor([[0.0158, 0.3978],
        [0.9604, 0.9592]])

we can create random tensors and also fill them with 0s, 1s

shape = (2,3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")

Random Tensor:
 tensor([[0.0121, 0.6030, 0.4696],
        [0.3624, 0.0542, 0.7875]])

Ones Tensor:
 tensor([[1., 1., 1.],
        [1., 1., 1.]])

Zeros Tensor:
 tensor([[0., 0., 0.],
        [0., 0., 0.]])

We can describe tensors shape, datatype, and the device on which they are stored

tensor = torch.rand(3,4)

print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")

Shape of tensor: torch.Size([3, 4])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu

Operations on Tensors

Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing, indexing, slicing), sampling and more are comprehensively described here_.

Each of these operations can be run on the GPU (at typically higher speeds than on a CPU). If you’re using Colab, allocate a GPU by going to Runtime > Change runtime type > GPU.

By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using .to method (after checking for GPU availability). Keep in mind that copying large tensors across devices can be expensive in terms of time and memory!

# We move our tensor to the GPU if available
if torch.cuda.is_available():
    tensor = tensor.to("cuda")

Try out some of the operations from the list. If you’re familiar with the NumPy API, you’ll find the Tensor API a breeze to use.

Standard numpy-like indexing and slicing:

tensor = torch.rand(4, 4)
print(tensor)
print(f"First row: {tensor[0]}")
print(f"First column: {tensor[:, 0]}")
print(f"Last column: {tensor[..., -1]}")
tensor[:,1] = 0
print(tensor)

tensor([[0.0444, 0.6594, 0.8852, 0.4272],
        [0.8485, 0.8111, 0.0237, 0.4294],
        [0.1016, 0.6855, 0.5890, 0.4343],
        [0.7649, 0.6575, 0.4558, 0.8507]])
First row: tensor([0.0444, 0.6594, 0.8852, 0.4272])
First column: tensor([0.0444, 0.8485, 0.1016, 0.7649])
Last column: tensor([0.4272, 0.4294, 0.4343, 0.8507])
tensor([[0.0444, 0.0000, 0.8852, 0.4272],
        [0.8485, 0.0000, 0.0237, 0.4294],
        [0.1016, 0.0000, 0.5890, 0.4343],
        [0.7649, 0.0000, 0.4558, 0.8507]])

Joining tensors You can use torch.cat to concatenate a sequence of tensors along a given dimension. See also torch.stack_, another tensor joining operator that is subtly different from torch.cat.

t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)

tensor([[0.0444, 0.0000, 0.8852, 0.4272, 0.0444, 0.0000, 0.8852, 0.4272, 0.0444,
         0.0000, 0.8852, 0.4272],
        [0.8485, 0.0000, 0.0237, 0.4294, 0.8485, 0.0000, 0.0237, 0.4294, 0.8485,
         0.0000, 0.0237, 0.4294],
        [0.1016, 0.0000, 0.5890, 0.4343, 0.1016, 0.0000, 0.5890, 0.4343, 0.1016,
         0.0000, 0.5890, 0.4343],
        [0.7649, 0.0000, 0.4558, 0.8507, 0.7649, 0.0000, 0.4558, 0.8507, 0.7649,
         0.0000, 0.4558, 0.8507]])

Arithmetic operations

y0 = tensor + tensor
print("Sum:", y0)

y0 = 2*tensor - tensor
print("Diff and multiply constant :", y0)

# This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value
# ``tensor.T`` returns the transpose of a tensor
y1 = tensor @ tensor.T
y2 = tensor.matmul(tensor.T)

y3 = torch.rand_like(y1)
torch.matmul(tensor, tensor.T, out=y3)


# This computes the element-wise product. z1, z2, z3 will have the same value
z1 = tensor * tensor
z2 = tensor.mul(tensor)

z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)

Sum: tensor([[0.0888, 0.0000, 1.7705, 0.8544],
        [1.6970, 0.0000, 0.0475, 0.8587],
        [0.2032, 0.0000, 1.1781, 0.8686],
        [1.5298, 0.0000, 0.9115, 1.7014]])
Diff and multiply constant : tensor([[0.0444, 0.0000, 0.8852, 0.4272],
        [0.8485, 0.0000, 0.0237, 0.4294],
        [0.1016, 0.0000, 0.5890, 0.4343],
        [0.7649, 0.0000, 0.4558, 0.8507]])

tensor([[1.9727e-03, 0.0000e+00, 7.8363e-01, 1.8249e-01],
        [7.1999e-01, 0.0000e+00, 5.6405e-04, 1.8435e-01],
        [1.0318e-02, 0.0000e+00, 3.4696e-01, 1.8861e-01],
        [5.8508e-01, 0.0000e+00, 2.0771e-01, 7.2372e-01]])

Single-element tensors If you have a one-element tensor, for example by aggregating all values of a tensor into one value, you can convert it to a Python numerical value using item():

agg = tensor.sum()
agg_item = agg.item()
print(agg_item, type(agg_item))

5.854728698730469 <class 'float'>