{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# unit 0.2 - Introduction to PyTorch and Tensors\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/culurciello/deep-learning-course-source/blob/main/source/lectures/02-tensors-pytorch.ipynb)\n",
        "\n",
        "Note: This notebook has input from [PyTorch tutorials](https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html)!\n",
        "\n",
        "## Tensors\n",
        "\n",
        "Tensors is the way for PyTorch to represent complex sets of numbers. Tensors are multi-dimensional vectors of data, can be 1D, 2D, 3D, 4D. 5D etc. They are the equivalent of numpy arrays. They are the fundamental data used by PyTorch for all neural network operations. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {},
      "outputs": [],
      "source": [
        "import torch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "A vector:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])"
            ]
          },
          "execution_count": 51,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "a = torch.Tensor(10)\n",
        "a"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "A more complex tensor - seems like an image of 4x4 pixels:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor([[[0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.]],\n",
              "\n",
              "        [[0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.]],\n",
              "\n",
              "        [[0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.],\n",
              "         [0., 0., 0., 0.]]])"
            ]
          },
          "execution_count": 52,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "a = torch.Tensor(3,4,4)\n",
        "a\n",
        "# a[0:2]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "But what is the first dimension? RGB?\n",
        "\n",
        "Here is a black/white image - why so?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0.]])"
            ]
          },
          "execution_count": 53,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "a = torch.Tensor(5,5)\n",
        "a"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "Tensors can be created directly from data. The data type is automatically inferred."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[1, 2],\n",
            "        [3, 4]])\n"
          ]
        }
      ],
      "source": [
        "data = [[1, 2],[3, 4]]\n",
        "x_data = torch.tensor(data)\n",
        "print(x_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "creating a tensor from numpy and vice-versa"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[1, 2],\n",
            "        [3, 4]])\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "np_array = np.array(data)\n",
        "x_np = torch.from_numpy(np_array)\n",
        "print(x_np)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "we can also copy tensors and make similar ones with same dims:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Ones Tensor: \n",
            " tensor([[1, 1],\n",
            "        [1, 1]]) \n",
            "\n",
            "Random Tensor: \n",
            " tensor([[0.0158, 0.3978],\n",
            "        [0.9604, 0.9592]]) \n",
            "\n"
          ]
        }
      ],
      "source": [
        "x_ones = torch.ones_like(x_data) # retains the properties of x_data\n",
        "print(f\"Ones Tensor: \\n {x_ones} \\n\")\n",
        "\n",
        "x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data\n",
        "print(f\"Random Tensor: \\n {x_rand} \\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "we can create random tensors and also fill them with 0s, 1s"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Random Tensor: \n",
            " tensor([[0.0121, 0.6030, 0.4696],\n",
            "        [0.3624, 0.0542, 0.7875]]) \n",
            "\n",
            "Ones Tensor: \n",
            " tensor([[1., 1., 1.],\n",
            "        [1., 1., 1.]]) \n",
            "\n",
            "Zeros Tensor: \n",
            " tensor([[0., 0., 0.],\n",
            "        [0., 0., 0.]])\n"
          ]
        }
      ],
      "source": [
        "shape = (2,3,)\n",
        "rand_tensor = torch.rand(shape)\n",
        "ones_tensor = torch.ones(shape)\n",
        "zeros_tensor = torch.zeros(shape)\n",
        "\n",
        "print(f\"Random Tensor: \\n {rand_tensor} \\n\")\n",
        "print(f\"Ones Tensor: \\n {ones_tensor} \\n\")\n",
        "print(f\"Zeros Tensor: \\n {zeros_tensor}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can describe tensors shape, datatype, and the device on which they are stored"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of tensor: torch.Size([3, 4])\n",
            "Datatype of tensor: torch.float32\n",
            "Device tensor is stored on: cpu\n"
          ]
        }
      ],
      "source": [
        "tensor = torch.rand(3,4)\n",
        "\n",
        "print(f\"Shape of tensor: {tensor.shape}\")\n",
        "print(f\"Datatype of tensor: {tensor.dtype}\")\n",
        "print(f\"Device tensor is stored on: {tensor.device}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Operations on Tensors\n",
        "\n",
        "Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing,\n",
        "indexing, slicing), sampling and more are\n",
        "comprehensively described [here](https://pytorch.org/docs/stable/torch.html)_.\n",
        "\n",
        "Each of these operations can be run on the GPU (at typically higher speeds than on a\n",
        "CPU). If you’re using Colab, allocate a GPU by going to Runtime > Change runtime type > GPU.\n",
        "\n",
        "By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using\n",
        "``.to`` method (after checking for GPU availability). Keep in mind that copying large tensors\n",
        "across devices can be expensive in terms of time and memory!\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {},
      "outputs": [],
      "source": [
        "# We move our tensor to the GPU if available\n",
        "if torch.cuda.is_available():\n",
        "    tensor = tensor.to(\"cuda\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Try out some of the operations from the list.\n",
        "If you're familiar with the NumPy API, you'll find the Tensor API a breeze to use.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Standard numpy-like indexing and slicing:**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[0.0444, 0.6594, 0.8852, 0.4272],\n",
            "        [0.8485, 0.8111, 0.0237, 0.4294],\n",
            "        [0.1016, 0.6855, 0.5890, 0.4343],\n",
            "        [0.7649, 0.6575, 0.4558, 0.8507]])\n",
            "First row: tensor([0.0444, 0.6594, 0.8852, 0.4272])\n",
            "First column: tensor([0.0444, 0.8485, 0.1016, 0.7649])\n",
            "Last column: tensor([0.4272, 0.4294, 0.4343, 0.8507])\n",
            "tensor([[0.0444, 0.0000, 0.8852, 0.4272],\n",
            "        [0.8485, 0.0000, 0.0237, 0.4294],\n",
            "        [0.1016, 0.0000, 0.5890, 0.4343],\n",
            "        [0.7649, 0.0000, 0.4558, 0.8507]])\n"
          ]
        }
      ],
      "source": [
        "tensor = torch.rand(4, 4)\n",
        "print(tensor)\n",
        "print(f\"First row: {tensor[0]}\")\n",
        "print(f\"First column: {tensor[:, 0]}\")\n",
        "print(f\"Last column: {tensor[..., -1]}\")\n",
        "tensor[:,1] = 0\n",
        "print(tensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Joining tensors** You can use ``torch.cat`` to concatenate a sequence of tensors along a given dimension.\n",
        "See also [torch.stack](https://pytorch.org/docs/stable/generated/torch.stack.html)_,\n",
        "another tensor joining operator that is subtly different from ``torch.cat``.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[0.0444, 0.0000, 0.8852, 0.4272, 0.0444, 0.0000, 0.8852, 0.4272, 0.0444,\n",
            "         0.0000, 0.8852, 0.4272],\n",
            "        [0.8485, 0.0000, 0.0237, 0.4294, 0.8485, 0.0000, 0.0237, 0.4294, 0.8485,\n",
            "         0.0000, 0.0237, 0.4294],\n",
            "        [0.1016, 0.0000, 0.5890, 0.4343, 0.1016, 0.0000, 0.5890, 0.4343, 0.1016,\n",
            "         0.0000, 0.5890, 0.4343],\n",
            "        [0.7649, 0.0000, 0.4558, 0.8507, 0.7649, 0.0000, 0.4558, 0.8507, 0.7649,\n",
            "         0.0000, 0.4558, 0.8507]])\n"
          ]
        }
      ],
      "source": [
        "t1 = torch.cat([tensor, tensor, tensor], dim=1)\n",
        "print(t1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Arithmetic operations**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Sum: tensor([[0.0888, 0.0000, 1.7705, 0.8544],\n",
            "        [1.6970, 0.0000, 0.0475, 0.8587],\n",
            "        [0.2032, 0.0000, 1.1781, 0.8686],\n",
            "        [1.5298, 0.0000, 0.9115, 1.7014]])\n",
            "Diff and multiply constant : tensor([[0.0444, 0.0000, 0.8852, 0.4272],\n",
            "        [0.8485, 0.0000, 0.0237, 0.4294],\n",
            "        [0.1016, 0.0000, 0.5890, 0.4343],\n",
            "        [0.7649, 0.0000, 0.4558, 0.8507]])\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "tensor([[1.9727e-03, 0.0000e+00, 7.8363e-01, 1.8249e-01],\n",
              "        [7.1999e-01, 0.0000e+00, 5.6405e-04, 1.8435e-01],\n",
              "        [1.0318e-02, 0.0000e+00, 3.4696e-01, 1.8861e-01],\n",
              "        [5.8508e-01, 0.0000e+00, 2.0771e-01, 7.2372e-01]])"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y0 = tensor + tensor\n",
        "print(\"Sum:\", y0)\n",
        "\n",
        "y0 = 2*tensor - tensor\n",
        "print(\"Diff and multiply constant :\", y0)\n",
        "\n",
        "# This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value\n",
        "# ``tensor.T`` returns the transpose of a tensor\n",
        "y1 = tensor @ tensor.T\n",
        "y2 = tensor.matmul(tensor.T)\n",
        "\n",
        "y3 = torch.rand_like(y1)\n",
        "torch.matmul(tensor, tensor.T, out=y3)\n",
        "\n",
        "\n",
        "# This computes the element-wise product. z1, z2, z3 will have the same value\n",
        "z1 = tensor * tensor\n",
        "z2 = tensor.mul(tensor)\n",
        "\n",
        "z3 = torch.rand_like(tensor)\n",
        "torch.mul(tensor, tensor, out=z3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Single-element tensors** If you have a one-element tensor, for example by aggregating all\n",
        "values of a tensor into one value, you can convert it to a Python\n",
        "numerical value using ``item()``:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "5.854728698730469 <class 'float'>\n"
          ]
        }
      ],
      "source": [
        "agg = tensor.sum()\n",
        "agg_item = agg.item()\n",
        "print(agg_item, type(agg_item))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}