{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# unit 0.1 - What is machine learning?\n",
        "\n",
        "The course teaches machine learning with neural networks, which are the revolutionary tool powering many product and solutions (internet search, message filtering, social-network algorithms, ChatGPT, Gemini, DALL-E, etc.). Neural networks is just one of many machine learning algorithms, but it has proven to be one of the most effective to date and possibly the only one to learn if you have to chose one!\n",
        "\n",
        "Machine learning at the core is the set of scientific tools we use to power computer with comprehension of tasks, functions, applications based on a set of examples. Examples are a set of $(inputs, desired_outputs)$ pairs, which are used to train computer algorithms. These algorithms are what we call \"machine learning\".\n",
        "\n",
        "## Functions and machine-learning\n",
        "\n",
        "If all we want to is to learn a relationship between inputs and outputs we could just use equations! No need for anything fancier, or any machine-leanrning. After all functions can provide an output $y$ for every input $x$. \n",
        "\n",
        "### Have functions? ...\n",
        "\n",
        "Suppose you want to find an equation for a problem:\n",
        "\n",
        "- PROBLEM 1 - time for an object to fall from a table 1m tall\n",
        "\n",
        "\n",
        "![](images/mofall.jpg)\n",
        "\n",
        "\n",
        "Here there is an equation and we can use it!\n",
        "\n",
        "The interesting thing is - how was this equation discovered?\n",
        "\n",
        "### ... or do not have functions?\n",
        "\n",
        "Now suppose we want to find the equation for another problem:\n",
        "\n",
        "- PROBLEM 2 - is this a picture of a cat?\n",
        "\n",
        "![](images/cat.jpg)\n",
        "\n",
        "In this case, there is no equation...\n",
        "\n",
        "But why?\n",
        "\n",
        "## a fancier dataset\n",
        "\n",
        "CIFAR 10 dataset - I give you an image, you tell me what it is about (category). \n",
        "\n",
        "![](images/cifar10.png)\n",
        "\n",
        "This is another example of small images and the variety that they can have in the data / pixels. Is there a function for this?\n",
        "\n",
        "\n",
        "## Let us try it...\n",
        "\n",
        "TRY TO USE [Gemini](https://gemini.google.com/app):\n",
        "\n",
        "\"can you write code to train a lenet5 in PyTorch on the cifar10 dataset?\"\n",
        "\n",
        "(WAIT - what? why are we using this tool? Aren't we supposed to learn to program in this course?)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision.transforms as transforms\n",
        "import torchvision.datasets as datasets\n",
        "import torch.nn.functional as F"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 170498071/170498071 [00:12<00:00, 14158887.88it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Extracting ./data/cifar-10-python.tar.gz to ./data\n",
            "Files already downloaded and verified\n"
          ]
        },
        {
          "ename": "TypeError",
          "evalue": "cross_entropy_loss(): argument 'target' (position 2) must be Tensor, not int",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "Cell \u001b[0;32mIn[2], line 52\u001b[0m\n\u001b[1;32m     49\u001b[0m outputs \u001b[38;5;241m=\u001b[39m model(data)\n\u001b[1;32m     51\u001b[0m \u001b[38;5;66;03m# Compute the loss\u001b[39;00m\n\u001b[0;32m---> 52\u001b[0m loss \u001b[38;5;241m=\u001b[39m \u001b[43mcriterion\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlabels\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     54\u001b[0m \u001b[38;5;66;03m# Backward pass\u001b[39;00m\n\u001b[1;32m     55\u001b[0m optimizer\u001b[38;5;241m.\u001b[39mzero_grad()\n",
            "File \u001b[0;32m/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py:1518\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1516\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1517\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1518\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
            "File \u001b[0;32m/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py:1527\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1522\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1523\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1524\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1525\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1526\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1529\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1530\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
            "File \u001b[0;32m/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/loss.py:1179\u001b[0m, in \u001b[0;36mCrossEntropyLoss.forward\u001b[0;34m(self, input, target)\u001b[0m\n\u001b[1;32m   1178\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;28minput\u001b[39m: Tensor, target: Tensor) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Tensor:\n\u001b[0;32m-> 1179\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mF\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcross_entropy\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mweight\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1180\u001b[0m \u001b[43m                           \u001b[49m\u001b[43mignore_index\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mignore_index\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mreduction\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreduction\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1181\u001b[0m \u001b[43m                           \u001b[49m\u001b[43mlabel_smoothing\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlabel_smoothing\u001b[49m\u001b[43m)\u001b[49m\n",
            "File \u001b[0;32m/opt/homebrew/lib/python3.11/site-packages/torch/nn/functional.py:3053\u001b[0m, in \u001b[0;36mcross_entropy\u001b[0;34m(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)\u001b[0m\n\u001b[1;32m   3051\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m size_average \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mor\u001b[39;00m reduce \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m   3052\u001b[0m     reduction \u001b[38;5;241m=\u001b[39m _Reduction\u001b[38;5;241m.\u001b[39mlegacy_get_string(size_average, reduce)\n\u001b[0;32m-> 3053\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_C\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_nn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcross_entropy_loss\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_Reduction\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_enum\u001b[49m\u001b[43m(\u001b[49m\u001b[43mreduction\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mignore_index\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlabel_smoothing\u001b[49m\u001b[43m)\u001b[49m\n",
            "\u001b[0;31mTypeError\u001b[0m: cross_entropy_loss(): argument 'target' (position 2) must be Tensor, not int"
          ]
        }
      ],
      "source": [
        "class LeNet5(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(LeNet5, self).__init__()\n",
        "\n",
        "        # Convolutional layers\n",
        "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
        "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
        "\n",
        "        # Pooling layers\n",
        "        self.pool1 = nn.AvgPool2d(2)\n",
        "        self.pool2 = nn.AvgPool2d(2)\n",
        "\n",
        "        # Fully connected layers\n",
        "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
        "        self.fc2 = nn.Linear(120, 84)\n",
        "        self.fc3 = nn.Linear(84, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # Convolutional layers\n",
        "        x = F.relu(self.conv1(x))\n",
        "        x = self.pool1(x)\n",
        "        x = F.relu(self.conv2(x))\n",
        "        x = self.pool2(x)\n",
        "\n",
        "        # Fully connected layers\n",
        "        x = x.view(-1, 16 * 5 * 5)\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.relu(self.fc2(x))\n",
        "        x = self.fc3(x)\n",
        "\n",
        "        return x\n",
        "\n",
        "# Load the CIFAR-10 dataset\n",
        "train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transforms.ToTensor())\n",
        "test_dataset = datasets.CIFAR10(root='./data', train=False, download=True, transform=transforms.ToTensor())\n",
        "\n",
        "# Create the LeNet-5 model\n",
        "model = LeNet5()\n",
        "\n",
        "# Define the loss function and optimizer\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "# Train the model\n",
        "for epoch in range(10):\n",
        "    # Train the model on the training dataset\n",
        "    for i, (data, labels) in enumerate(train_dataset):\n",
        "        # Forward pass\n",
        "        outputs = model(data)\n",
        "\n",
        "        # Compute the loss\n",
        "        loss = criterion(outputs, labels)\n",
        "\n",
        "        # Backward pass\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        # Print the progress\n",
        "        if (i + 1) % 100 == 0:\n",
        "            print('Epoch [{}/{}], Batch [{}/{}], Loss: {:.6f}'.format(epoch + 1, 10, i + 1, len(train_dataset), loss.item()))\n",
        "\n",
        "# Evaluate the model on the test dataset\n",
        "correct = 0\n",
        "total = 0\n",
        "with torch.no_grad():\n",
        "    for data, labels in test_dataset:\n",
        "        outputs = model(data)\n",
        "        _, predicted = torch.max(outputs.data, 1)\n",
        "        total += labels.size(0)\n",
        "        correct += (predicted == labels).sum().item()\n",
        "\n",
        "print('Accuracy of the network on the test images: {} %'.format(100 * correct / total))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you run the code above you will see:\n",
        "\n",
        "It does not work and we do not know why, but it is quite close to being correct!\n",
        "\n",
        "In this course we will learn how to code this properly, or fix this properly!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}