{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# unit 1.6 - Datasets\n", "\n", "\n", "We will now look at how data is organized in professional datasets. We will look at samples of data, datasets. We will look into train, development and test dataset splits; also at batches of samples and epochs used during training.\n", "\n", "Creating a proper dataset is a complex and time-consuming endeavour. Careful management of data is important in neural network training, as important as the books one needs to read to learn a subject proper. \n", "\n", "## Samples\n", "\n", "Samples are \"examples\" used for training neural networks. Typically, in supervised learning, they are composed of two parts: an input data, and a label. Inputs are data that is fed as input to a neural network. Labels are the \"ground truth\" or \"desired outputs\". This is the output that our neural network should produce given the input.\n", "\n", "![](images/dataset1.png)\n", "\n", "## Datasets\n", "\n", "A dataset is a collection of samples. The samples make up a list and this list is the dataset. A dataset is used to train or test a neural network.\n", "\n", "As an example, in supervised learning for categorization, a dataset may contain a set of pictures for each category. Imaging we want a neural network that can tell animals apart. The dataset is a collection of samples = (input=picture of animal, label=type of animal). A dataset contains a list of these samples.\n", "\n", "When training a neural network, the dataset samples need to be in random order. This was proven to be a more effective way of training. If a network is trained by providing all examples of one category, then another, etc, it can be more prone to getting stuck into local minima and forgetting older categories.\n", "\n", "Randomization of a dataset is a must in neural network training. To avoid learning problems, always check that you code and routines apply randomization during training.\n", "\n", "## Train, development, test datasets\n", "\n", "So far we have talked about \"The Dataset\" as one giant list. In most cases, we will split the dataset into 2 or 3 portions. \n", "\n", "![](images/dataset2.png)\n", "\n", "If you want to train a neural network on your data, then you need to use some 80-90% of your data to train, but you may want to save 10-20% of your data to test your trained neural network. Testing means checking that the neural network can perform with high accuracy (# correct / # total number of samples) on your data. If you train on ALL your data, the neural network may memorize all your samples well and give an high training accuracy (a low training loss), but then you would not have any ways to test if this network can perform well on unseen examples. Therefore in most cases, we split a dataset into two parts: training and testing datasets. \n", "\n", "If you are employing others to train a neural network then you will do the same: give them 80-90% of your data to trains, but save 10-20% of data to check their results later. The company you employed will receive a single dataset, and they will have to again divide it into 2 parts: train and development datasets, often called: train, dev sets.\n", "They will train and test on their own, but then provide you with a trained neural network that you can test on your saved test dataset. The company has never seen that test set so they cannot use these test samples for training. This will make sure you can verify that your neural network is able to perform on unseen data.\n", "\n", "To summarize: first randomize your entire dataset, then always split the dataset into train, test. If you are employing other to do the job you can split into train, test and only give them the train dataset. They will then again split into train2 and dev datasets. Effectively this splits a dataset into 3 portions: train, dev, test. \n", "\n", "You may find datasets split in 2 or 3 parts. If the dataset has 2 portions (train, test), you will only train on the \"train dataset\" and test on the \"train dataset\". If the dataset has 3 portions, you will only train on the \"train dataset\" and test on the \"dev dataset\". You will leave \"test\" for your final submission, after you trained your final neural network model.\n", "\n", "Dev and Test dataset are useful to redesign your neural network architecture, to improve on your hyper-parameters (all parameters that are not neural network weights).\n", "\n", "## Batches and epochs\n", "\n", "When using gradient-descent as method of training neural networks, such as back-propagation, researchers found out that by averaging gradients over multiple samples one can obtain faster training times and sometimes more accuracy. \n", "\n", "If your computing hardware allows it, running on the ENTIRE dataset at once is the fastest option. This is called \"batch gradient descent\". The slowest option is to run on one sample at a time. This is called \"stochastic gradient descent\". \n", "\n", "Training of a neural network requires running through the entire dataset multiple times. Each time you use the entire dataset is called \"epoch\". Running 10 epochs means you will use your dataset for training a total of 10 times.\n", "\n", "Unfortunately training on the entire dataset may be prohibitive if the dataset is large and does not fit into your computer memory. In this case, which is the typical case, instead of training on one sample at a time, we can use \"batches\" of samples. Usually a power of 2, or as many as your training computer hardware can afford: 64, 128, 256 or even more in some cases. This is called \"mini-batch gradient descent.\"\n", "\n", "![](images/dataset3.png)\n", "\n", "When training, you will then take a batch of samples and back-propagate in chunks. In addition to improving training times, batching data also can maximize the efficiency of your computing hardware. This means that your hardware can be fully utilized and run faster. Therefore batches are not only used in training, but also testing.\n", "\n", "![](images/batching.png)\n", "\n", "Why is batching helfpul during training? Because averaging gradients makes the training updates align in the direction of the best local minimum of the loss function. Even averaging a few samples, as in \"mini-batch\" gradient descent, we are more efficient that the pure stochastic gradient descent that utilizes one sample at a time. \n", "\n", "A final note: When training a neural network, dataset samples should be re-arranged randomly at each epoch.\n", "\n", "## When to stop training?\n", "\n", "How many epochs should we use to train a neural network? A naive answers would be to run until the train loss is the lowest. Sometimes the loss function on the train dataset will continue to decrease at each epoch indefinitely. Because of that, one needs to check both the train loss and also the test loss after every epoch!\n", "\n", "![](images/stopping.png)\n", "\n", "A neural network can begin to memorize very effectively all the training samples, thus showing a lower and lower loss at each epoch. But by learning specific samples, the network will also lose generalization capabilities. In a typical case the test loss will have a minimum as a function of epochs. The epoch when that minimum occurs is the ideal point when we stop training. This point cannot be known a priory, and is empirical. This means you will need to run training for as many epochs as you need to be convinced that you passed the minimum test loss. \n", "\n", "Your final trained model can be the set of weights which provided the minimum test loss. As such it is a good idea to use a script that saves the weights of the network with the minimum test loss.\n" ] }, { "cell_type": "markdown", "id": "523d528a", "metadata": {}, "source": [ "## Encoding data for neural networks and categorization problems:\n", "\n", "1-hot encoding is a way to encode categorical values. It is used to convert categorical data into a numerical format. In this encoding, each category is represented as a binary vector. The vector has a length equal to the number of categories. The vector has a 1 in the position of the category and 0 in all other positions. \n", "\n", "For example, consider a 5-class problem: A, B, C, D, and E. The 1-hot encoding for each class is as follows:" ] }, { "cell_type": "code", "execution_count": null, "id": "7a19bf39", "metadata": {}, "outputs": [], "source": [ "# 1-hot encoding for the 5 classes\n", "\n", "A = [1, 0, 0, 0, 0]\n", "B = [0, 1, 0, 0, 0]\n", "C = [0, 0, 1, 0, 0]\n", "D = [0, 0, 0, 1, 0]\n", "E = [0, 0, 0, 0, 1]" ] }, { "cell_type": "markdown", "id": "8f2f4c50", "metadata": {}, "source": [ "In this way, our neural network will have 5 output neurons, one representing each class. The output of the network will be a 1-hot encoded vector. The class with the highest value in the output vector will be the predicted class.\n", "\n", "If you were to use this encoding for sequence learning of characters, then you would have a 1-hot encoding for each character in the alphabet. This would be a 26-dimensional vector, or more if other symbols are included.\n", "\n", "If you want to categorize pictures of 1000 types of animals, then you would have a 1-hot encoding for each animal. This would be a 1000-dimensional vector.\n", "\n", "## Loss functions for categorization problems:\n", "\n", "Multi-class classification problems in neural networks use the loss function: [cross entropy](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html), which is better suited for this type of problem. The cross-entropy loss is a measure of how well the predicted probability distribution matches the true probability distribution." ] }, { "cell_type": "code", "execution_count": null, "id": "b898601d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([ 2.5992, -1.2690, 0.6240, 0.0716, -1.4380, 0.9057, 3.9832, 0.2268,\n", " 6.3996, -0.4410])\n", "tensor([1.9884e-02, 4.1553e-04, 2.7586e-03, 1.5878e-03, 3.5089e-04, 3.6562e-03,\n", " 7.9356e-02, 1.8544e-03, 8.8919e-01, 9.5097e-04])\n" ] } ], "source": [ "import torch\n", "\n", "a = torch.randn(10).float()*3\n", "print(a)\n", "\n", "b = torch.nn.functional.softmax(a, dim=0)\n", "\n", "print(b)" ] }, { "cell_type": "markdown", "id": "5adf7f24", "metadata": {}, "source": [ "## another way of embedding input data in sequence learning with categories:\n", "\n", "If you have a problem with input data that is categorical, such as a sequence of characters, then you can use an embedding layer. An embedding layer is a layer that maps each category to a vector. This is useful when the input data is a sequence of categories.\n", "\n", "To represent the encoding of a multi-class input we can use an embedding layer, basically a table that maps each class to a vector. The embedding layer is trained along with the rest of the network. The embedding layer is initialized with random values and is updated during training. The embedding layer is a matrix of size (number of classes x embedding size). The embedding size is a hyperparameter that needs to be set before training. The embedding size is the size of the vector that represents each class." ] }, { "cell_type": "code", "execution_count": null, "id": "ca10b908", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[-1.5157, -0.4179, -0.5143, 0.2364]], grad_fn=)\n" ] } ], "source": [ "em = torch.nn.Embedding(10, 4) # 10 classes, 4 dimensions code\n", "# print the endding for for class 2\n", "idx = torch.LongTensor([2])\n", "\n", "# print as table:\n", "print(em.weight)" ] }, { "cell_type": "markdown", "id": "dea1a800", "metadata": {}, "source": [ "as you can see there are 10 lines and 4 numbers each, meaning that we encoded 10 classes with a tensor 4-numbers long." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }