{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# unit 1.4 - Back-propagation\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/culurciello/deep-learning-course-source/blob/main/source/lectures/14-backpropagation.ipynb)\n", "\n", "Back-propagation is an optimization algorithm commonly used in training artificial neural networks. It's a supervised learning algorithm that adjusts the weights of the network to minimize the error between predicted and actual outputs.\n", "\n", "Back-propagation is an algorithm that is based on the concept of gradient-descent.\n", "\n", "## Gradient descent\n", "\n", "\n", "\n", "When we train a neural network we want to minimize the error between the samples from the dataset and the output of the neural networks.\n", "\n", "Training a neural network means minimizing the loss function over the dataset.\n", "\n", "This is not much different from when you play basketball: you (the neural network) want to get the ball in the basket.\n", "\n", "![](images/gd1.png)\n", "\n", "The error function is how much you are off.\n", "\n", "![](images/gd2.png)\n", "\n", "The correction (change in the neural network weights) is how much you have to correct to get the right output (the ball in the basket).\n", "\n", "![](images/gd3.png)\n", "\n", "The you just try again...\n", "\n", "![](images/gd4.png)\n", "\n", "and again...\n", "\n", "![](images/gd5.png)\n", "\n", "until your error is $~0$ (the ball IS IN the basket!) \n", "\n", "This whole process is called \"Gradient Descent\" and it is the way to find a good set of weights for your neural network that minimizes the loss function.\n", "\n", "![](images/gradient-descent.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## gradient descent code example\n", "\n", "We can now see an example of how this is implemented in python.\n", "\n", "Here is a possible loss function f(x):\n", "\n", "$f(x) = x³-4x²+6$" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABJR0lEQVR4nO3dd3zTdeI/8NcnSZPudO89KC17T2UKKip6nufp6eE4J5wo3hDv9z2/971T/N759bxTTnHhRNRTHDgA2TLLKBRooYMOumfSpm3SJJ/fH2kLnAgUmryTT17PxyN/ENvmReBhXrynJMuyDCIiIiIBVKIDEBERkfdiESEiIiJhWESIiIhIGBYRIiIiEoZFhIiIiIRhESEiIiJhWESIiIhIGBYRIiIiEkYjOsD52O12VFdXIygoCJIkiY5DREREF0GWZbS1tSEuLg4q1fnHPNy6iFRXVyMxMVF0DCIiIroElZWVSEhIOO/XuHURCQoKAuD4jQQHBwtOQ0RERBfDaDQiMTGx73P8fNy6iPROxwQHB7OIEBEReZiLWVbBxapEREQkDIsIERERCcMiQkRERMKwiBAREZEwLCJEREQkDIsIERERCcMiQkRERMKwiBAREZEwLCJEREQkjFOLSFVVFe644w6Eh4fDz88Pw4YNw759+5z5kkRERORBnHbEe0tLC6ZMmYIZM2bgm2++QWRkJIqKihAaGuqslyQiIiIP47Qi8r//+79ITEzEypUr+55LTU111ssRERGRB3JaEfniiy8wd+5c3HLLLdi6dSvi4+Px8MMP47777vvR7zGbzTCbzX2/NhqNzopHTtBisuBkkwkVTR2oNXahw2JDp8UKSZIQqNMgUKdBXIgvUiICkBIeAF8ftejIRERea0dxI77Or8HUjAhcMyxWWA6nFZHS0lK8/PLLWLJkCZ588knk5ubikUcegVarxYIFC875PcuWLcOf/vQnZ0WiAWbo7MbGgjrsKmlCblkzypo6Lvp7VRKQHRuMcSlhmJgWjisHRcBf69aXQRMRKcru0ia8v6cCNrsstIhIsizLzvjBWq0WY8eOxc6dO/uee+SRR5Cbm4tdu3ad83vONSKSmJgIg8GA4OBgZ8SkfrLa7Fh/rA6f7D+FbUUN6Lad/dcnVu+LpDB/xIf4IUCngb9WDbsso91sg7GrG6eaO3Cy0QRjl/Ws7/P1UWHaoEjMHxmPq3Ki4aPmhi4iImd6+P39+Dq/Fv9vXjZ+dUXagP5so9EIvV5/UZ/fTvsnaGxsLHJycs56Ljs7G5988smPfo9Op4NOp3NWJLoM7WYrVu0px9s7y1HV2tn3fFZ0EGZmR2F8ShhGJ4dC7+dzwZ8lyzLqjGbkljUjt6wZm4/Xo7K5E+uO1mHd0TpEBunw83GJuHNSMqKCfJ352yIi8lrF9e0AgPSoQKE5nFZEpkyZguPHj5/13IkTJ5CcnOyslyQn6LbZ8cHeCvzjuyI0mSwAgPAALW4bn4QbRsZhUHRQv3+mJEmI0fvi+hFxuH5EHGRZxtFqI77Kr8HH+06hoc2MFzcV49VtpbhtfBIemp6O6GAWEiKigWK12XGy0QQAyIhUaBF57LHHMHnyZDzzzDP42c9+hr179+LVV1/Fq6++6qyXpAG2s6QR/2/NEZT2/GVNjQjAQ9PSccPIuAFdaCpJEobG6zE0Xo8lVw3C+qN1eOP7UhyoaMVbO8uwam8F7rsiFQ9Nz0CgjutIiIguV0VzB7ptMvx81IgP8ROaxWlrRABg7dq1WLp0KYqKipCamoolS5acd9fMf+rPHBMNHGNXN5Z9XYgP9lYAACICtVg8exB+Pi7RZWs3ZFnGjuIm/GPjCeSWtQAAIoN0WHrNYNw0Kh6SJLkkBxGREq0/Wov7392PIXHB+OqRKwb857vFGhEAuO6663Ddddc58yVogB2saMGiVQf71oHcMTEJv796MIJ8L7z2YyBJkoSpmRGYkhGO9cfq8MzXBShv6sCSjw7hs7xqPHPTUCSE+rs0ExGRUhQ3ONaHZAheHwLwrhnqIcsy3vz+JH62YheqWjuRHO6P1fdPxF9uHObyEnImSZIwd0gM1j92JX47NwtajQrbTjRg7t+3YfXeCjhxQI+ISLGK6xxF5FLW+Q00FhGC2WrDox/m4X/WHkO3Tca1w2Kw9tdTMTEtXHS0PjqNGgtnZOCbxVdgTHIoTBYbnvg0H4tWHYShs1t0PCIij3Kivg0AR0TIDbR2WHDnG3vxeV41NCoJT12fg+W3jxY6CnI+6ZGB+PiBSXjimsHQqCR8lV+Def/cjvxTBtHRiIg8gt0u923d5YgICVXV2ombX96JvSebEaTT4K27x+PuKaluvxBUpZLw4LR0fPzgJCSG+eFUSyd++spOfHGoWnQ0IiK3V9Xaia5uO7QaFRJDxe6YAVhEvFZlcwduXbELJQ0mxOp98fFDkzA1M0J0rH4ZlRSKrx65AjOyImG22vHIBwfxt3WFsNu5boSI6McU9UzLpEUEQOMGp1iLT0AuV9HUgZ+/uhunWjqRGhGATx+ejMExnrk9OtjXB68vGIcHp6UDAJZvLsH97+5Hp8UmOBkRkXsq6lmomukG0zIAi4jXqWrtxM9fdeyMSYsIwOr7JyJWL35o7nKoVRKeuGYwXrh1JHQaFb4rqMMdb+yBoYOLWImI/tOJ3iLiBgtVARYRr9JisuCXb+xBtaEL6ZGOEqKko9NvHBWPVfdNQLCvBvvLW/CzFbtQa+gSHYuIyK0U90zNsIiQS3VYrLj7rdy+NSHv3jsBUQoqIb3GJIfh4wcnIzpYh+N1bbj55Z0o7Tm4h4jI28myjKJ6Ts2Qi9nsMn696iDyKlsR4u+Dd+8djzjBdws4U1ZMEP794GSkRgT0TEXtZhkhIgJQbehCh8UGH7WE5HD3OJ2aRcQL/HVdITYW1kOnUeHNu8YhI8o9WrAzJYb54+MHJyErOgj1bWbc9truvpsmiYi81Yk6x7RMakSAy+4OuxD3SEFOs+bgKazYWgoAeO6WERidFCo4ketEBOrw/n0TMCg6EHVGM257dTfKWEaIyIsV9y1UdZ9/kLKIKNihylb8/pN8AMDCGem4fkSc4ESuFxGow6r7JiIzKhC1xi7c9tpuVDZ3iI5FRCREkRsd7d6LRUShDB3dePj9A7BY7ZidHYXHr8oSHUmY3jKSERWIGkMXFry5F03tZtGxiIhcrsiNjnbvxSKiQLIs47f/PoSq1k4khfnj+VtHQqVy72PbnS0ySIf37p2A+BA/lDaacPdbuWg3W0XHIiJyGVmWT0/NRHNEhJzorZ1lWH+sDlq1CstvH41gN73AztVi9L54597xCAvQ4vApAx58dz/MVp7ASkTeodbYhTazFWqVhJTwANFx+rCIKMyRKgOe+boAAPDktYMxLEEvOJF7SY8MxMq7xsFfq8b3xY34zceHeTcNEXmF3qPdU8L9odW4z8e/+yShy2a22vD4R4fQbZMxJycaCyaniI7klkYkhmDFnWPgo5bw5aFq/P27E6IjERE5Xd9BZm60YwZgEVGU5zecwPG6NkQEarHsJ8MgSd69LuR8rsiMxNM3DQMAvLipGGsOnhKciIjIuXqPdh/kRutDABYRxcgta8ar2xznhSz7yXCEB+oEJ3J/Pxub2Hdr7+//nY/csmbBiYiInKd3aibDjXbMACwiitBpseE3Hx+CLAM/HZOAq3KiRUfyGL+bm4Wrh8TAYrPjgXf3o6KJZ4wQkfLIstx3qqq7XHbXi0VEAf65qQjlTR2I1fvij9fniI7jUVQqCc/fOgLD4vVoNllw/7v70GHhtl4iUpaGNjOMXVaoJMfx7u6ERcTDFdQY+6Zk/mf+UG7VvQT+Wg1e++VYRARqUVjbhic+yYcscycNESlH70LV5PAA+PqoBac5G4uIB7PZZSz9NB82u4yrh8RwSuYyxOh9sfz20dCoJHxxqBpv7igTHYmIaMAU1bnf0e69WEQ82Pt7ypFX2YpAnQb/fcMQ0XE83oS0cPxhXjYA4JmvC7CrpElwIiKigXH6aHcWERogTe1mPLfuOADgd1dnIUbvKziRMtw1OQU3jYqHzS5j0aoDqDN2iY5ERHTZitzw1t1eLCIe6rn1J2DssiInNhi/mJAsOo5iSJKEZ24ahuzYYDSZLFi8+iBsPHmViDyYLMtueetuLxYRD3SkyoDVuRUAgP++YQjUXn6h3UDz06qx/PZR8Neqsbu0GS9uKhIdiYjokjW0mdHS0Q2VxCJCA0CWZfz3F0chy8ANI+IwPjVMdCRFSosMxNM3DQUA/HNjEdeLEJHHOt6zUDXFDXfMACwiHufLwzXYV94CPx81ll47WHQcRbtpVAJuGZMAuwwsXn0QTe1m0ZGIiPrteK2jiGTFuN/6EIBFxKNYrPa+BaoPTU9HrN5PcCLl+9P8IciICkR9mxlLPjrEm3qJyOP0FpFBbna0ey8WEQ+yOrcCFc0diAjU4VdXpIqO4xX8tRosv300dBoVtp5owOvfl4qORETUL71Huw/miAhdDpPZin9udCyaXDw7E/5ajeBE3iMrJghPXe84p+W5dSdQUGMUnIiI6OLY7TJO9GzdHcQiQpfj9e0n0dhuQUq4P34+LlF0HK9z2/hEzM6OhsVmx2Mf5sFstYmORER0QZUtHejstkGrUSE5zF90nHNiEfEAje1mvLqtBADw+Jws+Kj5x+ZqkiTh2ZuHITzAcR/N8+tPiI5ERHRBhbWnb9zVuOlnh3umorO8tKkYJosNw+L1mDcsVnQcrxURqMOzNw8HALy6vRS7S7mll4jc24neHTNuulAVYBFxe9WtnXh/TzkA4PdXD4aKh5cJdVVONG4dmwhZBh7/6BDaurpFRyIi+lGFde69dRdgEXF7r2wtQbdNxoTUMEzNjBAdhwD81/U5SAzzQ1VrJ/705THRcYiIflTviIi7LlQFWETcWp2xC6tzKwE4dsqQewjUafD8z0ZCkoB/7z+F747ViY5ERPQDZqsNJxtNANx36y7AIuLWVmwthcVqx9jkUExKCxcdh84wLiUM91+RBgD4w2f5MHRyioaI3EtpgwlWu4wgXw1igt33hnYWETdV39bVtzbkkVmZkCSuDXE3j101CKkRAagzmvHMVwWi4xARneXMg8zc+TOERcRNvb79JMxWO0YmhuAKrg1xS74+avz1p8MhScCH+yqxvahBdCQioj6Fbn60ey8WETfU1G7Gu7scoyGLORri1salhOGXE5MBAE98kg+T2So4ERGRwwk3v+yuF4uIG3pzx0l0djvODZmeFSk6Dl3A764ejPgQxy6av35bKDoOERGA0yMi7nyGCMAi4nZMZmvfaMjCGRkcDfEAAToN/rfnoLO3d5Vj78lmwYmIyNu1dXWjqrUTAEdEqJ8+3lcJY5cVqREBuConWnQcukhTMyNw61jHHUBPfHqYd9EQkVBF9Y6L7qKDdQjx1wpOc34sIm7EZpfxxo6TAIB7pqZCzVNUPcqT87IREahDaYMJK7aWio5DRF7suIcsVAVYRNzKuqO1qGzuRKi/D346OkF0HOonvZ8P/uu6bADAS5uLUdZzkBARkav1FhF3PsisF4uIm5BlGa9uc/wr+s5JKfDTqgUnoktxw4g4XJEZAYvVjv/6/AhkWRYdiYi8EEdEqN/2l7cgr7IVWo0Kv5yULDoOXSJJkvDn+UOh1aiwvagRXxyqFh2JiLzQ6cPMggUnuTAWETfROxpy8+h4RATqBKehy5ESEYBFMzIAAH9eW8Dj34nIpRrazGgyWSBJQEZUoOg4F8Qi4gbKGk3YUOC4OO3eqWmC09BAeGBaGtIiA9DYbsbf1vFsESJynd7RkOQwf4+Y5mcRcQPv7i6HLAPTsyI9or3Shek0ajx94zAAwPt7KnCwokVwIiLyFp5ytHsvFhHBOi02fLyvEgCwYFKK2DA0oCalh+Mno+Mhy8CTa47AarOLjkREXqCgxggAyI51//UhAIuIcF8cqoKxy4qkMH9MG8Tj3JXmD9dmI8TfBwU1Rry/p0J0HCLyAiwidNFkWcY7Pce53zExCSoeYKY44YE6PD4nCwDwf+uPo9lkEZyIiJSs22ZHUZ3jVNUcFhG6kAMVrThabYROo8ItYxJFxyEnuX18EnJig2HssuJv646LjkNECnay0QSLzY5AnQYJoX6i41wUFhGB3tvtGA25fkQcQgPc+y4AunRqlYQ/zR8CAFidW4H8UwbBiYhIqXqnZbJigjxmlJ1FRJDGdjO+OlwDADzAzAuMSwnD/JFxkGXgqS+OwG7niatENPCO9a0P8YwdMwCLiDAf5lbCYrNjRIIewxNCRMchF1h6TTb8tWocqGjFZ3lVouMQkQIV1Di27nrKQlXAhUXk2WefhSRJePTRR131km7LbpexqmcHxZ3csus1YvS++PXMTADAsm8K0dbFE1eJaGB52o4ZwEVFJDc3FytWrMDw4cNd8XJub0dJI6paOxHkq8F1w2NFxyEXumdqClIjAtDQZsaLm4pFxyEiBWlsN6OhzQxJ8oxbd3s5vYi0t7fjF7/4BV577TWEhoY6++U8woe5jgPMbhwZD18f9z9+lwaOTqPGH6/LAQC8+f1JFNe3C05ERErROxqSEh4Af61GcJqL5/QisnDhQsybNw+zZ8++4NeazWYYjcazHkrTYrJg/VHHvTK3juOWXW80Y3AUZg2OgtUu45mvC0THISKFKPDAhaqAk4vI6tWrceDAASxbtuyivn7ZsmXQ6/V9j8RE5X1Qf5ZXBYvNjpzYYAyN14uOQ4L8YV42NCoJmwrrsb2oQXQcIlKAvoWqMZ6zPgRwYhGprKzE4sWL8f7778PX1/eivmfp0qUwGAx9j8rKSmfFE0KW5b5pGY6GeLe0yEDc2bNt++mvCmDjdl4iukyeuFAVcGIR2b9/P+rr6zF69GhoNBpoNBps3boV//znP6HRaGCz2X7wPTqdDsHBwWc9lCS/yoDC2jZoNSrcODJedBwSbPGsTOj9fFBY29Z38SER0aWwWO0oaXCsORvMqRmHWbNmIT8/H3l5eX2PsWPH4he/+AXy8vKgVnvfIs3VPaMhVw+Jgd7fR3AaEi3EX4tHZjm28z63/gTazVbBiYjIUxXXt6PbJiPYV4P4EM842r2X05bVBgUFYejQoWc9FxAQgPDw8B887w06LTZ8mVcNgNMydNqdE5Px7q4ylDV14JUtJfjN3CzRkYjIA/VOywyODYYkecbR7r14sqqLfHOkBm1mKxLD/DApLVx0HHITWo0KS6/NBgC8tr0UVa2dghMRkSfqLSKecuPumVxaRLZs2YIXXnjBlS/pNtYcdBzp/dPRiR5zERG5xpycaExIDYPZasffvi0UHYeIPFBBrWdu3QU4IuISdcYu7ChuBADcNIqLVOlskiThv67LgSQBn+VVI6+yVXQkIvIgsix75B0zvVhEXOCLvGrYZWBMciiSwv1FxyE3NDRej5+MSgAA/GXtMcgyt/MS0cWpbzOj2WSBSgIGRXNEhM6hd1qGoyF0Pr+dmwVfHxX2lbdgw7E60XGIyEMc61kfkhYZ6JHXhrCIONnx2jYcqzHCRy1h3jBecEc/Lkbvi3unpgIA/rruOKw2u+BEROQJPPUgs14sIk72WZ5jNGR6VhRCA7SC05C7e2BaOkL9fVBc345/7z8lOg4ReYDCnvUhnnTj7plYRJzIbpfxOadlqB+CfX2waKbjkLO/f3cCnZYfnkBMRHQmT966C7CIONWek82oNnQhSKfBzMFRouOQh7hjYhLiQ/xQZzRj5c6TouMQkRvr6rahtNEEgFMzdA6f9YyGXDss1iMXEJEYOo0av5k7CADw8pYStJgsghMRkbsqqDHCZpcREahFdLBOdJxLwiLiJF3dNnydXwMAuGk0p2Wof+aPiEd2bDDauqxYvrlYdBwiclNHqx3TMkPi9B53tHsvFhEn2XqiAW1mK2L1vhifEiY6DnkYlUrCE9cMBgC8s6scp1o6BCciInd0tNoAABgS55nTMgCLiNN8ddgxGnLtsFge6U6X5MrMCExOD4fFZsfz60+IjkNEbuhIlWNEZGi8XnCSS8ci4gRd3TZ8V+A4kOq64Tw7hC6NJJ0eFVmTV4VjPUOwREQA0G2z43itY+suR0ToLFuO16PDYkN8iB9GJoaIjkMebHhCCK4bHgtZBv66jhfiEdFpRXXtsNjsCPLVICnMc68PYRFxgrU90zLzhsd67OIhch+/mZMFjUrCluMN2FXSJDoOEbmJM9eHePJnDYvIAOuwWLGxoB4AeKQ7DYiUiADcNj4JAPDc+uO8EI+IAJy9Y8aTsYgMsM2FDejstiExzA/DEzz7Lwe5j0UzM6DTqLC/vAVbjjeIjkNEbqB3RGRovOeuDwFYRAbcV/nVAIB5w+I8eqiM3Et0sC8WTE4B4BgVsds5KkLkzex2mSMi9EMmsxWbCh3TMtwtQwPtwWnpCNRpcLTaiG+P1oqOQ0QCnWwyocNig6+PCmkRAaLjXBYWkQG0qbAeXd12pIT7e/RWKnJPYQFa3DM1FQDw/IYTsHFUhMhr9Y6GZMcGQ6P27I9yz07vZr7ibhlysl9dkQq9nw+K69vxeV6V6DhEJMjRKs8/UbUXi8gA6bTYsOWEY1rmmqGcliHnCPb1wYPT0gEAL3xXhG6bXXAiIhKhd0RkqIevDwFYRAbM9qIGdHXbER/ip4iGSu5rweRkRATqUNHcgY/2VYqOQ0QuJssyjvSdIcIiQj3WHXUc6T5nSDSnZcip/LUaLJzhGBV5cWMxurptghMRkStVtXaitaMbGpWEQTGBouNcNhaRAWC12bGx0FFE5g6JEZyGvMHtE5IQp/dFrbEL7+0uFx2HiFyod1omMzoIOo1acJrLxyIyAPaebEZrRzfCArQYlxImOg55AZ1GjUdmZQIAXt5SApPZKjgREblK70LVoQpZBsAiMgDW9ZzpMDs7CmoVp2XINW4ek4CUcH80mSxYueOk6DhE5CJ9C1XjPX99CMAictlkWcb6Y5yWIdfzUavw2FWDAAArtpXC0NEtOBERucKRauVs3QVYRC7b4VMG1Bi64K9VY0pGhOg45GWuHx6HrOggtHVZ8dr2UtFxiMjJGtrMqDOaIUmOw8yUgEXkMq0/5piWmZ4VCV8fz180RJ5FpZL6RkXe2lmG1g6L4ERE5Ey9F92lRgQgQKcRnGZgsIhcpt5tu5yWIVHm5EQjOzYY7WYrXt/OtSJESqaUi+7OxCJyGUoa2lFc3w4ftYQZg6NExyEvpVJJeHS2YwfNyh0n0WLiqAiRUh2qbAUAjEhgESEA3/UsUp2YFo5gXx/BacibzcmJRk5sMEwWG17/nmtFiJQqv2fr7jCF7JgBWEQuy8ZCx90ys7OjBSchbydJp0dF3tpRhmaOihApTn1bF2oMXZAk5WzdBVhELllrhwX7y1sAADM5LUNu4KqcaAyJ6xkV4Q4aIsXJP+UYDcmIDFTMQlWAReSSbT3RAJtdxqDoQCSG+YuOQ9QzKuLYQfP2To6KECnNoZ4iMjwhRGyQAcYicok290zLzBzMaRlyH7OzozA03jEqwnNFiJQl/1QrAGC4ghaqAiwil8Rqs2PLiQYAwKxsTsuQ+5AkCY/OOj0q0tRuFpyIiAaCLMs43DciwiLi9Q5WtqK1oxsh/j4YlRgiOg7RWWZlR2FYvB4dFhte47kiRIpQbehCk8kCjUpSzImqvVhELsHGAse0zPRBkdCo+RaSezlzB807uzgqQqQEh3vOD8mKCVLcKd78FL0Emwod54fM5LZdclMzB0dheIJjVORVrhUh8niHFDotA7CI9FtlcwdO1LVDrZIwLTNSdByiczprVGRnORo5KkLk0fKrWgEob8cMwCLSb5t6dsuMTQ6F3p+nqZL7mpEVhREJenR22/DaNo6KEHkqu/30QlUlnajai0Wkn3pPU+VuGXJ3Z54r8s4ujooQeary5g60dVmh06iQFRMkOs6AYxHphw6LFbtLmwDwNFXyDNOzIjEiMQSd3Ta8ylERIo90uOf8kJy4YPgocIOE8n5HTrSntBkWqx3xIX5IjwwUHYfoghznijjWiry3u5ynrRJ5oL7zQxQ4LQOwiPTL1p5DzKZlRUKSJMFpiC7O9KzIvnNF3uDNvEQe53DfiaohQnM4C4tIP2zrLSKDuFuGPIckSVg0MwMA8PbOchg6ugUnIqKLZbPLOFJlBKDMrbsAi8hFq2jqQGmjCRqVhMnp4aLjEPXLVdnRGBwThHazFSt38rRVIk9RXN+Ozm4bArRqpCl0SQCLyEXaWuQYDRmdHIogX27bJc+iUp0eFXnz+5No6+KoCJEn6J2WGRKvh1qlzCUBLCIXidMy5OmuGRqLtMgAGLuseHd3ueg4RHQReheqjlDotAzAInJRLFY7dhY3AmARIc+lVklYNMMxKvL69pPosFgFJyKiCzlc1XOQmUIXqgIsIhdlf3kLTBYbIgK1yFHYrYfkXW4YEYfkcH80myxYtadCdBwiOg+L1Y6C6p6FqgrduguwiFyUbT3rQ67IjIRKoXN05B00ahUenp4OAFixrRRd3TbBiYjoxxyrMcJisyPU3wfJ4f6i4zgNi8hF2Hqc60NIOW4alYD4ED80tJnxYW6l6DhE9CMOVrQAAEYlhSr67CoWkQuob+vCsRrH0NjUzAjBaYgun1ajwoM9oyKvbC2B2cpRESJ3dLCiFQAwKjFEaA5nYxG5gO0nHItUh8XrERGoE5yGaGDcMiYBUUE61Bi68OmBKtFxiOgc8ipbAQAjk0KE5nA2FpEL2N6zPuTKQRwNIeXw9VHjgWmOUZF/bSlGt80uOBERnamx3YyK5g5IEjCCIyLeS5ZlfF/suG13agbXh5Cy3D4+CRGBWlQ2d+LzvGrRcYjoDHk90zIZkYEIVvghmiwi53Girh2N7Wb4+qgwOjlEdByiAeWnVeNXV6QBAP61uRg2uyw4ERH1OljZu1A1RGwQF2AROY8dPYeYjUsJg06jFpyGaODdMTEZIf4+KG00Ye1hjooQuYu+hapJoWKDuIBTi8iyZcswbtw4BAUFISoqCjfeeCOOHz/uzJccUL1FZGoG14eQMgXqNLh3SioAYPnmYtg5KkIknM0u9x3tzhGRy7R161YsXLgQu3fvxoYNG9Dd3Y05c+bAZDI582UHRLfNjt2ljvUhU1hESMF+OTkFQToNTtS1Y/2xWtFxiLxecX072s1WBGjVyIwKEh3H6TTO/OHffvvtWb9+6623EBUVhf379+PKK6905ktftsOnWmGy2BDq78Nj3UnR9H4+uGtKCl7cVIwXNxVj7pAYRR+eROTueg8yG54Qotgbd8/k0jUiBoNjqCksLOyc/91sNsNoNJ71EOX7IsdoyOT0CB7rTop3z5RUBGjVOFptxKbCetFxiLza6fUhIUJzuIrLiojdbsejjz6KKVOmYOjQoef8mmXLlkGv1/c9EhMTXRXvB3rXh3BahrxBaIAWd0xKBgC8uKkYssy1IkSinN4xo/yFqoALi8jChQtx5MgRrF69+ke/ZunSpTAYDH2Pykox92CYzNa+vwhTMsKFZCBytV9NTYNOo0JeZSt2lTSJjkPkldq6ulFU3w4AGKnwg8x6uaSILFq0CGvXrsXmzZuRkJDwo1+n0+kQHBx81kOEvWXN6LbJSAj1Q1KYcm88JDpTZJAOPx/nGIVcvqVYcBoi73T4lAGyDCSE+iEyyDuuFXFqEZFlGYsWLcKaNWuwadMmpKamOvPlBsyOotPbdrloj7zJ/dPSoVFJ2FHc1Ldgjohc58wbd72FU4vIwoUL8d5772HVqlUICgpCbW0tamtr0dnZ6cyXvWw7Srhtl7xTfIgfbhoVDwBYvrlEcBoi7+MtN+6eyalF5OWXX4bBYMD06dMRGxvb9/jwww+d+bKXpandjIIax26dyelcH0Le58Hp6ZAk4LuCOhTWitu5RuRtZFnGwZ4bd71lxwzggqmZcz3uuusuZ77sZdld2gwAGBwThPBA75ifIzpTemQgrh0WCwD4F0dFiFymsrkTzSYLtGoVcuK85/wq3jXzH3pPU52YxtEQ8l4PT08HAKw9XI2yRvc/CZlICXp3aw6JD/aq+81YRP4DiwgRMCROj5mDo2CXgVe2clSEyBX2lzuKiLds2+3FInKGxnZz3/7tCannPv2VyFssnOEYFfnkwClUt7r3AnMiJdhX5igiY5O96/OHReQMe85YHxIaoBWchkisMclhmJgWhm6bjFe3lYqOQ6Ro7WZr3+LwsSnes3UXYBE5C6dliM62cEYGAGB1bgUa282C0xAp18GKFth7DjKLDvYVHcelWETOwCJCdLapGREYkaBHV7cdb35/UnQcIsU6PS3jXaMhAItIn971IZLE9SFEvSRJwsM9oyLv7iqHobNbcCIiZepdqDomxfs+f1hEepxeHxLM9SFEZ7gqOxqDogPRZrbi3V1louMQKY7VZu872p0jIl7s9LSM97VRovNRqSQ8PN0xKvLmjjJ0WKyCExEpS2FtG0wWG4J0GgyKDhIdx+VYRHpwfQjRj7tueCySwvzRbLLgg72VouMQKUrvtMyo5FCoVd530SqLCLg+hOhCNGoVHpzmOFfktW2lMFttghMRKce+cu+dlgFYRACcvT4kxJ/rQ4jO5eYx8YgO1qHW2IVPD1SJjkOkGPvLHJ9BLCJejOtDiC5Mp1HjvivSADiOfbfa7IITEXm+6tZOVBu6oFZJGOlFN+6eiUUEp4vIhFSuDyE6n9snJCHU3wflTR34Kr9GdBwij9c7LZMTGwx/rUZwGjG8voi0mCx998uM5/oQovPy12pwz5RUAMC/NpfAbpcFJyLybL3TMmO8dFoGYBHpa6MZUYEI4/khRBf0y8kpCNRpcLyuDd8V1ImOQ+TR+haqetn9Mmfy+iKS29NGx3nxXwKi/tD7+eDOSckAgOVbSiDLHBUhuhTtZisKanouuvOyG3fP5PVFZO/J3iLivX8JiPrr3qmp8PVR4VBlK3YUN4mOQ+SRDpSfvuguRu9dF92dyauLSIfFiiNVBgAsIkT9ERGow8/HJQEAlm8uFpyGyDPtOcmNEoCXF5G8ilZY7TJi9b5ICPUTHYfIo9x/ZRp81BJ2lTb1nQxJRBevd0Te2w/S9Ooikttz7fK4lDBIkvcdq0t0OeJC/PCTUQkAOCpC1F9d3TYcqnSMyHv7jk0vLyJcqEp0OR6cng6VBGwqrMfRaoPoOEQeI6+yFRabHVFBOiSH+4uOI5TXFhGrzY4DPdcuj/PyNkp0qVIjAjBveBwA4F9bSgSnIfIcvdMy41M5Iu+1ReRotREdFhv0fj4YFOV91y4TDZSHpzsuw/s6vwalDe2C0xB5Bq4POc1ri0juGZcMqbzw2mWigZIdG4zZ2VGQZeBljooQXVC3zd63wHu8l++YAVhEOC1DNAAenpEBAFhzsApVrZ2C0xC5tyNVBnR22xDq74PMqEDRcYTzyiIiyzL29e2Y4UJVoss1OikUk9PDYbXLeHUrR0WIzufMgzQ5Iu+lRaSkwYQmkwU6jQrD4kNExyFShIU9oyKrcyvR0GYWnIbIfe05Y6EqeWkR6Z2WGZkYAq3GK98CogE3OT0cIxNDYLba8fr3paLjELklm13u+wzy9hNVe3nlp3B0sA6zBkdhxuAo0VGIFEOSJCzqGRV5b1c5WjssghMRuZ/CWiPauqwI1GmQHcsdmwCgER1AhJmDozFzcLToGESKMys7CtmxwSioMWLljjI8dtUg0ZGI3Erv+pAxyaHQqL1yLOAH+C4Q0YCRJAkLZzjOFXlrZxnazVbBiYjcS9/5IWlcH9KLRYSIBtQ1Q2ORFhkAQ2c33ttdLjoOkduw22XsLu29cZdFpBeLCBENKLVKwsPTHWtFXt9eiq5um+BERO6hsLYNLR3d8NeqMTwhRHQct8EiQkQDbv7IOCSE+qGx3YLVeytExyFyCztLGgE4tu36cH1IH74TRDTgfNQqPDDNsVZkxbZSWKx2wYmIxOudlpmczm27Z2IRISKnuGVMAqKCdKgxdOHTA6dExyESymqzY0+pY6HqpLQIwWncC4sIETmFr48a91+ZBgB4eWsJrDaOipD3OlJtRJvZimBfDXLigkXHcSssIkTkNLdPSEKovw/KmzrwVX6N6DhEwuwq6dktkxYONe+XOQuLCBE5jb9Wg3unpgIAlm8uht0uC05EJEbvQlWuD/khFhEicqo7J6UgSKfBibp2rD9WJzoOkctZrPa+G98nsYj8AIsIETmV3s8Hv5ycDMAxKiLLHBUh73LoVCs6u20ID9BiUBTvl/lPLCJE5HT3TEmFn48a+VUGbCtqFB2HyKV2FjvWh0xMD4eK60N+gEWEiJwuPFCH2yckAQBe2lQkOA2Ra+0qdZTvSWmcljkXFhEicon7r0yDVq1CblkL9vQc7ESkdF3dNhwobwXAhao/hkWEiFwiOtgXt4xNAAC8tLlYcBoi19hf3gKLzY7oYB1SIwJEx3FLLCJE5DIPTkuHWiVhe1EjDlW2io5D5HTfFzumZaakR0CSuD7kXFhEiMhlEsP8MX9kHACOipB32F7UAACYmslj3X8MiwgRudTD0zMgScCGY3UorDWKjkPkNM0mC45WO/6OT81gEfkxLCJE5FIZUYG4dmgsAOBfm0sEpyFynh3FjZBlYHBMEKKCfUXHcVssIkTkcg/PSAcArD1cjZONJsFpiJzj+54zczgacn4sIkTkckPi9Jg1OAp2GXh5C9eKkPLIssz1IReJRYSIhFg4MwMA8OmBKlS1dgpOQzSwShtNqDZ0QatWYUIqzw85HxYRIhJidFIoJqeHw2qXsWIr14qQsvROy4xNCYWfVi04jXtjESEiYRbNcIyKrM6tRH1bl+A0RAOH0zIXj0WEiISZlB6O0UkhsFjteGP7SdFxiAZEt82O3aXNAIArMyMFp3F/LCJEJIwkSVjUs1bk3d3laDFZBCciunx5la1oN1sRFqBFTmyw6Dhuj0WEiISakRWFnNhgdFhseHMHR0XI820/4ZiWmZweDpWKx7pfCIsIEQklSRIemeUYFXlrRxkMHd2CExFdnu0998twWubisIgQkXBzcmIwOCYIbWYr3uCoCHmw1g5L34WOXKh6cZxeRJYvX46UlBT4+vpiwoQJ2Lt3r7Nfkog8jEol4ZFZmQCAlTtOwtDJURHyTNuKGmGXgUHRgYgL8RMdxyM4tYh8+OGHWLJkCZ566ikcOHAAI0aMwNy5c1FfX+/MlyUiD3T1kBgMig5EW5cVb+0oEx2H6JJsOe74fJueFSU4iedwahF5/vnncd999+Huu+9GTk4OXnnlFfj7++PNN9905ssSkQdSqST8eqZjVOSN70th7OKoCHkWu13Gtp6FqtOzuD7kYjmtiFgsFuzfvx+zZ88+/WIqFWbPno1du3ad83vMZjOMRuNZDyLyHtcOi0VGVCCMXVa8s7NMdByifjlSbUBjuwUBWjXGJoeJjuMxnFZEGhsbYbPZEB0dfdbz0dHRqK2tPef3LFu2DHq9vu+RmJjorHhE5IbUKgm/7jlX5PXvT6LdbBWciOjibTl++jRVrYZ7QS6WW71TS5cuhcFg6HtUVlaKjkRELnbd8DikRQagtaMbb3NUhDzIZq4PuSROKyIRERFQq9Woq6s76/m6ujrExMSc83t0Oh2Cg4PPehCRdzlrVGR7KUwcFSEP0GyyIK9n2y7Xh/SP04qIVqvFmDFjsHHjxr7n7HY7Nm7ciEmTJjnrZYlIAa4fHofUiAC0dHTj3d3louMQXdD2ogbIMjA4Jgixem7b7Q+nTs0sWbIEr732Gt5++20UFBTgoYcegslkwt133+3MlyUiD6dRq/pu5n1tWyk6LBwVIffWuz6E0zL959Qicuutt+K5557DH//4R4wcORJ5eXn49ttvf7CAlYjoP80fGYfkcH80mSx4j6Mi5MbsdhlbuW33kjl9seqiRYtQXl4Os9mMPXv2YMKECc5+SSJSAI1ahYU9oyKvbitFp8UmOBHRuR2uMqDZZEGQToMxyaGi43gct9o1Q0R0pptGxSMxzA+N7Ra8v4ejIuSeNhc6dstMzYyAj5ofq/3Fd4yI3JbPGWtFVmwrRVc3R0XI/WwsdOwOnTGY60MuBYsIEbm1n4xOQEKoHxrazFi1p0J0HKKz1Bg6caTKCEkCZrKIXBIWESJyaz5nrBV5ZWsJR0XIrXxX4JiWGZ0UiohAneA0nolFhIjc3s2jExAf4of6NjM+2MtREXIf3x1zTMvMzuZu0EvFIkJEbk+rOT0qsnxzCXfQkFtoN1uxq6QJAHBVDqdlLhWLCBF5hFvGJvTsoDHj3d1louMQYfuJBlhsdqSE+yM9MlB0HI/FIkJEHsFHrcKvZ2YCAF7ZyjtoSLwNBaenZSRJEpzGc7GIEJHH+MmoeKRGBKDZZMFbvJmXBLLa7H3nh8zO4fqQy8EiQkQeQ6NWYfEsx6jIq9tKYezqFpyIvNWBila0dHRD7+eDsTxN9bKwiBCRR7l+RBwyogJh6OzGm9+fFB2HvNR3PdMyMwdHQcPTVC8L3z0i8ihqlYRHZztGRd7YfhKtHRbBicgbcdvuwGERISKPc+3QWAyOCUKb2YrXt3NUhFyruL4dpY0m+KglXDkoQnQcj8ciQkQeR6WS8NhVgwAAK3ecRLOJoyLkOt8eqQEATE6PQJCvj+A0no9FhIg80pycaAyND4bJYsOKrSWi45AX+eZILQDgmqExgpMoA4sIEXkkSZKwpGdU5O1dZahv6xKciLxBRVMHjlYboZKAq7htd0CwiBCRx5qRFYWRiSHo6rbj5S0cFSHn+/aoY1pmQmo4wnnJ3YBgESEijyVJEh6f4xgVeX9PBWoNHBUh5+qblhnGaZmBwiJCRB5takYExqeEwWK148VNRaLjkILVGDpxsKIVADB3CIvIQGERISKPduaoyIe5lShvMglOREq1rmc0ZExyKKKDfQWnUQ4WESLyeBPSwjFtUCSsdhnPbzghOg4pFHfLOAeLCBEpwm/nZgEAvjhUjWPVRsFpSGka2szYW9YMALiaRWRAsYgQkSIMjdfjuuGxkGXgufXHRcchhVl/rBayDAxP0CMh1F90HEVhESEixXh8ThbUKgmbCuuR2/OvV6KB8G3PtAxHQwYeiwgRKUZqRAB+NjYRAPDXbwshy7LgRKQEje1m7CxpAuC454gGFosIESnK4lmZ0GlUyC1rwZbjDaLjkAJ8nV8Dm13G8AQ9UiICRMdRHBYRIlKUGL0v7pqcAgD467rjsNs5KkKX58tD1QCAG0bECU6iTCwiRKQ4D05LR5BOg4IaI748XC06DnmwqtZO5Ja1QJKAecM5LeMMLCJEpDihAVo8MC0NAPD8hhPottkFJyJPtbZnNGRcShhi9X6C0ygTiwgRKdLdU1IREahFeVMHPsytFB2HPFTviBqnZZyHRYSIFClAp8GvZ2YCAP6xsQgdFqvgRORpShvacaTKCLVKwrXDOC3jLCwiRKRYt41PQlKYPxrazHht20nRccjDfNEzLTM1IwJhAVrBaZSLRYSIFEurUeF3VzuOfl+xrQT1bV2CE5GnkGWZu2VchEWEiBRt3rBYjEwMQYfFhhe+KxIdhzzEsRojShpM0GpUmDMkWnQcRWMRISJFkyQJf5iXDQD4MLcSxfVtghORJ/jsYBUAYGZWFIJ8fQSnUTYWESJSvHEpYZiTEw2bXcaz3xSKjkNuzmqzY81Bx7TMT0bHC06jfCwiROQVfn/NYKhVEr4rqMfu0ibRcciNbStqQGO7GWEBWkzPihIdR/FYRIjIK6RHBuL28UkAgGe+LuDR7/SjPtnvmJaZPzIOWg0/Jp2N7zAReY3FszMRqNPg8CkDj36nczJ0dGPDsToAwM2jEwSn8Q4sIkTkNSICdXiw5+j3v357HF3dNsGJyN18ebgaFpsdg2OCMCQuWHQcr8AiQkRe5d6paYgJ9kVVayfe2VUmOg65mU8OnALgGA2RJElwGu/AIkJEXsVPq8aSOYMAAC9tKkaLySI4EbmLkoZ2HKxohVolYf4oHmLmKiwiROR1bh6dgMExQTB2WfHCdydExyE38WnPaMiVmRGICvIVnMZ7sIgQkddRqyT88bocAMB7eypwoo6HnHk7m13Gpwccu2VuHsNFqq7EIkJEXmlyRgTmDnEccvbntccgy9zO6822FTWgxtAFvZ8PZmfzSHdXYhEhIq/1h2tzoFWrsL2oEd8V1IuOQwJ9sKcCgOMkVV8fteA03oVFhIi8VlK4P+69IhUA8PRXx2C2cjuvN6ozdmFjoaOI9h56R67DIkJEXm3hjAxEBulQ1tSBt3aUiY5DAny8rxI2u4xxKaHIjA4SHcfrsIgQkVcL1Gnwu7lZAIAXNxWjoc0sOBG5kt0u44O9lQCA2zgaIgSLCBF5vZtHJ2B4gh7tZiueW3dcdBxyoW1FDahq7YTezwfXDosVHccrsYgQkddTqSQ8db1jO+9H+ytxpMogOBG5ygd7uUhVNBYRIiIAY5LDMH9kHGQZ+OPnR3g7rxeoN3b17ZbitIw4LCJERD2WXpMNf60aBypa8e/9p0THISf7qGeR6tjkUAziIlVhWESIiHrE6H3x6OxMAMCz3xaitYP30ChVt82O93vODrl9AkdDRGIRISI6w91TUpEZFYhmkwV/48JVxVp/tA41hi5EBGoxbzgXqYrEIkJEdAYftQr/M38oAGDV3gocqmwVG4icYuWOkwCA2yckQ6fhIlWRWESIiP7DpPRw3NizcPW/Pj8CGxeuKkr+KQP2lbfARy3hDk7LCMciQkR0Dk/Oy0aQToPDpwx9WzxJGVbudIyGzBsWi6hgX8FpiEWEiOgcooJ8sWTOIADA39YdR1M7T1xVgoY2M9YeqgEA3DUlVXAaAlhEiIh+1J0Tk5ETGwxDZzee/aZQdBwaAKv2VMBis2NkYghGJoaIjkNwUhEpKyvDvffei9TUVPj5+SE9PR1PPfUULBZuhSMiz6FRq/DnG4cAAD7efwq7S5sEJ6LLYbHa8d6ecgDA3VNSxIahPk4pIoWFhbDb7VixYgWOHj2Kv//973jllVfw5JNPOuPliIicZkxyGG4bnwgAePLTfHR12wQnokv15aFqNLSZERWkwzVDuWXXXWic8UOvvvpqXH311X2/TktLw/Hjx/Hyyy/jueeec8ZLEhE5zRPXZOO7gnqUNpqwfHMxHp+TJToS9ZPdLuOVrSUAHGfFaDVcmeAuXPYnYTAYEBYWdt6vMZvNMBqNZz2IiETT+/ngTzc4pmhe3lKC47VtghNRf20qrEdRfTuCdBr8YiK37LoTlxSR4uJivPjii3jggQfO+3XLli2DXq/veyQmJroiHhHRBV0zNAazs6Nhtct44tPDPFvEg8iyjH9tKQYA/GJiMoJ9fQQnojP1q4g88cQTkCTpvI/CwrNXlldVVeHqq6/GLbfcgvvuu++8P3/p0qUwGAx9j8rKyv7/joiInECSJPz5xiEI1GlwsKIV7+0uFx2JLlJuWQsOVLRCq1HhHi5SdTuSLMsXXesbGhrQ1HT+VeNpaWnQarUAgOrqakyfPh0TJ07EW2+9BZWqfwMwRqMRer0eBoMBwcHB/fpeIiJneGdXGf74+VEEaNXYsGQa4kL8REeiC7jnrVxsKqzHbeOTsOwnw0TH8Qr9+fzu12LVyMhIREZGXtTXVlVVYcaMGRgzZgxWrlzZ7xJCROSO7piQjM8OVuFARSueXJOPlXeNgyRJomPRjyioMWJTYT1UEvDAlWmi49A5OKUdVFVVYfr06UhKSsJzzz2HhoYG1NbWora21hkvR0TkMiqVhP+9eTi0GhW2HG/Ax/tOiY5E57GiZ6fMNcNikRIRIDgNnYtTisiGDRtQXFyMjRs3IiEhAbGxsX0PIiJPlxkdhMevchz//ue1x1Dd2ik4EZ1LSUM7vjhUDQB4aFq64DT0Y5xSRO666y7IsnzOBxGREvzqijSMSgpBm9mK339ymP9/c0P/+K4IdhmYnR2NofF60XHoR3DhBhHRJVCrJDx3ywjoNCpsL2rE6lzu8nMnx2vb8OVhx2jIY1dlCk5D58MiQkR0idIjA/HbuY5TVv+y9hhOtXQITkS9/rHxBGTZcf7LkDiOhrgzFhEiostw95RUjEsJhcliw+/+fRh2HnQm3NFqA77Or4UkAY/OHiQ6Dl0AiwgR0WVQqyT87acj4Oejxs6SJry546ToSF7v7xuKAADXDY9DVkyQ4DR0ISwiRESXKSUiAP/vumwAwF+/PY6j1QbBibzX4VOt+K6gDioJWDyLa0M8AYsIEdEAuH18Eq7KiYbFZsfi1XnotNhER/I6sizj2W8c14zcODIeGVGBghPRxWARISIaAJLkOOgsKkiH4vp2PP31MdGRvM6W4w3YWdIErVqFx67i2hBPwSJCRDRAwgK0+L+fjQAAvLe7AhuO1QlO5D2sNjue+boAAHD3lBQkhvkLTkQXi0WEiGgAXZEZifuuSAUA/P6Tw6g3dglO5B0+3n8KRfXtCPH3wcMzMkTHoX5gESEiGmC/mZuFnNhgNJssWLw6DzZu6XUqk9mK5zecAAA8MjMTej8fwYmoP1hEiIgGmE6jxj9vGwV/rRq7Spvw954PSXKOFzcVo6HNjORwf9wxMVl0HOonFhEiIifIiArEszcPBwC8tLkYmwq5XsQZShra8cb3pQCAp67PgVbDjzVPwz8xIiInuWFEHH45yfEv9Mc+PMQj4AeYLMv47y+OotsmY9bgKMwcHC06El0CFhEiIif6w7xsjEgMgaGzGwvfPwCzleeLDJR1R2uxvagRWrUKf7w+R3QcukQsIkRETqTTqLH89lEI8ffBoVMG/GVtgehIitButuLPPe/lA9PSkBweIDgRXSoWESIiJ0sI9cffbx0JAHh3dzk+yq0UG0gBnlt3HFWtnUgI9cND09NFx6HLwCJCROQCM7Ki8Ohsx90nf/gsH/vKmgUn8lz7y1vw9q4yAMAzNw2Dv1YjNhBdFhYRIiIXeWRmJq4ZGoNum4wH39uPqtZO0ZE8jsVqxxOfHIYsAzePTsCVgyJFR6LLxCJCROQiKpWE//vZCGTHBqOx3YL739nHy/H66aXNxSiqb0d4gBb/b1626Dg0AFhEiIhcyF+rwWu/HIPwAC2OVhux5KM82Hny6kU5WNGC5ZuLAQB/mj8EoQFawYloILCIEBG5WEKoP16+Ywy0ahW+OVKLp7/mTpoLMZmteOxDx3H580fG4brhcaIj0QBhESEiEmB8ahj+dovj5NU3vj+J17eXCk7k3v7yVQHKmjoQq/fF/8wfKjoODSAWESIiQeaPjMcT1wwGADz9dQG+OlwjOJF7Wne0Fh/srQAA/N8tI3ipncKwiBARCfTAlWn45aRkyDLw2Ed52FXSJDqSWylvMuE3Hx8CANx/ZRomZ0QITkQDjUWEiEggSZLw1PVDMCcnGharHfe+nYsDFS2iY7mFrm4bHn7/ANq6rBiTHIrfzs0SHYmcgEWEiEgwtUrCP28bhSkZ4eiw2HDXm3txtNogOpZwf/ryKI5WGxEWoMVLt4+Cj5ofWUrEP1UiIjfg66PGa78ci7HJoTB2WXHnG3tRXN8mOpYw7+wqwwd7KyFJwD9+PhKxej/RkchJWESIiNyEv1aDN+8eh2HxejSbLLjttT04Ued9ZWTbiQb86ctjAIDfzR2MKzJ5eqqSsYgQEbmRYF8fvHPPeAyOCUJDmxm3rtiFI1XeM01TXN+Ghe8fgM0u4+bRCXhwWproSORkLCJERG4mNECL1fdPxIgEPVo6unHba7uxv1z5C1hrDJ1Y8GYu2sxWjEsJxTM/GQpJkkTHIidjESEickMh/lq896sJGJcSirYuK+58Yw++L2oUHctpmtrNuOP1Pahq7URqRABeuWMMdBq16FjkAiwiRERuKsjXB2/fMx5XZEY4dtOs3IuPcitFxxpwbV3duGtlLkoaTIjV++K9X01AeKBOdCxyERYRIiI35q/V4PUFYzF/ZBysdhm/++Qw/vptoWIuymvtsOCON/Yiv8qAsAAt3r13AuJDuEPGm7CIEBG5OZ1GjRduHYlHZmYAAP61pQSPrD6IDotVcLLL09huxm2v7cGhylaE+jsW6WZEBYqORS7GIkJE5AEkScKSOVn420+HQ6OSsPZwDW5cvgPF9e2io12SyuYO3LpiFwpqjIgI1GH1/ZMwNF4vOhYJwCJCRORBbhmbiPd+NQERgTqcqGvH/Je+xxeHqkXH6pf95c24cfmOvjUhHz0wEVkxQaJjkSAsIkREHmZiWji+XjwVE9PCYLLY8MgHB7H003y0m91/qubTA6dw22t70GSyYEhcMNY8PAVpkZyO8WYsIkREHigqyBfv3TsBC2ekAwA+2FuBq1/Y5ra393ZYrPjdvw9hyUeHYLHaMScnGh8/OAkxel/R0UgwSZZlt116bTQaodfrYTAYEBwcLDoOEZFb2lnciN/++zCqWjsBALeOTcTvrs5ymy2w+acMePTDgyhpMEElAb+emYnFszKhUvGwMqXqz+c3iwgRkQK0m614+qsCfLC3AgAQ7KvB43OycPuEJGG31prMVvzf+hN4a+dJ2GUgOliHF24dhUnp4ULykOuwiBAReal9Zc344+dHcazGCABIDPPDIzMzcdOoeGhcVEisNjs+PVCFv393AjWGLgDADSPi8NT1OW4zSkPOxSJCROTFbHYZq/aU4x8bi9HYbgbgKCQLJqXglrGJ0Pv5OOV1zVYbvs6vwYsbi1HaaOp73b/cOAzTBvEGXW/CIkJEROi02PDOrjK8srUELR3dAAB/rRrXDI3F9SNiMSUjYkCmbUoa2vHJ/lP4MLcSTSYLACAsQIuHpqXjzknJ8PXhnTHehkWEiIj6dFpsWHOwCm/tPIkTdacPQAvx98GktHBMTg/H2JQwpEUGXNRFcw1tZhyqbEVueTM2HKtDaYOp77/FBPvizknJWDA5BYE6jVN+P+T+WESIiOgHZFnGvvIWfHmoGl/n16Cx3XLWf1erJKSE+yNW74cQfx8E+/lAlh1rPtrNVlQbulDV0tk33dPLRy1hSkYEfj4uCbOzo1y2FoXcF4sIERGdl9Vmx6FTrdhV0oSdJU3IrzKgreviDkSTJCAzKhAjEkJw5aBITM+KRJCvc9adkGdiESEion6RZRl1RjOK6tvQ2G5Gi6kbhs5uqFUSNGoJ/j5qxOj9EBfii9SIABYPOq/+fH5zAo+IiCBJEmL0vjzplFyOE3lEREQkDIsIERERCcMiQkRERMKwiBAREZEwLCJEREQkDIsIERERCcMiQkRERMKwiBAREZEwLCJEREQkDIsIERERCcMiQkRERMKwiBAREZEwLCJEREQkjFvfvivLMgDHdcJERETkGXo/t3s/x8/HrYtIW1sbACAxMVFwEiIiIuqvtrY26PX6836NJF9MXRHEbrejuroaQUFBkCRpQH+20WhEYmIiKisrERwcPKA/m07j++wafJ9dg++za/B9dh1nvdeyLKOtrQ1xcXFQqc6/CsStR0RUKhUSEhKc+hrBwcH8i+4CfJ9dg++za/B9dg2+z67jjPf6QiMhvbhYlYiIiIRhESEiIiJhvLaI6HQ6PPXUU9DpdKKjKBrfZ9fg++wafJ9dg++z67jDe+3Wi1WJiIhI2bx2RISIiIjEYxEhIiIiYVhEiIiISBgWESIiIhKGRQTA008/jcmTJ8Pf3x8hISGi4yjG8uXLkZKSAl9fX0yYMAF79+4VHUlxtm3bhuuvvx5xcXGQJAmfffaZ6EiKtGzZMowbNw5BQUGIiorCjTfeiOPHj4uOpTgvv/wyhg8f3ne41qRJk/DNN9+IjqV4zz77LCRJwqOPPirk9VlEAFgsFtxyyy146KGHREdRjA8//BBLlizBU089hQMHDmDEiBGYO3cu6uvrRUdTFJPJhBEjRmD58uWioyja1q1bsXDhQuzevRsbNmxAd3c35syZA5PJJDqaoiQkJODZZ5/F/v37sW/fPsycORPz58/H0aNHRUdTrNzcXKxYsQLDhw8XF0KmPitXrpT1er3oGIowfvx4eeHChX2/ttlsclxcnLxs2TKBqZQNgLxmzRrRMbxCfX29DEDeunWr6CiKFxoaKr/++uuiYyhSW1ubnJmZKW/YsEGeNm2avHjxYiE5OCJCA85isWD//v2YPXt233MqlQqzZ8/Grl27BCYjGhgGgwEAEBYWJjiJctlsNqxevRomkwmTJk0SHUeRFi5ciHnz5p31/2oR3PrSO/JMjY2NsNlsiI6OPuv56OhoFBYWCkpFNDDsdjseffRRTJkyBUOHDhUdR3Hy8/MxadIkdHV1ITAwEGvWrEFOTo7oWIqzevVqHDhwALm5uaKjKHeNyBNPPAFJks774IciEfXXwoULceTIEaxevVp0FEXKyspCXl4e9uzZg4ceeggLFizAsWPHRMdSlMrKSixevBjvv/8+fH19RcdR7ojI448/jrvuuuu8X5OWluaaMF4mIiICarUadXV1Zz1fV1eHmJgYQamILt+iRYuwdu1abNu2DQkJCaLjKJJWq0VGRgYAYMyYMcjNzcU//vEPrFixQnAy5di/fz/q6+sxevTovudsNhu2bduGl156CWazGWq12mV5FFtEIiMjERkZKTqGV9JqtRgzZgw2btyIG2+8EYBjOHvjxo1YtGiR2HBEl0CWZfz617/GmjVrsGXLFqSmpoqO5DXsdjvMZrPoGIoya9Ys5Ofnn/Xc3XffjcGDB+P3v/+9S0sIoOAi0h8VFRVobm5GRUUFbDYb8vLyAAAZGRkIDAwUG85DLVmyBAsWLMDYsWMxfvx4vPDCCzCZTLj77rtFR1OU9vZ2FBcX9/365MmTyMvLQ1hYGJKSkgQmU5aFCxdi1apV+PzzzxEUFITa2loAgF6vh5+fn+B0yrF06VJcc801SEpKQltbG1atWoUtW7Zg3bp1oqMpSlBQ0A/WNwUEBCA8PFzMuiche3XczIIFC2QAP3hs3rxZdDSP9uKLL8pJSUmyVquVx48fL+/evVt0JMXZvHnzOf/uLliwQHQ0RTnXewxAXrlypehoinLPPffIycnJslarlSMjI+VZs2bJ69evFx3LK4jcvivJsiy7vv4QERERKXjXDBEREbk/FhEiIiIShkWEiIiIhGERISIiImFYRIiIiEgYFhEiIiIShkWEiIiIhGERISIiImFYRIiIiEgYFhEiIiIShkWEiIiIhGERISIiImH+P1iybSRtyby+AAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "f_x = lambda x: (x**3)-4*(x**2)+6\n", "\n", "x = np.linspace(-1,4,500) # plot the curve\n", "plt.plot(x, f_x(x))\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Gradient descent is the way we can find the minimum of a function like f(x).\n", "\n", "Here is some code implementing gradient descent and an helper function to visualize the algorithm." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot_gradient(x, y, x_vis, y_vis):\n", " plt.subplot(1,2,2)\n", " plt.scatter(x_vis, y_vis, c = \"b\")\n", " plt.plot(x, f_x(x), c = \"r\")\n", " plt.title(\"Gradient Descent\")\n", " plt.show()\n", " plt.subplot(1,2,1)\n", " plt.scatter(x_vis, y_vis, c = \"b\")\n", " plt.plot(x,f_x(x), c = \"r\")\n", " plt.xlim([2.0,3.0])\n", " plt.title(\"Zoomed in Figure\")\n", " plt.show()\n", " \n", "def gradient_desc(x_start, steps, learning_rate, delta=0.01):\n", " \n", " # These x and y value lists will be used later for visualization.\n", " x_grad = [x_start]\n", " y_grad = [f_x(x_start)]\n", " # Keep looping until number of iterations\n", " for i in range(steps):\n", " # compute the derivative:\n", " x_start_derivative = (f_x(x_start)-f_x(x_start-delta))/delta # derivative approximation\n", " \n", " # next point is - learning_rate * derivative:\n", " x_start -= (learning_rate * x_start_derivative) \n", " \n", " # update the lists for visualization:\n", " x_grad.append(x_start)\n", " y_grad.append(f_x(x_start))\n", " \n", " print (\"Local minimum occurs at: {:.2f}\".format(x_start)) \n", " print (\"Number of steps: \",len(x_grad)-1)\n", " plot_gradient(x, f_x(x) ,x_grad, y_grad)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now try this on our loss function f(x) and its derivative. \n", "\n", "We do this starting at value x = 0.5, using 100 steps, and with a learning-rate of 0.05" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Local minimum occurs at: 2.67\n", "Number of steps: 100\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAARMAAAGzCAYAAAAMtfNWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7i0lEQVR4nO3deXxU1f3/8ddkQhZCCFsCgYTFiEBAQCObbEGQRbRiwKVgRVSw/kKFonX5+q1oq1+ktgWqVsRapVWUAkErrSiLLCphR9kFZQ2QBANJQAgwOb8/jjeTkBAScu/cOzOf5+Mxj3uZ3Mx8GMg755x77rkupZRCCCFqKMTuAoQQgUHCRAhhCgkTIYQpJEyEEKaQMBFCmELCRAhhCgkTIYQpJEyEEKaQMBFCmELCJADdf//9tGzZssxzLpeL5557zpZ6RHCQMDHRvn37GD9+PNdccw21a9emdu3aJCcnk56ezjfffGN3eZabM2cO06dPr/LxLVu2xOVy4XK5CAkJoV69elx77bWMGzeOtWvXWleojY4cOcJzzz3Hli1b7C7FdKF2FxAoFi1axN13301oaCijRo2iU6dOhISEsGvXLjIyMnj99dfZt28fLVq0sKW+M2fOEBpq7T/3nDlz2LZtGxMnTqzy93Tu3JnHHnsMgMLCQnbu3Mm8efN48803+fWvf82f//xni6q1x5EjR3j++edp2bIlnTt3trscU0mYmOC7777jnnvuoUWLFixbtoz4+PgyX586dSp//etfCQmpvCF4+vRpoqKiLKkxIiLCktetqWbNmnHvvfeWeW7q1KmMHDmSadOm0bp1ax555BGbqhPVokSNjRs3TgEqMzOzyt8zevRoFRUVpfbu3auGDBmi6tSpo26//XallFKrVq1SI0aMUImJiSosLEwlJCSoiRMnqh9//LHc6yxcuFC1b99ehYeHq/bt26uMjAw1evRo1aJFizLHAWry5Mllnjt8+LAaM2aMiouLU2FhYSo5OVm99dZbZY75/PPPFaDmzp2rXnjhBdWsWTMVHh6ubrrpJrVnz56S4/r27auAMo+La7hYixYt1NChQyv8WmFhoWrQoIFq1qyZKi4uLnne4/GoadOmqeTkZBUeHq7i4uLUuHHjVF5eXpnvX79+vRo4cKBq2LChioiIUC1btlRjxowpc4zH41HTp09XHTp0UOHh4apRo0Zq0KBBav369WWO++c//6muv/56FRERoerXr6/uvvtudfDgwTLH9O3bV7Vv315t375dpaamqsjISNW0aVM1derUcp/lxY+333670s/JX0jLxASLFi3i6quvplu3btX6vgsXLjBo0CB69erFH//4R2rXrg3AvHnz+PHHH3nkkUdo2LAh69at45VXXuHw4cPMmzev5Ps/++wzhg8fTnJyMlOmTOGHH35gzJgxJCQkXPa9s7Oz6d69Oy6Xi/HjxxMbG8snn3zCgw8+SEFBQbmuyksvvURISAiPP/44+fn5/OEPf2DUqFElYxvPPPMM+fn5HD58mGnTpgFQp06dan0epdWpU4c77riDt956ix07dtC+fXsAHn74Yd555x3GjBnDo48+yr59+3j11VfZvHkzX375JbVq1SInJ4eBAwcSGxvLU089Rb169di/fz8ZGRll3uPBBx/knXfeYciQITz00ENcuHCB1atXk5mZyQ033ADAiy++yG9/+1vuuusuHnroIXJzc3nllVfo06cPmzdvpl69eiWvd+LECQYPHkxaWhp33XUX8+fP58knn+Taa69lyJAhtGvXjt/97nc8++yzjBs3jt69ewNw4403XvHn5Ch2p5m/y8/PV4AaNmxYua+dOHFC5ebmljxKtyxGjx6tAPXUU0+V+76KWiBTpkxRLpdLHThwoOS5zp07q/j4eHXy5MmS5z777LMKWwVc1DJ58MEHVXx8vDp+/HiZ4+655x4VExNTUoPx27Rdu3aqqKio5LgZM2YoQG3durXkuaFDh162NVJaZS0TpZSaNm2aAtRHH32klFJq9erVClDvvfdemeMWL15c5vmFCxcqoFwLo7Tly5crQD366KPlvma0hPbv36/cbrd68cUXy3x969atKjQ0tMzzRsvsH//4R8lzRUVFqkmTJmr48OElz61fvz6gWiOlydmcGiooKAAq/i2cmppKbGxsyeO1114rd0xF4wGRkZEl+6dPn+b48ePceOONKKXYvHkzAEePHmXLli2MHj2amJiYkuNvvvlmkpOTK61ZKcWCBQu47bbbUEpx/PjxksegQYPIz89n06ZNZb5nzJgxhIWFlfzZ+K36/fffV/peNWF8poWFhYBuscXExHDzzTeXqTklJYU6derw+eefA5S0FhYtWsT58+crfO0FCxbgcrmYPHlyua+5XC4AMjIyKC4u5q677irzfk2aNKF169Yl71e63tLjP2FhYXTt2tXSz8hJpJtTQ9HR0QCcOnWq3NfeeOMNCgsLyc7OLjfICBAaGlphl+TgwYM8++yz/Pvf/+bEiRNlvpafnw/AgQMHAGjdunW572/Tpk25MCgtNzeXkydPMmvWLGbNmlXhMTk5OWX+3Lx58zJ/rl+/PkC5+sxkfKbGZ7xnzx7y8/OJi4ur8Hij5r59+zJ8+HCef/55pk2bRmpqKsOGDWPkyJGEh4cDetC8adOmNGjQ4JLvv2fPHpRSFX7GALVq1Srz54SEhJIgMtSvXz8opgWAhEmNxcTEEB8fz7Zt28p9zRhD2b9/f4XfGx4eXu4Mj8fj4eabbyYvL48nn3yStm3bEhUVRVZWFvfffz/FxcU1rtl4jXvvvZfRo0dXeEzHjh3L/Nntdld4nLJw1U/jM7366qsBXXdcXBzvvfdehcfHxsYCumUxf/58MjMz+fjjj/n000954IEH+NOf/kRmZmaVx3KKi4txuVx88sknFf79L34dOz4jJ5EwMcHQoUP529/+xrp16+jatWuNXmvr1q18++23zJ49m/vuu6/k+SVLlpQ5zpivsmfPnnKvsXv37krfIzY2lujoaDweDwMGDKhRvaVd/Fu5Jk6dOsXChQtJTEykXbt2ACQlJbF06VJ69uxZpit4Kd27d6d79+68+OKLzJkzh1GjRvHBBx/w0EMPkZSUxKeffkpeXt4lWydJSUkopWjVqhXXXHONKX8vMz8jp5ExExM88cQT1K5dmwceeIDs7OxyX6/Obybjt1vp71FKMWPGjDLHxcfH07lzZ2bPnl3S9QEdOjt27LjsewwfPpwFCxZU2KLKzc2tcr2lRUVFlanlSp05c4Zf/OIX5OXl8cwzz5T8AN511114PB5+//vfl/ueCxcucPLkSUB3vS7+zI0JYkVFRQAMHz4cpRTPP/98udcyvjctLQ23283zzz9f7vWUUvzwww/V/rsZ84iMWgOJtExM0Lp1a+bMmcPPf/5z2rRpUzIDVinFvn37mDNnDiEhIVU6Zdu2bVuSkpJ4/PHHycrKom7duixYsKDCsYkpU6YwdOhQevXqxQMPPEBeXh6vvPIK7du3r3AMp7SXXnqJzz//nG7dujF27FiSk5PJy8tj06ZNLF26lLy8vGp/DikpKcydO5dJkybRpUsX6tSpw2233Vbp92RlZfHuu+8CujWyY8cO5s2bx7Fjx3jsscd4+OGHS47t27cvDz/8MFOmTGHLli0MHDiQWrVqsWfPHubNm8eMGTMYMWIEs2fP5q9//St33HEHSUlJFBYW8uabb1K3bl1uueUWAPr168cvfvEL/vKXv7Bnzx4GDx5McXExq1evpl+/fowfP56kpCReeOEFnn76afbv38+wYcOIjo5m3759LFy4kHHjxvH4449X6zNKSkqiXr16zJw5k+joaKKioujWrRutWrWq5qftQL4/gRS49u7dqx555BF19dVXq4iICBUZGanatm2rfvnLX6otW7aUOdaYtFaRHTt2qAEDBqg6deqoRo0aqbFjx6qvv/66wlOKCxYsUO3atVPh4eEqOTm5WpPWsrOzVXp6ukpMTFS1atVSTZo0Uf3791ezZs0qOcY4NTxv3rwy37tv375y9Zw6dUqNHDlS1atXr8qT1vhp4pbL5VJ169ZV7du3V2PHjlVr16695PfNmjVLpaSkqMjISBUdHa2uvfZa9cQTT6gjR44opZTatGmT+vnPf66aN29eMrHt1ltvVRs2bCjzOhcuXFAvv/yyatu2rQoLC1OxsbFqyJAhauPGjeU+4169eqmoqCgVFRWl2rZtq9LT09Xu3btLjjEmrV2son+Ljz76SCUnJ6vQ0NCAOk3sUipIRoeEEJaSMRMhhCkkTIQQppAwEUKYQsJECGEKCRMhhCkkTIQQpnD0pLXi4mKOHDlCdHR0QE9DFsLJlFIUFhbStGnTSlcLdHSYHDlyhMTERLvLEEIAhw4dqnQWt6PDxLj0/NChQ9StW9fmaoQITgUFBSQmJpb8PF6Ko8PE6NrUrVtXwkQIm11uqEEGYIUQppAwEUKYQsJECGEKCRMhhCkkTIQQppAwEUKYQsJECGEKCRMhhCkkTIQQppAwEUKYwvIwycrK4t5776Vhw4ZERkZy7bXXsmHDBqvfVgjhY5Zem3PixAl69uxJv379+OSTT4iNjWXPnj0l96kVQgQOS8Nk6tSpJCYm8vbbb5c8V9nNhoqKikruuAb6akW/d+YMLF8OmzZBQQHEx0OfPpCSArJGi7DLq6/C99/DyJFwww3mvKaVN+Vp166dmjhxohoxYoSKjY1VnTt3LnODp4tNnjy55KZMpR/5+flWlmmNc+eUevllpRo0UArKP5KTlfrwQ7urFMEqNVX/P3z//csemp+fX6WfQ0tvwhUREQHApEmTuPPOO1m/fj0TJkxg5syZjB49utzxFbVMEhMTyc/P968lCLKy4M47Yc0a/edmzWDAAGjQQP82WLoUTp/WXxs1CmbOhDp17KtXBJ+OHWHrVvj0Uxg4sNJDCwoKiImJufzPoTkxV7FatWqpHj16lHnuV7/6lerevXuVvr+qiego+/Yp1aqVTv2YGKXeekup8+fLHnPihFJPPqlUSIg+rksXpXJybChWBK2mTfX/vYtumVqRqv4cWno2Jz4+nuTk5DLPtWvXjoMHD1r5tvb54Qe4+WbYtw+SkvQ4yQMPQOhFQ1P16sFLL8GqVdCwIaxfD4MGQWGhLWWLIKOU/r8K+v+fSSwNk549e7J79+4yz3377be0aNHCyre1h8cDd90Fe/dCixawciVcdVXl39OzJ3zxBcTGwubNkJYG58/7pl4RvM6cAWM4wV/C5Ne//jWZmZn83//9H3v37mXOnDnMmjWL9PR0K9/WHtOn67M2UVHw8cd6nKQq2raF//5Xf9/SpfDMM5aWKURJq6RWLXPH6szqgl3Kxx9/rDp06KDCw8NV27ZtKz2bczG/GTPZsUOp8HDdB63G36+MBQu8Z3r++19z6xOitM2b9f+zxo2rdLgjzubUVJVHke2kFPTrp7s1gwfrVsaVzh/51a/0+f+4ONi5U5/9EcJsy5bps4vJybB9+2UPr+rPoVybU1Mff6yDJCIC3nijZhPRXn5Z/wPn5MATT5hXoxClWTD4ChImNXP+PPzmN3r/17+G5s1r9noRETBrlt5/6y0dUkKYLS9PbyVMHOSf/4Rvv9VnY556ypzX7NkTHn5Y70+YoM8SCWEmaZk4jMcDU6fq/SeeADPHdF54AWJi4Ouv4d13zXtdIcAbJiaPyUmYXKkPP9Stkvr1vS0JszRq5D1F/Mwz8OOP5r6+CG7SMnEQpfQMVoDx4+Ey92C9Ir/6lR6DycqCN980//VF8JIwcZD162HDBggP1z/0VoiI8LZOpk6Fs2eteR8RfGQA1kHeeENv77xTD75aZfRoSEyEo0fh73+37n1EcJExE4c4eRLef1/v//KX1r5XeLj3LNGUKd7rKYSoCenmOMS77+oLpdq3hxtvtP79HngAmjaFw4fhgw+sfz8R2IqL4cQJvS9hYrPZs/V23DjfLLsYEeEdl5kxQw/+CnGlTpzQgQISJrb69ls98Op2wz33+O59x46FyEi9TMHq1b57XxF4jC5O3boQFmbqS0uYVMecOXp78836YjxfadgQ7rtP70+f7rv3FYHn+HG9bdTI9JeWMKkqpeC99/T+qFG+f/9HH9XbDz/UK7kJcSUkTBxg40a9ilrt2jBsmO/fPzlZt4iUgr/9zffvLwKDESYmj5eAhEnVffih3t5yi30ryY8bp7dvvw0XLthTg/BvxpiJtExsZISJHa0Sw89+pv8THD0Kn3xiXx3Cf0k3x2Z79+oVqUJDdcvELmFhelYsSFdHXBkJE5t99JHepqbqq4Tt9OCDAKj//Iev5h/h/fdhxQpZ9kRUkYyZ2Mzo4tx+u61lANCuHcfb9sTl8fDxnbMZOVIvQduyJWRk2F2ccDxpmdgoJwe+/FLvOyBMMjLgN7seAmAMb6Nvx6xXKhgxQgJFXIYMwNpo8WJ9Ova66/QVvDbyePRKjvMZzo9Ecg17uIENgHeW/cSJ0uURlZCWiY0++0xvhwyxtw70TPrDh+EU0fybnwEwkjklX1cKDh2SGffiEjwe71omEiY+Vlys77IHesKYzY4e9e6/h56Few8fEILnkscJUeLECW8T1oJ7MkmYVGbrVsjO1rfu7NHD7mqIj/fuf8ogfqAB8RzjJpZf8jghShjjJTEx+tagJpMwqYzRxUlN1QsV2ax3b0hI0CsfnCeMf3EXAKPQ1wy5XHpYp3dvO6sUjmXheAlImFRuyRK9HTjQ3jp+4nbrJU1AB4fR1Ukjg0jOAPqiYrfbpgKFs0mY2OTMGVi1Su87YLzEkJYG8+dDs2bwFTeynxbUpZB7G37C/Pn660JUyMIJayBhcmlffKHXXE1IgLZt7a6mjLQ02L8fln8ewtlbhgMwc+ACCRJROWmZ2MS4z+9NN/lmecZqcrv1UE7bZ3SYhPxnkSw4LSpn4YQ1kDC5NKOL07evvXVcTvfuesHpggJYtszuaoSTScvEBmfPwtq1er9PH3truZyQELjjDr2/YIG9tQhnkzCxwbp1cO4cNGkCSUl2V3N5w3VXh48+kkWTxKXJAKwNjC5Onz6OHC8pp3dv/dvmhx+8Yz1CXEzGTGxQOkz8QWio94pmuWxYXIp0c3zswgX46iu97y9hAt6uzsKF3pssCWG4cMF7Jz8JEx/ZvBlOn9YrqrVvb3c1Vde/v76x0tGjsH693dUIp7H4Ij+QMCnviy/0tmdPfabEX4SFweDBen/RIntrEc5jdHHq1dPdYgv40U+Lj2Rm6q0DrhKutltv1duPP7a3DuE8Fg++goRJecb8km7d7K3jSgwZggoJga+/5qNXD8lC08LL4sFXkDApKzsbDhzQp4O7dLG7mmrLWNWI9aG6RfXJr/4jC00LLwkTHzNaJcnJejDTj2Rk/LSg9Dnd1bkN3dWRhaYFYPmENZAwKcsYL/GzLo6x0LRSsAgdJv1ZRm1Oy0LTQpOWiY/56XiJsdA0wHbas58WRFBUspyjLDQtZADWlzwe7/yM7t3traWayi4g7eJjbgO8XZ2KjxNBRVomPrRzJxQW6sWj/WmyGuUXkDa6OreyCOMmXRUdJ4KIjJn4kNHFueEGv1tEtfRC0wAr6cuPRNKUo1zLVlloWkjLxKc2btRbPzwlfPFC00VEsIJUAAbzKSALTQe93Fy9jY217C0kTAybN+ttSoq9dVyh0gtNg76vDsBt4Z/JQtPBrqgI8vP1voSJxS5cgK+/1vvXX29vLTVgLDT9+edw88s6THqxmrTBP9pbmLCX0cVxu/UFrBaRMAHYvVvf2qJOHbj6arurqRFjoelbH2sDzZvjKiqSBZOCndHFadTI0otXJUwANm3S286d/etK4cq4XDBIt0749FN7axH28sF4CUiYaEaY+HEXp0ISJgIgJ0dv4+IsfRsJEwjcMLnpJt3S2rULDh60uxphF2mZ+EhxsfdMTqCFSf363ksDjJuwi+AjYeIj33+vZ75GREC7dnZXYz7p6gjp5viI0cXp2NGy5exsZYTJ0qVyT51gJS0THzG6ONddZ28dVunSRXd3Tp6UhaaDlYSJj2zdqrcdO9pbh1Xcbj0QC7B8ub21CHsYYSLdHIsZYXLttfbWYSUJk+BmjJkESsvkpZdewuVyMXHiRF+95eWdPOk9Zdqhg62lWMoIky+/1DdlF8Hj3DmfXJcDPgqT9evX88Ybb9DRaV2Jbdv0NiHB0msWbNemDTRtqi/4WrPG7mqEL/nouhzwQZicOnWKUaNG8eabb1L/Mn+ZoqIiCgoKyjwsFQxdHNBT66WrE5yMLo7F1+WAD8IkPT2doUOHMmDAgMseO2XKFGJiYkoeiYmJ1hYXLGEC3jBZtszeOoRv+ehMDlgcJh988AGbNm1iypQpVTr+6aefJj8/v+Rx6NAhK8sLzjBZt05P0hPBwYdhYtksrUOHDjFhwgSWLFlCRERElb4nPDyc8PBwq0oqS6ngCpMWLeCqq/SM39Wr4ZZb7K5I+IKPZr+ChS2TjRs3kpOTw/XXX09oaCihoaGsXLmSv/zlL4SGhuKx+yYuhw/rUW63G9q2tbcWX5Fxk+ATCC2T/v37s9X4zf+TMWPG0LZtW5588kncdi9IatTWpg34qjVkt/794W9/kzAJJoEQJtHR0XS4aO5GVFQUDRs2LPe8LYKpi2Po109vt2zRN2Wy8LYHwiF8NPsVgnkGbDCGSePG+p5ASsGKFXZXI3zBR7NfwcKWSUVWOOk/cDCGCehxk+3byfrncladG058vL6fjt29TmGRQDk17FgXLujVxyDowmRNpB6ELfxoGSNH6p5Py5aQkWFvXcIi0s2x2L59+pqFyEh9yjRIZGTA0D/0pRgXbdlNY44BkJUFI0ZIoAScc+f09WcgLRPL7Nypt23aBM5q9Jfh8cCECXCC+nyDvkaqN6sBPYQCMHGiPk4ECB9elwPBGiZGFycQl2m8hNWr9dQa0PciBuiL9346SsGhQ/o4ESCMLk7Dhj75pRmcYWK0TIJlshpw9Kh3fxV9AOjDqkqPE37Oh+MlEKxhEoQtk/h47/5qegPQka3UJ++Sxwk/58PTwhCMYaJUULZMevfWy7a4XJBLHDvQQWqMm7hckJiojxMBwoenhSEYwyQ7W1+TExICrVvbXY3PuN0wY4bed7nKdnVcLv389Oky3ySgSDfHYkarpFUrfa+cIJKWBvPnQ7NmZcMkIUE/n5Zmc4HCXD7u5gTgjWIuIwjHS0pLS4Pbb4d1GX3gLrghZBP7vi7AXb+u3aUJs0k3x2JBOF5yMbcbetyZAFddhau4GPfar+wuSVjBh2uZQDCHSZC2TMroo7s6rCp/ilgEgOxsvW3c2CdvF3xhYnRzgrhlUqKvnrzGypWVHyf8k4SJhQoLvdNAJUy8LZP16+HHH+2tRZjr9Gk4dUrvS5hYYPduvY2LgwYN7K3FCVq10qd2zp+HtWvtrkaYyWiVRERAdLRP3jK4wkTGS8pyuaSrE6hKd3GMiUQWC64w+fZbvW3Txt46nEQGYQOTESZNmvjsLYMrTPbs0dsgmvl6WUaYrFmjbx8qAoOPB18h2MJk7169vfpqe+twkrZt9a0jz56FTZvsrkaYRcLEQkpJy6QiLhf07Kn3v/zS3lqEeSRMLJSbCwUF+ocnKcnuapxFwiTwSJhYyGiVJCYG3QV+l1U6TIw1HIV/kzCxkDFeIl2c8lJS9F0Nc3O9oSv8m4SJhYwfEhl8LS88HLp00fvS1QkMEiYWksHXysm4SeA4e1aPD4KEiSWkm1M5I0y++MLeOkTNGa2SsDCIifHZ2wZHmMhp4cu78Ua93b3be78V4Z9smEoPwRImOTn6imGXS1/cJspr2NB7zdJXsliSX7NhvASCJUyMLk7z5nJauDIybhIYJEwsJF2cqunVS29l3MS/HdP3kJYwsYKESdUYLZMNG/QZAeGfpGViIZljUjVJSXrhqHPnYONGu6sRV0rCxEJyWrhqSl/0J10d/yVhYhGl4Lvv9L60TC7PGDeRQVj/JWFikbw872xAOS18eUbL5Kuv5KI/fyVhYpHvv9fbpk3ltHBVXHed/px++MG7ALfwH0VFcPKk3vfhko0QDGGyb5/eSqukasLCoFs3vS/jJv7HuItfrVpQv75P3zrww8RomVx1lb11+JPSXR3hX4wuTlycT6fSQzCEibRMqq9HD71ds8beOkT12TReAsEQJtIyqb7u3fV21y4WvJnHihXg8dhakagqCRMLScuk2jJWNeL7UD0n52/j1tKvH7RsCRkZ9tYlqkDCxCIeDxw4oPelZVIlGRkwYgSsvqC7Oj3QXZ2sLP28BIrDSZhY5PBhuHBBn6Fo2tTuahzP44EJE/T0kjWUDRNjysnEidLlcbSjR/XWx6eFIdDDxBgvadkSQgL7r2qG1at1/oI3TLqxlhB0eigFhw7p44RDGWESH+/ztw7snzAZL6kW4/8hwDY6UEgd6lJIMjsueZxwGAkTi8iZnGop/f+vGDfr6Ap4uzoVHSccRsLEItIyqZbevSEhwTvX6eJxE5dL38Osd2+7KhSVKiyE06f1voSJyaRlUi1uN8yYofddrrJhYgTM9On6OOFARqukTh398LHADhNpmVRbWhrMnw/NmkEmevJaW3bTPj6P+fP114VD2djFgUAOk9OnvefcpWVSLWlpsH8/LPi8IQXx1wCwZWamBInTSZhYZP9+va1XTz9EtbjdkJoKdQfqro57nVyn43hGmNg0pypww0TGS8whF/35D2mZWMRombRsaWcV/s8Ik7VrZeqr00mYWOTgQb1t0cLeOvxd+/YQHQ2nTsH27XZXIyojYWIR4wK/5s3trcPfud3QVU9ek66Ow0mYWMQIE2mZ1JyMm/gHCROLSJiYR8LE+c6cgRMn9L6EiYnOnvXOMZEwqTlj5bVvv9Wr1gvnMe4vHB5u21SIwAyTQ4f0NioKGjSwt5ZA0KABtGmj9zMz7a1FVKx0F8fHC0kbLA2TKVOm0KVLF6Kjo4mLi2PYsGHs9sW9WEoPvtr0wQYc6eo4m83jJWBxmKxcuZL09HQyMzNZsmQJ58+fZ+DAgZw2rmy0ioyXmE/CxNkcECahVr744sWLy/z5nXfeIS4ujo0bN9KnTx/r3ljCxHxGmKxbpyevyaXDzhLoYXKx/Px8ABpcYhyjqKiIoqKikj8XGPcIri4JE/MlJ+vJa4WFsG0bdOpkd0WiNAeEic8GYIuLi5k4cSI9e/akQ4cOFR4zZcoUYmJiSh6JiYlX9mbG7FeZsGYet9t721Dp6jiPzRf5gQ/DJD09nW3btvHBBx9c8pinn36a/Pz8ksch46xMdUnLxBrGKeK1a+2tQ5TngJaJT7o548ePZ9GiRaxatYqEhIRLHhceHk54eHjN3szj8Z4aljAxlwzCOpcDwsTSlolSivHjx7Nw4UKWL19OK1+seHb0qL5XTmio3CvHbEY3Z/duyMuztxbhdeEC5Obq/UANk/T0dN59913mzJlDdHQ0x44d49ixY5w5c8a6NzW6OAkJcsbBbA0bwjV65TXp6jhIdra+qVFoKDRqZFsZlobJ66+/Tn5+PqmpqcTHx5c85s6da92byuCrtYxxE5kJ6xxGF6dxY1tvNmfpmIky7inpSzL4aq0ePeAf/5BxEydxwHgJBOK1ORIm1ip9Rqe42N5ahCZhYhEJE2t16KAvoCwogF277K5GABw5orcSJiaTMRNrhYZCly56X7o6zpCVpbeVTLvwhcALk8OH9fZKZ8+Ky5NBWGcxwqRZM1vLCKwwKSyEn67/sTulA5pMXnMW4xeohImJjISOidEXpQlrGJPXduzwhrewj7RMLGBMo5dWibUaN9b3b1YK1q+3u5rgduaMdzayhImJjOaehIn1pKvjDMaZnMhI22+DK2EirowMwjpD6TM5Ni9RKmEirozRMsnM1N0dYQ+HjJeAhIm4Uh07QkSE7q/v2WN3NcHLIWdyIFDDROaYWC8sDFJS9L50dewjLROLyNkc35JBWPtJmFjg9Gnv7RElTHxDBmHtJ2FiAeNDrVMH6ta1t5ZgYbRMvvkGTp2yt5Zg5ZDrciCQwqT04Kvcxc83mjbV41PFxbBhg93VBJ/iYu88E2mZmEjO5NhDujr2ycnR67+GhECTJnZXE4BhImdyfOunrs7xj9fw/vuwYoW+QYDwAaOL07ixXhrCZoEXJtIy8akVZ3XLxPNVJiNHKvr1g5YtISPD3rqCgoMGXyGQwkROC/tcRgYM/p/rKSKMxuTQin2A/j8+YoQEiuUkTCwiLROf8nhgwgQoIpzNXAdAd/S4iTG7fuJE6fJYymH/5yVMxBVZvdr7kWeiuzo98E5eU0o3FlevtqO6ICEtEwucPQvHj+t9CROfMBZEB1iDHoQ1WiaXOk6YTMLEAsaHWrs21K9vby1BovRC6EbLpDNbiODMJY8TJpMwsYBMWPO53r29H/dBmnOEeGpxgRQ2Avr5xER9nLCIhIkF5EyOz7ndMGOG3ne5XCWtk+5kluT59Olyu2fL5OfrexeBY+ZWBUaYyOCrLdLSYP58/Yux9CBsQoJ+Pi3N5gIDmfELtEEDfVM0B5AwETWSlgb798PPZ+hB2FsbrmHf90qCxGpGmDikVQISJsIEbjdc91AKhIYS/sNR3EcP211S4HPgnSsDI0xSUmDAAGjXzu5Kglft2tCpk96XxZKsJy0Ti/z2t7BkCaSm2l1JcJMriH3HaJlImIiAZISJtEysZ7RMpJsjApKx8tqmTVBUZG8tgU66OSKgXXUVNGoE587Bli12VxO4ioulZSICnMslK9b7Qm6uDmyXSy+d6RASJsJcMghrPaNVEh8PtWrZW0spEibCXDIIaz0HzjEBCRNhti5d9ALHBw96V04X5nLg4CtImAizRUdDhw56f+1ae2sJVNIyEUFDBmGtJS0TETRkENZaEiYiaBhhsmEDnD9vby2BSLo5Imhcc41ePvPMGX0fYmGe8+e9C+tKy0QEvJAQ6NZN78u4ibmOHNFL/4eHQ2ys3dWUIWEirGEMwsq4ibmMLk5Cgg5tB3FWNSJwyCCsNRw6+AoSJsIqXbvq7XffQU6OvbUEkgMH9NZhg68gYSKsUq8eJCfrfZm8Zp79+/W2VStby6iIhImwjlynYz4jTFq2tLOKCkmYCOvIIKz5JExEUDJaJuvWgcdjby2BoLjYO2YiYSKCSrt2+sK/06dh2za7q/F/2dl6OUy325G3dZEwEdZxu72T16SrU3NGFychAUJDbS2lIhImwloyCGseB4+XgISJsJoMwppHwkQENaObs3s35OXZW4u/kzARQa1hQ2jdWu/L5LWakTARQU+6OuaQMBFBTwZha87hc0xAwkT4gtEyWbtW/1CI6nP4HBPwUZi89tprtGzZkoiICLp168a6det88bbCKTp0gNq1oaAAdu2yuxr/ZHRxmjVz5BwT8EGYzJ07l0mTJjF58mQ2bdpEp06dGDRoEDlyWXrwCA3V99MB6epcKYePl4APwuTPf/4zY8eOZcyYMSQnJzNz5kxq167N3//+d6vfWjiJDMLWTLCHyblz59i4cSMDBgzwvmFICAMGDGBNBb+hioqKKCgoKPMQAUIGYWsm2MPk+PHjeDweGjduXOb5xo0bc+zYsXLHT5kyhZiYmJJHogOXphNXyAiTHTsgP9/eWvxRsIdJdT399NPk5+eXPA4Z610K/9e4sV4dTCm9JIGoHiNMWrSwtYzKWBomjRo1wu12k52dXeb57OxsmjRpUu748PBw6tatW+YhAogsMn1lPB7Yt0/vJyXZW0slLA2TsLAwUlJSWLZsWclzxcXFLFu2jB7GgJwIHj/9m//wn0zefx9WrJA1k6rk8GF9861atRw7xwR80M2ZNGkSb775JrNnz2bnzp088sgjnD59mjFjxlj91sJhPj/zU8tkbSYjRyr69dNDABkZtpblfN99p7ctW+pJaw5l+eyXu+++m9zcXJ599lmOHTtG586dWbx4cblBWRHYMjLg50924iQRNCSP1uxhD9eQlQUjRsD8+ZCWZneVDmWEiYO7OOCjAdjx48dz4MABioqKWLt2Ld2My9JFUPB4YMIEOEcYG0kBoAf6FLFS+piJE6XLc0kSJkJoq1frbj9AJrqr0x3vIKxS+kZ1q1fbUZ0f+P57vZUwEcHu6FHv/hr0IGzpMKnoOFGKtEyE0OLjvftGy6Qj3xDFqUseJ36ilISJEIbevfUZTZcLjtCMgyTippgb2ADo5xMT9XHiInl53hnDV11lby2XIWEiLOd2w4wZet/l8rZOerAGl0s/P326o8962sdolTRtCpGR9tZyGRImwifS0vTp32bNyg7CJiTIaeFK+UkXB3wwz0QIQ1oa3H47fD2zB4yHIfUz2fe9wh3qsrs05/KjMJGWifAptxuuf/A6qFWLsBM5uA/us7skZ5MwEaISERFw/fV6Xy76q5wRJg4ffAUJE2EXWSypaqRlIsRl3Hij3n7xhb11ONmZM3DkiN6XMBHiEnr10ttvvpGV1y7FWMOkbl19Z0SHkzAR9mjaVI8DFBdLV+dSSndxXM4/4yVhIuxjTHmVK/wq9u23emvcq9nhJEyEfYwwkXGTihlh0qaNvXVUkYSJsI8xbrJ2rb71pShr9269veYae+uoIgkTYZ9rroG4OB0kGzbYXY3zGC0TCRMhLsPl8rZOpKtTVkGBd4EXCRMhqsAIExmELWvPHr2Ni4N69WwtpaokTIS9jEHYL7/Up4mFZoyX+MngK0iYCLt17gxRUXDyJGzfbnc1zuFn4yUgYSLsFhpacnMu6eqUImEixBWQyWvlSTdHiCtQOkyMG+kEM6WkZSLEFenWTXd3srLgwAG7q7Hf0aNw6hSEhPjF1cIGCRNhv9q1IUXf6U+6OnhbJa1aQViYvbVUg4SJcAYZN/Hys2tyDBImwhnkoj8vP7smxyBhIpyhZ0+93bkTcnPtrcVufjj4ChImwikaNoT27fV+sHd1du3SW+nmCHGFUlP1dsUKO6uw15kz3hXWjHD1ExImwjn69dPbzz+3tw477d6t55k0aKAv8vMjEibCOfr00dtt24J33MS4Pql9e79Y97U0CRPhHLGx0KGD3l+1yt5a7LJjh94mJ9tbxxWQMBHOEuxdndItEz8jYSKcJdgHYaVlIoRJjHGT7dshJ8feWnzt7Fm/PZMDEibCaRo1go4d9X6wjZvs3q1Xm6tfHxo3truaapMwEc5jdHWCbdzEj8/kgISJcKJgHTfx4/ESkDARTtSnD8rlgh07yJiZw4oV4PHYXZQPGGHih+MlIGEiHChjZUN2hOpxkw8eWUG/ftCyJWRk2FuX5YxujrRMhKi5jAwYMQKWnE8FIJUVgF6EbcSIAA6UoiLYu1fvS8tEiJrxeGDCBH1pygpSAW+YGEvDTpwYoF0e40xOvXrQpInd1VwRCRPhGKtXw+HDen8VfSjGRTI7iSMb0IFy6FCArlCwdave+umZHJAwEQ5i3FoX4AQN+JpOANzE8kseFzC+/lpvO3Wyt44akDARjhEfX/bPSxkAwACWVnpcQJAwEcI8vXtDQoK3lW+Eyc0sARQuFyQmepeLDSgSJkKYx+2GGTP0vssFq+lNEWE05xBt0OuiTp+ujwso2dn64XJ5l2DwQxImwlHS0mD+fGjWDM5Qmy/oBcCIekuZP19/PeAYrZLWrfVN3P2UhIlwnLQ02L9fX5rT6G7d1fl9nyWBGSQQEF0ckDARDuV260t0Oj1+MwCuFZ/DhQv2FmUVI0yMq6X9lISJcLbrrtOLKxcUwPr1dldjjW++0VtpmQhhIbcbbrpJ7y9ZYm8tVigq0jceAwkTISx3s+7qBGSY7Nypu2/16unz3n5MwkQ43wA9CEtmJhQW2luL2TZv1ttOnfx2Gr1BwkQ431VX6ceFC7Bypd3VmGvDBr294QZ76zCBhInwD0ZX57PP7K3DbBs36q2EiRA+Mniw3n7yib11mOn8ediyRe+npNhaihkkTIR/6N8fatXSCwjt2WN3NebYvl2fzYmJgaQku6upMUvCZP/+/Tz44IO0atWKyMhIkpKSmDx5MufOnbPi7UQwiI72XuH33//aW4tZjPGSlBQI8f/f65b8DXbt2kVxcTFvvPEG27dvZ9q0acycOZP/+Z//seLtRLC45Ra9DZQwMcZLAqCLA+BSylgQz1ovv/wyr7/+Ot9//32Vv6egoICYmBjy8/OpW7euhdUJv7Bzp15sOTwcfvjBry+KA6BLF906mTsX7rrL7mouqao/hz5rW+Xn59OgQYNKjykqKqKgoKDMQ4gSbdvqZeqLivz/Bl3nznmn0QfAmRzwUZjs3buXV155hYcffrjS46ZMmUJMTEzJI9HPZwQKk7lcMGSI3vf3rs62bTpQ6teHVq3srsYU1QqTp556CpfLVelj165dZb4nKyuLwYMHc+eddzJ27NhKX//pp58mPz+/5HHo0KHq/41EYCs9buKbHro11q3T25QUv5/5agitzsGPPfYY999/f6XHXHXVVSX7R44coV+/ftx4443MmjXrsq8fHh5OeHh4dUoSwaZfPz1mcuAA7NoF7drZXdGVWbNGb3v0sLcOE1UrTGJjY4mNja3SsVlZWfTr14+UlBTefvttQgLg1JdwgKgovdDJp5/q1om/hslXX+ltAIWJJT/hWVlZpKam0rx5c/74xz+Sm5vLsWPHOHbsmBVvJ4KN0dX5z3/sreNK5eZ6797Xvbu9tZioWi2TqlqyZAl79+5l7969JCQklPmaj85Ei0A2dKi+9d+qVXDihB7E9CeZmXrbrp3/1V4JS1om999/P0qpCh9C1FhSkl7F3ePxz9ZJAHZxQK7NEf5q2DC9/fBDO6u4Msbg64032luHySRMhH/6KUw8/1nMv2afYcUKP7mh+fnz3rVspWUihP0y9l/PEXcC7rOn+cf9y+jXT0+Ozciwu7LL+OYb+PFHvUxj27Z2V2MqCRPhdzIyYMSdLhZ4hgEwjA8ByMqCESMcHijGeEm3bgFxpXBpgfW3EQHP49EncpSCDxkGwM/4NyF4SibETpzo4C7PihV627evrWVYQcJE+JXVq+HwYb2/ij6coB5x5NIDPaipFBw6pI9znOJi7xq2qam2lmIFCRPhV44e9e5foBaLuBWAO1h4yeMcY9s279IJAXKlcGkSJsKvxMeX/fNC7gBgOAsAdcnjHMHo4vTqpZegDDASJsKv9O4NCQneC20/YQiF1KElB+jGWlwufS8rY4VHRzHCJAC7OCBhIvyM2w0zZuh9lwvOEsm/+RkA9zAXgOnT9XGOUnq8pF8/e2uxiISJ8DtpaTB/PjRrpv88l7sBuMf9L+b/q5i0NBuLu5StWyEvD+rUgeuvt7saS0iYCL+Ulgb79+vVG0fNHsSFqBiaeI6QFveF3aVVzFhmMkDHS0DCRPgxt1sPP9x9XzihI4bpJ+fOtbOkS1u8WG+N+yYHIAkTERju1l0d5s/X9yR2kh9/9A6+GmvYBiAJExEYBgyAhg0hJweWLbO7mrJWrNAr6jdv7r8rw1WBhIkIDLVqwT336P3Zs+2t5WLG/ZGHDAmYxaMrImEiAoex2PnChZCfb2spZZQOkwAmYSICR0qKvuPf2bPwr3/ZXY22dy98951uOd10k93VWErCRAQOl8vbOnnnHTsr8TJaJb166ZuvBzAJExFY7r1XrxPy1VewZ4/d1XiXlRw61NYyfEHCRASW+HgYNEjv2906OX7cO4XekdNyzSVhIgLPmDEAqLfeYuWSc7z/PvasEfvRR/pNr7suYO4nXBkJExF4hg3jTP14XNnZ/HXgQkaOxJ41Yo03C4JWCUiYiACU8XEt/nBiHADpvFbyvE/XiM3Ph6VL9b6EiRD+x1gjdhZjuYCbPqymA1sBfLtG7IIFcO6cnvGanGzxmzmDhIkIKMYasUdoVrIKW+nWic/WiP3nP/X23nstfiPnkDARAaX02q+vkQ7AvbxLPU5c8jjTHTzovbBPwkQI/1R67deV9OUbrqUOp3mE1y95nOnee09vU1P1xX1BQsJEBJSya8S6mMqTAExkOhGcsX6NWKW881t+8QuL3sSZJExEQLl4jdi53M0+WhJHLg/yd8DiNWKXLYNvv9VT5++806I3cSYJExFwSq8R6yGUl/kNAE+6X2bBB+etPVP717/q7X33Bfy1OBeTMBEBqfQasanvjOFc/TgSPQe44+Tb1r3p4cN61ivA//t/1r2PQ4XaXYAQVjHWiIVIKPhfePRR1HPPsbr5vWSdqE18vB47Ma3L88or+pYWqalBM7ekNGmZiODw8MOcjmuF6+hRPhkyw/wp9nl53i7OpEkmvKD/kTARQSFjURgP5/wegCeZSkOOAyZOsf/LX+DUKejUCW69tYYv5p8kTETAM6bYz+HnbKYz9cgvOWVsyhT7Eye8p5CeeSag13mtjISJCHjGFHtFSMnU+gf5Oz3RN+yq8RT755+HkyehffuguaivIhImIuCVnjq/hhuZxVgA3mQskfxY8rVly66gdbJzJ7z6qt535E2OfUfCRAS8i6fOP8VLHCGeduziTzxW8vwLL1RzQLa4GMaP1wn0s58F9N36qkLCRAS8slPs4QQNuI9/APAIMxnBvJJjDx+G4cPhd7+rQivltddg+XKIjIQ//9mi6v2HhIkIeBdPsQdYxgCm8gQA/+A+urK2zPdMngyhofp44zF4cKkD1q6F3+iZtfzxj5CUZPHfwvkkTERQKD3F3vAML7KIoURylo+5jU5sqfQ1Pv30pzD67ju44w59y8+f/QweecTS2v2FhIkIGsYU+//9X/1nD6HcwwdsIIU4cllJX26i8vsUt2cbh65O1aO6HTrAu+8G7angi0mYiKDidkP//t4/n6YO/VnGSvoQQwFLuJnpTCCWnDLfF0YRE5nGOrqSyGEORLWDJUuC7mK+yriUMqbtOE9BQQExMTHk5+dTt25du8sRAcLj0WdtsrK8k9YiOMMr/IqHeAuAIsJYRR/20YoG5NGPz2lIHgCfMpBRvMdx1cimv4FvVfXnUMJEBKWMDD2N/uL//QP5lN/zW7qyvtz3HKYZv+NZ3uJBinGX+95AJWEixGVkZMCjj+oWSlmKtuyiD6towjFOUo8tdOYLelGMd1Kac39yzFXVn0NZgkAErbQ0uP12+P3v9Yx4Lxe7aMcu2l3ye407kAovGYAVQc3thuee804ZqarFiy0px69JmAgB/OEPMG8eVKU3HSzdm+qSMBHiJyNG6DWOli6FJ54o//VBgyRIKiMDsEKISlX151BaJkIIU0iYCCFMIWEihDCFhIkQwhQSJkIIU0iYCCFMIWEihDCFhIkQwhQSJkIIUzj6qmFjcm5BQYHNlQgRvIyfv8tNlnd0mBQWFgKQmJhocyVCiMLCQmJiYi75dUdfm1NcXMyRI0eIjo7G5eeL9hYUFJCYmMihQ4fkOqMakM/RHNX5HJVSFBYW0rRpU0JCLj0y4uiWSUhICAkJCXaXYaq6devKD4EJ5HM0R1U/x8paJAYZgBVCmELCRAhhCgkTHwkPD2fy5MmEh4fbXYpfk8/RHFZ8jo4egBVC+A9pmQghTCFhIoQwhYSJEMIUEiZCCFNImAghTCFh4iOvvfYaLVu2JCIigm7durFu3Tq7S/IrU6ZMoUuXLkRHRxMXF8ewYcPYvXu33WX5vZdeegmXy8XEiRNr/FoSJj4wd+5cJk2axOTJk9m0aROdOnVi0KBB5OTk2F2a31i5ciXp6elkZmayZMkSzp8/z8CBAzl9+rTdpfmt9evX88Ybb9CxY0dzXlAJy3Xt2lWlp6eX/Nnj8aimTZuqKVOm2FiVf8vJyVGAWrlypd2l+KXCwkLVunVrtWTJEtW3b181YcKEGr+mtEwsdu7cOTZu3MiAAQNKngsJCWHAgAGsWbPGxsr8W35+PgANGjSwuRL/lJ6eztChQ8v8v6wpR181HAiOHz+Ox+OhcePGZZ5v3Lgxu3btsqkq/1ZcXMzEiRPp2bMnHTp0sLscv/PBBx+wadMm1q9fb+rrSpgIv5Oens62bdv44osv7C7F7xw6dIgJEyawZMkSIiIiTH1tCROLNWrUCLfbTXZ2dpnns7OzadKkiU1V+a/x48ezaNEiVq1aFXBr3fjCxo0bycnJ4frrry95zuPxsGrVKl599VWKiopwu91X9NoyZmKxsLAwUlJSWLZsWclzxcXFLFu2jB49ethYmX9RSjF+/HgWLlzI8uXLadWqld0l+aX+/fuzdetWtmzZUvK44YYbGDVqFFu2bLniIAFpmfjEpEmTGD16NDfccANdu3Zl+vTpnD59mjFjxthdmt9IT09nzpw5fPTRR0RHR3Ps2DFArwAWGRlpc3X+Izo6utw4U1RUFA0bNqzx+JOEiQ/cfffd5Obm8uyzz3Ls2DE6d+7M4sWLyw3Kikt7/fXXAUhNTS3z/Ntvv83999/v+4JEObKeiRDCFDJmIoQwhYSJEMIUEiZCCFNImAghTCFhIoQwhYSJEMIUEiZCCFNImAghTCFhIoQwhYSJEMIUEiZCCFP8f9bafPFH+PmDAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR4AAAGzCAYAAAD5S3jmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAodklEQVR4nO3de1wU5f4H8M+ywIIEKyh4gQUU73lJRQ3RFLGDaZnlpSxNTS0T73oKOyWWnqj0pF3MvGX+NLWOqVnej7cyNTVBzRTFG4iieNtF0BXh+f0xsbqy3HTnYYHP+/Wa18qzszPfXZnPzjwz86ARQggQEUnkVNoFEFHFw+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgoYf2zTffQKPR4MyZM4XON3nyZGg0GjlF/S04OBgDBw6Uuk4qGoPHDjQaTZHT5MmTS7vMcufMmTMFft6PP/54aZdHhXAu7QLKg8WLFxf43OTJk3Hy5Em0adNGYkWO6Z133kFMTIzdl9u3b1907drVqs3X1xcAkJiYCCcnfr86GgaPHfTr189m+/z583Hy5EmMHDkSTz31lOSqHI+zszOcne3/K9eiRYsC/w90Op3d11ccmZmZ8PDwKJV1lwX8KlDJkSNHMGrUKDRv3hzTpk2zei4zMxPjx4+HwWCATqdD/fr1MX36dNw/UMCdO3cwZcoUhISEQKfTITg4GG+//TbMZrPVfMHBwXj66aexfft2hIaGwt3dHU2aNMH27dsBACtXrkSTJk3g5uaGli1bIj4+Pl+9x44dQ69eveDj4wM3NzeEhoZizZo1Nt9Xp06d4O7ujoCAAEydOhW5ubnF+kxs9fFoNBqMGDECq1evRuPGjaHT6fDoo49iw4YNxVpmUWz18Rw6dAgdOnSweg8LFy7M109V0CHy/cvM6+PasWMHhg8fDj8/PwQEBFieX79+Pdq3bw8PDw94enqiW7duOHLkiF3eX1nFPR4VZGVloU+fPtBqtVi+fLnVt64QAt27d8e2bdswePBgPPbYY9i4cSP++c9/IjU1FTNmzLDMO2TIECxatAi9evXC+PHj8fvvvyMuLg5Hjx7FqlWrrNaZlJSEl156Ca+//jr69euH6dOn45lnnsFXX32Ft99+G8OHDwcAxMXFoU+fPlaHIEeOHEF4eDj8/f0RExMDDw8PfP/99+jRowd++OEHPPfccwCAtLQ0RERE4M6dO5b55s6dC3d394f6vHbu3ImVK1di+PDh8PT0xGeffYaePXsiOTkZVapUKdbnffnyZas2vV4PFxeXfPOmpqYiIiICGo0GEydOhIeHB+bPn2+XPaPhw4fD19cXkyZNQmZmJgDlMHzAgAGIiorCRx99hKysLMyePRvt2rVDfHw8goODH3q9ZZIgu3v11VcFALFo0aJ8z61evVoAEFOnTrVq79Wrl9BoNCIpKUkIIURCQoIAIIYMGWI134QJEwQAsXXrVktbUFCQACB27dpladu4caMAINzd3cXZs2ct7XPmzBEAxLZt2yxtkZGRokmTJuLWrVuWttzcXNG2bVtRt25dS9uYMWMEAPH7779b2i5duiT0er0AIE6fPl3o5xIbGyvu/5UDIFxdXS3vWwghDh48KACIzz//vNDlnT59WgCwOeW9v6CgIDFgwADLa0aOHCk0Go2Ij4+3tF25ckX4+Pjkew8ARGxsbL713r/MhQsXCgCiXbt24s6dO5b2jIwMUblyZTF06FCr16elpQm9Xp+vvSLhoZadLV26FF9//TX69++PV155Jd/z69atg1arxahRo6zax48fDyEE1q9fb5kPAMaNG5dvPgBYu3atVXujRo0QFhZm+TmvM7tTp04IDAzM137q1CkAwNWrV7F161b06dMHGRkZuHz5Mi5fvowrV64gKioKJ06cQGpqqqWmxx9/HK1bt7Ysz9fXFy+//HJxPx6bOnfujJCQEMvPTZs2hZeXl6XGorz22mvYvHmz1dSsWTOb827YsAFhYWF47LHHLG0+Pj4P/R4AYOjQodBqtZafN2/ejOvXr6Nv376Wz/Xy5cvQarVo06YNtm3b9tDrLKt4qGVHJ06cwLBhw1CvXj18+eWXNuc5e/YsatasCU9PT6v2hg0bWp7Pe3RyckKdOnWs5qtevToqV65smS/PveECKIcaAGAwGGy2X7t2DYByiCaEwLvvvot3333XZs2XLl2Cv78/zp49a/PsXP369W2+rrjurx0AvL29LTUWpW7duujcuXOx5j179qxVQOe5/3N+ELVq1bL6+cSJEwCU8LfFy8vroddZVjF47MRsNuOFF17A7du3sXz5cjzyyCN2WW5xL7i795u2OO3i747svI7hCRMmICoqyua89tgoC1NUjY4mJyfHZvv9fV15n+3ixYtRvXr1fPOrcYavrKi479zOJkyYgPj4eHz66ado3rx5gfMFBQXhf//7HzIyMqz2eo4dO2Z5Pu8xNzcXJ06csOwNAcDFixdx/fp1y3wPq3bt2gAAFxeXIvcagoKCLN/i90pMTLRLLTIEBQUhKSkpX7utNm9vb1y/ft2q7fbt27hw4UKx1pV3+Ojn51fsPbKKgn08drBq1Sp88cUX6N69e76+m/t17doVOTk5+OKLL6zaZ8yYAY1GY7neJ++CuJkzZ1rN98knnwAAunXrZpfa/fz80LFjR8yZM8fmBpWenm5V+549e7B3716r57/99lu71CJDVFQUdu/ejYSEBEvb1atXbb6HkJAQ/PLLL1Ztc+fOLXCPx9a6vLy88MEHHyA7Ozvf8/d+thUN93ge0oULFzB48GBotVpERkZiyZIlNucLCQlBWFgYnnnmGUREROBf//oXzpw5g2bNmmHTpk348ccfMWbMGMu3ZLNmzTBgwADMnTsX169fR4cOHbB3714sWrQIPXr0QEREhN3ew6xZs9CuXTs0adIEQ4cORe3atXHx4kXs3r0b586dw8GDBwEAb775JhYvXowuXbpg9OjRltPpQUFBOHTokN3qUdObb76JJUuW4Mknn8TIkSMtp9MDAwNx9epVq0PbIUOGYNiwYejZsyeefPJJHDx4EBs3bkTVqlWLtS4vLy/Mnj0b/fv3R4sWLfDiiy/C19cXycnJWLt2LcLDw/N9AVUYpXtSrezbtm1bgad0753uPf2akZEhxo4dK2rWrClcXFxE3bp1xbRp00Rubq7VsrOzs8V7770natWqJVxcXITBYBATJ060Ou0thHJ6t1u3bvlqAyCio6Ot2vJOQU+bNs2q/eTJk+KVV14R1atXFy4uLsLf3188/fTTYsWKFVbzHTp0SHTo0EG4ubkJf39/MWXKFLFgwYKHOp1+f4157+nez8yWgt5LUcuJj48X7du3FzqdTgQEBIi4uDjx2WefCQAiLS3NMl9OTo546623RNWqVUWlSpVEVFSUSEpKKvB0+r59+2zWsG3bNhEVFSX0er1wc3MTISEhYuDAgWL//v2Fvr/yTCOEg/bgEUk0ZswYzJkzBzdu3Ciws5vsh308VOHcvHnT6ucrV65g8eLFaNeuHUNHEvbxUIUTFhaGjh07omHDhrh48SIWLFgAk8lU4HVMZH8MHqpwunbtihUrVmDu3LnQaDRo0aIFFixYgCeeeKK0S6sw2MdDRNKxj4eIpGPwEJF0Dt3Hk5ubi/Pnz8PT01P6IOFEpBBCICMjAzVr1rTbMLIOHTznz5/Pd3c1EZWOlJQUq5EVH4ZDB0/eTZQpKSkVeggBotJkMplgMBjyDeXyMBw6ePIOr7y8vBg8RKXMnt0d7FwmIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJJ2qwZOamop+/fqhSpUqcHd3R5MmTbB//341V0lEZYBq92pdu3YN4eHhiIiIwPr16+Hr64sTJ07A29tbrVUSURmhWvB89NFHMBgMWLhwoaXt/j9qfz+z2Qyz2Wz52WQyqVUeEZUi1Q611qxZg9DQUPTu3Rt+fn5o3rw55s2bV+hr4uLioNfrLRPH4iEqn1Qb7N3NzQ0AMG7cOPTu3Rv79u3D6NGj8dVXX2HAgAE2X2Nrj8dgMMBoNHJYDKJSYjKZoNfr7bodqhY8rq6uCA0Nxa5duyxto0aNwr59+7B79+5iLUONN0xEJaPGdqjaoVaNGjXQqFEjq7aGDRsiOTlZrVUSURmhWvCEh4cjMTHRqu348eMICgpSa5VEVEaoFjxjx47Fnj178MEHHyApKQlLly7F3LlzER0drdYqiaiMUC14WrVqhVWrVmHZsmVo3LgxpkyZgpkzZ+Lll19Wa5VEVEY49J8wZucyUekrU53LREQFYfAQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSSQueDz/8EBqNBmPGjJG1SiJyUFKCZ9++fZgzZw6aNm0qY3VE5OBUD54bN27g5Zdfxrx58+Dt7V3ovGazGSaTyWoiovJH9eCJjo5Gt27d0Llz5yLnjYuLg16vt0wGg0Ht8oioFKgaPMuXL8eBAwcQFxdXrPknTpwIo9FomVJSUtQsj4hKibNaC05JScHo0aOxefNmuLm5Fes1Op0OOp1OrZKIyEFohBBCjQWvXr0azz33HLRaraUtJycHGo0GTk5OMJvNVs/ZYjKZoNfrYTQa4eXlpUaZRFQENbZD1fZ4IiMjcfjwYau2QYMGoUGDBnjrrbeKDB0iKr9UCx5PT080btzYqs3DwwNVqlTJ105EFQuvXCYi6VTb47Fl+/btMldHRA6KezxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0qgZPXFwcWrVqBU9PT/j5+aFHjx5ITExUc5VEVAaoGjw7duxAdHQ09uzZg82bNyM7Oxv/+Mc/kJmZqeZqicjBaYQQQtbK0tPT4efnhx07duCJJ54ocn6TyQS9Xg+j0QgvLy8JFRLR/dTYDp3tspRiMhqNAAAfHx+bz5vNZpjNZsvPJpNJSl1EJJe0zuXc3FyMGTMG4eHhaNy4sc154uLioNfrLZPBYJBVHhFJJO1Q64033sD69euxc+dOBAQE2JzH1h6PwWDgoRZRKSqzh1ojRozAzz//jF9++aXA0AEAnU4HnU4noyQiKkWqBo8QAiNHjsSqVauwfft21KpVS83VEVEZoWrwREdHY+nSpfjxxx/h6emJtLQ0AIBer4e7u7uaqyYiB6ZqH49Go7HZvnDhQgwcOLDI1/N0OlHpK3N9PBIvESKiMoT3ahGRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOtWDZ9asWQgODoabmxvatGmDvXv3qr1KInJwqgbPd999h3HjxiE2NhYHDhxAs2bNEBUVhUuXLqm5WiJycKoGzyeffIKhQ4di0KBBaNSoEb766itUqlQJX3/9tZqrJSIHp1rw3L59G3/88Qc6d+58d2VOTujcuTN2795t8zVmsxkmk8lqIqLyR7XguXz5MnJyclCtWjWr9mrVqiEtLc3ma+Li4qDX6y2TwWBQqzwiKkUOdVZr4sSJMBqNliklJaW0SyIiFTirteCqVatCq9Xi4sWLVu0XL15E9erVbb5Gp9NBp9OpVRIROQjV9nhcXV3RsmVLbNmyxdKWm5uLLVu2ICwsTK3VElEZoNoeDwCMGzcOAwYMQGhoKFq3bo2ZM2ciMzMTgwYNUnO1ROTgVA2eF154Aenp6Zg0aRLS0tLw2GOPYcOGDfk6nImoYtEIIURpF1EQk8kEvV4Po9EILy+v0i6HqEJSYzt0qLNaRFQxMHiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIOgYPEUnH4CEi6Rg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukYPEQkHYOHiKRj8BCRdAweIpKOwUNE0jF4iEg6Bg8RScfgISLpVAmeM2fOYPDgwahVqxbc3d0REhKC2NhY3L59W43VEVEZ46zGQo8dO4bc3FzMmTMHderUwZ9//omhQ4ciMzMT06dPV2OVRFSGaIQQQsaKpk2bhtmzZ+PUqVPFfo3JZIJer4fRaISXl5eK1RFRQdTYDqX18RiNRvj4+BQ6j9lshslkspoAAEuWAHLykYgkkBI8SUlJ+Pzzz/H6668XOl9cXBz0er1lMhgMyhPR0UBEBPDXXxKqJSK1lSh4YmJioNFoCp2OHTtm9ZrU1FR06dIFvXv3xtChQwtd/sSJE2E0Gi1TSkqK8oSbG7BjB9CsGRATA2RmluxdEpFDKVEfT3p6Oq5cuVLoPLVr14arqysA4Pz58+jYsSMef/xxfPPNN3ByKtkOluXY8vBheL3zDvDjj8oTQUHAZ58B3buXaHlEVHJq9PGo1rmcmpqKiIgItGzZEkuWLIFWqy3xMvK94Z9+AkaOBM6eVWbo3h349FMgONi+xRORRZnpXE5NTUXHjh0RGBiI6dOnIz09HWlpaUhLS3u4BT/zDHDkiHK45ewMrFkDNGoEfPghwGuEiMoMVfZ4vvnmGwwaNMjmcyVZXaFJ+9dfSqfz9u3Kzw0bAl98AXTq9IBVE5EtZWaPZ+DAgRBC2JzsplEjYOtWYPFiwM8POHoUiIwEevYEzpyx33qIyO7K9r1aGg3Qrx9w7JjS96PVAitXKns/sbFAVlZpV0hENpTt4Mnj7a2c5UpIUK73uXULeP99oEED4L//tVx8mJOjHJktW6Y85uSUZtFEFVf5CJ48jRsDW7YAK1YAgYFASgrQpw8QEYH/fXIIwcFKLr30kvIYHKzsIBGRXOUreADl8KtnT6XPZ/Jky8WHEeObI+ZcNHxw9zqk1FSgVy+GD5Fs5S948lSqBMTGIufIMfzk3hta5CIaX+IkQjAe0+EKs+X2rzFjeNhFJFP5DZ6//ZochO43v0cEtiIBzVAZRkzHP3EMDdAH30EIgZQU4NdfS7tSooqj3AfPhQvK43ZEoCX+wCB8jVTURC2cwXd4EbvQFmHYZZmPiNRX7oOnRo27/86FFt9gEOrhOCbhPdyAB8KwB7sQjsg5vYGTJ0uvUKIKpNwHT/v2QECA0uecJwsemIJJqIsTmI8hyIET/HasUK7/GTcOKOJGWCJ6OOU+eLRa5T5SwDp8AOCipgZe08zDtk8SgKgoIDsbmDEDqF0bmDoVuHFDer1EFUG5Dx4AeP555dIef3/r9oAApb3z2CbAhg3K1KwZYDIB774LhIQoFyaazaVTOFE5JW3M5Qdh75vTcnKUs1cXLih9P+3bK3tEVnJzge+/V4InKUlpCwxUrgnq31+5K56oAilT4/HYQ6kO9p6dDSxcCLz3HnD+vNLWoIFyCPb88/mP24jKqTJzd3q54OICvPaastczfTpQpYpyM2qvXkCrVsD69RyAnugBMXiK4u4OjB8PnDoFTJoEPPII8McfQNeuQJs2wM8/M4CISojBU1xeXsph16lTShBVqgTs26eMitiqlTIaIgOIqFgYPCXl66scep0+Dbz5JuDhoewBPfss0KIFsGqV0kFNRAVi8DwoPz/go4+U0Q5jYpRDsIQEpeO5eXPlPD0DiMgmBs/DqloViItTAuhf/wI8PYFDh4DevYFHHwW+/prXARHdh8FjL1WqKKfaz5xROqH1euUs2ODBypXQ06YBRmNpV0nkEBg89ubjo3RCJycrYVOzpnId0JtvKhcivvUWeCs8VXQMHrV4eQETJiid0AsXKjegmkzAxx8rY64OGaLsEd2H40JTRcDgUZurKzBwIPDnn8op93btlD8+uGCB8id6nnkG2LQJEAIrV4LjQlOFwOCRxclJCZlffwV++005/S6EcgFiVBRMhkbY2nMWrp/LsHoZx4Wm8ojBUxratgVWrwYSE4FRoyA8PeGVegxfYATOIQAzMAYhUG5Q5bjQVB4xeEpTvXrAp59i57JzGIHPkYh60MOEMfgUx1EPP6MborABELkcF5rKFQaPAzhn8sIsjEBDHEUXrMdadIUTBLphHTbgKRxHPbyFD3HtaFppl0pkFwweB5A3LrSAEzaiC57GWtTFcczEaJjgiTo4iQ8xEc+ONCh/M2zjRl4VTeoTQhkc7+mn7b5ojsfjAHJylLNXqan57zOthEy8iO8w3HUeWt7ec/eJoCDl4sQBA5Trg4jsJTsb+O475dKPw4dhAqAHOB5PeVPYuNA3NR5YqHkVZ5ftVm7FGDUKqFwZOHtWuUI6OBiIjAQWLeIY0fRwjEZg5kxlyN/+/YHDh5V7EIcPt/+6hAMzGo0CgDAajaVdihQ//CBEQIAQyn6PMhkMSruVrCwhFi8WomNH65krVRKif38hNm8W4s6dUnkPVAYlJAjx+utCeHjc/V2qVk2IDz4Q4upVVbZDHmo5mGKNC32vs2eBJUuUPZ4TJ+62+/sDffoAL7wAtG7NoVrJmtkM/PAD8OWXynVleRo2VK7deOUVwM0NAMdcLu1yHJsQwO+/A//3f8Dy5cC1a3efCw5WQqhPH2XMoAJCqMShR2XPsWPKl9SCBUB6utLm7KwM5zJ8OPDEE/l+P1TZDu2276SCinaoZTe3bgmxerUQffta7z4DQtSpI8TEiUL8/rsQOTmWl9g6zAsIsHGYR2XP5ctCfPGFEK1bW/8H+/sL8f77Qpw/X+jLeahFJZeVBaxbp5ylWLsWuHnz7nM1agDdu+M33x6InBoBM3RWL8374luxQvlCpDLk9m3l/33RIuX/PTtbaddqgaeeAgYNArp3L9afa+KhFj2cGzeUe8NWrVJ+Ke85C2aCJzagC37G09iEf+AiqgNQwicgQLnJnoddDu7mTeWG4x9+AH76Cbh+/e5zLVoo/TZ9+yqjZ5YAg4fsx2wGtm1D6uwfoVnzI2rCeoygA2iOjYjCBnTBboRh0zZXdOxYOqVSITIylC+RlSuVPZvMzLvP1agB9OunBE7jxg+8CgYP2d2yZcDLL+UiFPvxLH5EF2xASxywmicDj8DYshMCXokEOnZUfomdincJGDus7UwI4Phx5er1DRuArVuth9Y1GJTj4p49lZuR7fBhM3jI7rZvV8b9uZcfLuJJbEYXbMA/sAl+SLeewcdHOfvRsSPQoQPQtKnNIFq5Ehg9Gjh37m5bQIBysST7jErAZAK2bLkbNmfPWj9fp44SND17AqGhdr90gsFDdlfY7RoA4IRcdKkWj59GboLTrzuAnTutd+cB5UrqNm2sppU7qqBXr/zLLG6HdYXeU0pPVz7nnTuVD+HAAesxUVxdlQ8kKgro0kXZA1XxOi0GD6li5UplsDHAOihshkR2tvJ3xHbsUHaXdu60eavGaec6+O1OG+xFaxxEMxxEMxhR2bLcwjqsi7unVFA43due14+alqZsz97eys7D4sXW63z2WWWIJOmys5Vra/74427YJCbmn69uXSVkoqKUPU0PD2klMnhINbY2doNBuXWn0MOiO3eAgweVixf37FEejx+3OetZBFpC6CCa4c1FjdH6xdrKN/g9dRRnT6mgcOrbV+m3ure9JFTdGm7eVIbAPXBAmeLjlfvvbP35o0cfVYbJbd9eeQwKUrGwwjF4SFX2Orz5Yd5VzH1tHx7HHrTAATTDQQTjrO2ZnZyAWrWAevWQW7ce3vm/+th7vS5OoxbOIQC377m2KG9P6ZNPlIuw1frNfajl5uYqx62Jifmn5GTbC/f0BB57DAgLU0ImPFzpR3MQDB4qE2x1WOtxHU1x6J79nYNo7n4MzjcLv6P+IvyQjECkwIBkBOIcAnDL0w+nMqoiHb64jKq4jKq4gUcA2Kefw+ZhV3a2chvK1avAlSvKcdu5c0BKyt3HlBQldPIu1rOlalXlmprmzZXHFi2Uv7tWzLOEpYHBQ2VCUR3Wlj6eUwLa9DTl0CwxEUfXHMeJtcdRD8cRiGRUws38Ly7ALehwFT7IhIdlykIly7/N0EFAAwENcuFk9W9X3IYbblkmd9xEZNtbylXf168rQZORUWQNFs7OytAS9esrw9vWr3938vUtczfsqrEdFn29NFEJ5Y0v1KuXso3Z6rCeORPQOmuUY7oaNYAOHXCxHvDs2rw5BargCgxIgQEpCEQyDEhBAM6hKi7DF+mWR3fcghvM+S6CfCi7CmivXFn5q7FVqijpaTDkf6xRA3BxsV8t5RD3eEg1Je2wLs6eUtWqd2+qzuOOLPgiHd64ds/+zt2pErKgg/nvfRwBJ+RaPWbDBbfghpt/R9gtuGHpSndlWAhvb6W/pUoVJXQqzDn9u3ioRWVOSTusizq1/913wLhxBYeTPZTaqXUHxUMtKnO0WpToHq/nn1dOmds6VZ63p6TV2j6MsxeGjvq4x0MOqag9pYIO41580YGv4ymjeKhFdI9yceVyGcDgISLp1NgOHfeqJSIqtxg8RCQdg4eIpGPwEJF0DB4iko7BQ0TSMXiISDoGDxFJx+AhIukc+ibRvIuqTSZTKVdCVHHlbX/2vMnBoYPnypUrAACDwVDKlRDRlStXoNfr7bIshw4en78HvE5OTrbbG1abyWSCwWBASkpKmbm/jDXLURZrBgCj0YjAwEDL9mgPDh08Tn8PgK3X68vUfxQAeHl5sWYJWLM8TnYckJ6dy0QkHYOHiKRz6ODR6XSIjY2FTqcremYHwZrlYM3yqFG3Qw8ERkTlk0Pv8RBR+cTgISLpGDxEJB2Dh4ikY/AQkXSlFjxxcXFo1aoVPD094efnhx49eiAxMbHI1/33v/9FgwYN4ObmhiZNmmDdunUSqlU8SM3z5s1D+/bt4e3tDW9vb3Tu3Bl79+6VVPGDf855li9fDo1Ggx49eqhX5H0etObr168jOjoaNWrUgE6nQ7169aT9fjxozTNnzkT9+vXh7u4Og8GAsWPH4tatWxIqVsyePRtNmza1XE0dFhaG9evXF/oau2yDopRERUWJhQsXij///FMkJCSIrl27isDAQHHjxo0CX/Pbb78JrVYrPv74Y/HXX3+Jd955R7i4uIjDhw87bM0vvfSSmDVrloiPjxdHjx4VAwcOFHq9Xpw7d85ha85z+vRp4e/vL9q3by+effZZ9Yv924PUbDabRWhoqOjatavYuXOnOH36tNi+fbtISEhw2Jq//fZbodPpxLfffitOnz4tNm7cKGrUqCHGjh0rpWYhhFizZo1Yu3atOH78uEhMTBRvv/22cHFxEX/++afN+e21DZZa8Nzv0qVLAoDYsWNHgfP06dNHdOvWzaqtTZs24vXXX1e7PJuKU/P97ty5Izw9PcWiRYtUrKxgxa35zp07om3btmL+/PliwIABUoPnfsWpefbs2aJ27dri9u3bEisrWHFqjo6OFp06dbJqGzdunAgPD1e7vEJ5e3uL+fPn23zOXtugw/TxGI1GACj0Dtjdu3ejc+fOVm1RUVHYvXu3qrUVpDg13y8rKwvZ2dl2vdO3JIpb8/vvvw8/Pz8MHjxYRlmFKk7Na9asQVhYGKKjo1GtWjU0btwYH3zwAXJycmSVaaU4Nbdt2xZ//PGH5dD71KlTWLduHbp27Sqlxvvl5ORg+fLlyMzMRFhYmM157LYNPnAs2lFOTo7o1q1bkUnv4uIili5datU2a9Ys4efnp2Z5NhW35vu98cYbonbt2uLmzZsqVVaw4tb866+/Cn9/f5Geni6EEKW6x1PcmuvXry90Op149dVXxf79+8Xy5cuFj4+PmDx5sqRK7yrJ78ann34qXFxchLOzswAghg0bJqFCa4cOHRIeHh5Cq9UKvV4v1q5dW+C89toGHSJ4hg0bJoKCgkRKSkqh8zlS8BS35nvFxcUJb29vcfDgQRUrK1hxajaZTCI4OFisW7fO0laawVPcz7lu3brCYDCIO3fuWNr+85//iOrVq6tdYj7FrXnbtm2iWrVqYt68eeLQoUNi5cqVwmAwiPfff19SpQqz2SxOnDgh9u/fL2JiYkTVqlXFkSNHbM5bboInOjpaBAQEiFOnThU5r8FgEDNmzLBqmzRpkmjatKlK1dlWkprzTJs2Tej1erFv3z4VKytYcWuOj48XAIRWq7VMGo1GaDQaodVqRVJSkqSKS/Y5P/HEEyIyMtKqbd26dQKAMJvNapWYT0lqbteunZgwYYJV2+LFi4W7u7vIyclRq8QiRUZGitdee83mc/baBkutj0cIgREjRmDVqlXYunUratWqVeRrwsLCsGXLFqu2zZs3F3g8am8PUjMAfPzxx5gyZQo2bNiA0NBQlau0VtKaGzRogMOHDyMhIcEyde/eHREREUhISJAyDO2DfM7h4eFISkpCbm6upe348eOoUaMGXF1d1SwXwIPVnJWVlW9wLa1Wa1leacnNzYXZbLb5nN22wZLnoX288cYbQq/Xi+3bt4sLFy5YpqysLMs8/fv3FzExMZaff/vtN+Hs7CymT58ujh49KmJjY6WeTn+Qmj/88EPh6uoqVqxYYfWajIwMh635frIPtR6k5uTkZOHp6SlGjBghEhMTxc8//yz8/PzE1KlTHbbm2NhY4enpKZYtWyZOnTolNm3aJEJCQkSfPn2k1CyEEDExMWLHjh3i9OnT4tChQyImJkZoNBqxadMmmzXbaxssteABYHNauHChZZ4OHTqIAQMGWL3u+++/F/Xq1ROurq7i0UcfLbQjzBFqDgoKsvma2NhYh635frKD50Fr3rVrl2jTpo3Q6XSidu3a4t///rdVn4+j1ZydnS0mT54sQkJChJubmzAYDGL48OHi2rVrUmoWQohXX31VBAUFCVdXV+Hr6ysiIyMtoWOrZiHssw1yPB4iks5hruMhooqDwUNE0jF4iEg6Bg8RScfgISLpGDxEJB2Dh4ikY/AQkXQMHiKSjsFDRNIxeIhIuv8HJwWSJJN1Kf4AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "gradient_desc(0.5, 100, 0.05)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note this notebook was inspired by [this post](https://machinelearningmind.com/2019/10/06/gradient-descent-introduction-and-implementation-in-python/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## A numerical example of neural network training \n", "\n", "Let's consider a simple two-layer neural network with two inputs (x1 and x2), two neurons in the first layer (hidden layer), and two neurons in the second layer (output layer). We'll use a mean squared error (MSE) loss function.\n", "\n", "Here's the architecture:\n", "- Inputs: x1, x2\n", "- Hidden layer neurons: h1, h2\n", "- Output layer neurons: o1, o2\n", "\n", "![](images/backpropagation.png)\n", "\n", "The forward pass can be expressed as follows:\n", "\n", "$h_1 = w_{11} \\cdot x_1 + w_{21} \\cdot x_2 + b_1$\n", "\n", "$h_2 = w_{12} \\cdot x_1 + w_{22} \\cdot x_2 + b_2$\n", "\n", "$o_1 = w_{31} \\cdot h_1 + w_{41} \\cdot h_2 + b_3$\n", "\n", "$o_2 = w_{32} \\cdot h_1 + w_{42} \\cdot h_2 + b_4$\n", "\n", "Here, $w$ represents weights, $b$ represents biases, and the subscripts denote the connection between neurons.\n", "\n", "Let's assume the target outputs are $y_1$ and $y_2$. The MSE loss is given by:\n", "$L = \\frac{1}{2} \\sum_{i=1}^{2} (y_i - o_i)^2$\n", "\n", "Now, we want to minimize this loss using backpropagation.\n", "\n", "### 1. Compute the error terms at the output layer:\n", "\n", "$ \\delta_{o1} = (o_1 - y_1) \\cdot \\sigma'(o_1)$\n", "\n", "$ \\delta_{o2} = (o_2 - y_2) \\cdot \\sigma'(o_2)$\n", "\n", "Here, $\\sigma'$ is the derivative of the activation function used in the output layer (for simplicity, assume a linear activation, so $\\sigma'(x) = 1$).\n", "\n", "### 2. Compute the error terms at the hidden layer:\n", "\n", "$ \\delta_{h1} = (\\delta_{o1} \\cdot w_{31} + \\delta_{o2} \\cdot w_{32}) \\cdot \\sigma'(h1)$\n", "\n", "$ \\delta_{h2} = (\\delta_{o1} \\cdot w_{41} + \\delta_{o2} \\cdot w_{42}) \\cdot \\sigma'(h2)$\n", "\n", "\n", "### 3. Update the weights and biases using the error terms and learning rate ($\\alpha$):\n", "\n", "$ w_{31}^{new} = w_{31} - \\alpha \\cdot \\delta_{o1} \\cdot h_1$\n", "\n", "$ w_{41}^{new} = w_{41} - \\alpha \\cdot \\delta_{o1} \\cdot h_2$\n", "\n", "$ w_{32}^{new} = w_{32} - \\alpha \\cdot \\delta_{o2} \\cdot h_1$\n", "\n", "$ w_{42}^{new} = w_{42} - \\alpha \\cdot \\delta_{o2} \\cdot h_2$\n", "\n", "$ w_{11}^{new} = w_{11} - \\alpha \\cdot \\delta_{h1} \\cdot x_1$\n", "\n", "$ w_{21}^{new} = w_{21} - \\alpha \\cdot \\delta_{h1} \\cdot x_2$\n", "\n", "$ w_{12}^{new} = w_{12} - \\alpha \\cdot \\delta_{h2} \\cdot x_1$\n", "\n", "$ w_{22}^{new} = w_{22} - \\alpha \\cdot \\delta_{h2} \\cdot x_2$\n", "\n", "\n", "### 4. Update the biases similarly:\n", "\n", "$ b_3^{new} = b_3 - \\alpha \\cdot \\delta_{o1}$\n", "\n", "$ b_4^{new} = b_4 - \\alpha \\cdot \\delta_{o2}$\n", "\n", "$ b_1^{new} = b_1 - \\alpha \\cdot \\delta_{h1}$\n", "\n", "$ b_2^{new} = b_2 - \\alpha \\cdot \\delta_{h2}$\n", "\n", "This process is repeated iteratively until the network converges to a solution that minimizes the error. The learning rate ($\\alpha$) is a hyperparameter that determines the step size during weight and bias updates. Adjustments may be needed based on the specific problem and dataset.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 2 }