ift7025-projet/Code/Rapport-Final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "- Projet final présenté par: **François Pelletier**\n",
    "- Matricule: **908144032**\n",
    "- Dans le cadre du cours: **IFT-7025**\n",
    "\n",
    "# Librairies utilisées"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import load_datasets as ld\n",
    "import matplotlib.pyplot as plt\n",
    "import DecisionTree\n",
    "import NeuralNet\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iris Dataset\n",
    "\n",
    "La présentation des résultats liés à ce jeu de données servira aussi à expliquer les concepts et répondre aux questions. Pour les autres jeux de données, seuls les résultats et la discussion seront détaillés.\n",
    "\n",
    "- Chargement du jeu de données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train1, train_labels1, test1, test_labels1 = (\n",
    "    ld.load_iris_dataset(train_ratio = 0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entrainement de l'arbre de décision\n",
    "\n",
    "Dans cette section, on entraine un arbre de décision basé sur la mesure d'entropie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt1 = DecisionTree.DecisionTree(attribute_type=\"continuous\")\n",
    "start_time = time.time()\n",
    "_ = dt1.train(train1, train_labels1,verbose=False)\n",
    "dt1_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Courbe d'apprentissage\n",
    "\n",
    "Je trace une courbe d'apprentissage qui effectue une validation croisée sur 10 sous-ensembles de l'ensemble d'entrainement. Je fais la moyenne de l'accuracy pour chaque sous-ensemble de test, puis j'itère en ajoutant un exemple à la fois à l'ensemble d'entrainement.\n",
    "\n",
    "- Bogue connu: J'ai débuté la courbe d'apprentissage avec 30 données, car j'ai de la difficulté à exécuter la validation croisée avec moins sans que ça plante vu que je crée les sous-groupes à l'avance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def courbe_apprentissage_dt(dt,train,train_labels,nb_cv_split=10):\n",
    "    accuracy_cum = []\n",
    "    range_lc = range(30,len(train_labels))\n",
    "    for i in range_lc:\n",
    "        range_lc_split_test = np.array_split(range(i),nb_cv_split)\n",
    "        range_lc_split_train = (\n",
    "            [np.setdiff1d(range(i),t) for t in range_lc_split_test])\n",
    "        accuracy_cv = []\n",
    "        for r_i in range(nb_cv_split):\n",
    "            try:\n",
    "                training = dt.train(train[range_lc_split_train[r_i]], \n",
    "                                     train_labels[range_lc_split_train[r_i]],\n",
    "                                     verbose=False)\n",
    "                testres = dt.test(train[range_lc_split_test[r_i]], \n",
    "                                train_labels[range_lc_split_test[r_i]],\n",
    "                                verbose=False)\n",
    "                accuracy_cv.append(testres[1])\n",
    "            except:\n",
    "                pass\n",
    "        accuracy_cum.append(np.mean(accuracy_cv))\n",
    "    return range_lc,accuracy_cum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/francois/ift7025-projet/Code/metrics.py:40: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  myPrecision = cm[label_num,label_num] / sum(cm[:,label_num])\n"
     ]
    }
   ],
   "source": [
    "dt_range_lc1,dt_accuracy_cum1 = courbe_apprentissage_dt(dt1,train1,train_labels1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voici le graphique de la courbe d'apprentissage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5a0733358>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl823d9+PHXW5IlH/J95XBiO1eTNEmTJk1LL0rPlEILbLCWMbqN0Y1xw2DdY5yFMrbx49gGjA4Kg0FLaRnt2kBbaEtbeuRo7qRJnNtx4tuO5UOypM/vD32/sizLtmTLlhy/n4+HH7G/+kr62LHf+uj9fX/eHzHGoJRSanZwZHoASimlpo8GfaWUmkU06Cul1CyiQV8ppWYRDfpKKTWLaNBXSqlZRIO+UkrNIhr0lVJqFtGgr5RSs4gr0wOIV1FRYerq6jI9DKWUmlG2b9/eZoypHO+8rAv6dXV1bNu2LdPDUEqpGUVETiRznqZ3lFJqFtGgr5RSs4gGfaWUmkU06Cul1CyiQV8ppWYRDfpKKTWLaNBXSqlZRIO+UkpNg817znC8rTfTw9Cgr5RSU+33h1r525++xr/97nCmh6JBfyyb95zhvfdvIV2bx58bGORd33uZR3eeTsvjKaWy37mBQe5+ZDcA2050Zng0GvTH9NS+szx/qJWWHv+kHyscNnzswZ1sOdbB9iz4j1czW3f/II/vbuLuR3bz+0OtmR6OGsOXH99P87kB3nrRPE529NHSM5DR8STVe0dENgHfApzA940xX427vRa4H6gEOoD3GGMardtCwB7r1JPGmFvTNPYp19Dqi/zb4qO6KHdSj/WN3x7imddbEIH+QCgdw1Oz0C+2neKXr51m6/EOguHIO9A2n583Lhu3z5bKgGcPtvDQtkY+cM1iblhZzf/tamL78U5uXj03Y2Mad6YvIk7g28DNwErgDhFZGXfa14AfG2PWAPcA/xRzW78xZq31MWMCfjhsONISuejS0OKb1GP9es8Z/v2ZBm6/ZAH1FQX0DWrQV6kLBMN8+pHdNHb1cdfVi3jkA2/gpgurOdKa+YuDaqTu/khaZ1m1l49dv5RV84rxuBwZT/Ekk97ZCDQYY44aYwLAg8BtceesBH5nff5sgttnnKbufvqt4DyZoH/wbA+f/MUu1i0s4Yu3XUi+26kzfTUh7b1+jIEPvHEJn960nPW1ZSytKuRkRx+BYDjTw1NxvvT4ftp8Ab72zovwuJy4XQ4uqimZEUF/PnAq5utG61isXcAfWZ+/HSgUkXLr61wR2SYir4jI2yY12mlkB3q3yzFq0G/s7OMvf7SV5nOJc3T9gRB3/WQbBR4X//me9XhcTvJzXPQFglM2bnX+avcFACj3uqPHFlUWEAobTnb0ZWpYKoE2n5+HtzfyvivrWVNTEj2+vq6Ufae7MzrxSyboS4Jj8eUsfwe8UUR2AG8ETgN2ZFtojNkAvBv4pogsHvEEIndZLwzbWluz46KUHeivXlrB4VGC/u8OtPDM6y18/alDCW//4UvHONHex7duXxu9JpCnM301QW2+SEFBhdcTPbao0gvA0dbJpSBVeh1utuPH8GstG2pLCYYNuxq7MjEsILmg3wgsiPm6BmiKPcEY02SMeYcxZh3wj9axbvs269+jwHPAuvgnMMbcZ4zZYIzZUFmZHRekGlp8lBW4uaSujDafn+6+wRHn7G7sBuAX209xuLln2G1dfQG++9wRrltexeWLK6LH83Kc9GnQVxNgz/Qr4mb6gOb1s0xDSyQeLKnyDju+vrYUIKMVfMkE/a3AUhGpFxE3cDvwWOwJIlIhIvZj/QORSh5EpFREPPY5wBXA/nQNfio1tPhYUullaXXkP62htWfEOXtPd7N2QQn5bhf/+uTBYbd997kj+PxBPrXpgmHH893O6LUCpVJhz/TLY2b6Rbk5VBZ6dKafZQ63+Cj0uKgu8gw7XpLvZkmVl23HOzI0siSCvjEmCHwIeBI4ADxkjNknIveIiF2Ncw1wUEQOAdXAvdbxFcA2EdlF5ALvV40xWR/0jTE0tPpYXOVlSWUhMPJibn8gxOGWHq5eWsFdVy/iqf3N0Vfvpq5+fvjScd6+bj7L5xQNu5+md7JDZ2+AD/zPdtp9k1+DMV3aewN4XA4K3M5hxxdVFHA0C5b3qyGHm30sqfYiMjI7vqG2lO0nOgmH07PoM1VJLc4yxmw2xiwzxiw2xtxrHfucMeYx6/OHjTFLrXP+yhjjt46/ZIxZbYy5yPr3B1P3raRPe2+Arr5BllR5mV+ahyfBxdz9Z7oJG1g1v5j3XVlPhdfDP//6dYwxfPO3h8DAJ25YNuKx892a3skGW4938Ou9Z9mawRlXqtp8fiq8nhGBZFGllyM6088qh1t8LI1L7djW15ZybiAYXQc03XRFbgJ2gF9S5cXpEBZVekdczN1j5fPX1JRQ4HHx0euWsOV4B//1wlEe3t7In72hlprS/BGPned20T8YytirvIo40x2puGo+N4Nm+r7AsHy+bXFlAV19g3T0BjIwKhWvszdAm8/P0qrChLdvqCsDYNvxzOT1NegnYAd9+5V6SZV3xEx/9+luKryeaM7u9o0LqS3P5yubX6fA7eKDb1qS8LHzrbfmA0Gd7WdSU3c/AGdHKbfNRm0+/7B8vm2xVvBkFXsGv6Q68Uy/rjyf8gI3205k5l2mBv0EGlp8FLidzC2OlFkuqfRyuqt/WC5+7+luVs8vir7VznE6+OSNkYu2d129iLKCkTMyiFTvAJriybAzXfZMf+YE/XZfgPIEv1dDFTwa9LOBXa45WnpHRFhv5fUzIaneO7NNQ0vkIq4d0JdUeTEm8ke1an4xfYEgDS0+Nq0a3j/jrWvmsrAsn9Xzi0d97Dxrpq8XczPrjDXTb5kh6R1jDO29fioKR870a0rzcTsdHNWyzaxwuKWHfLeTecV5o56zoa6Up/Y309rjpzLB/+lU0pl+Ana5ps0u27RnUvubzhE2jAjuIsLaBSU4HYnWs0XY6R0t28ysJmumP1PSO+cGggyGTMKZvtMh1FXka61+lmho8bGkyotjjDiwvjaS18/EbF+DfpyegUHOnhtgccxbs7ryApwOib5t23Pavog7+ox+NHbQ1/RO5oTCJprWmSnpnUSrcWMtqvBytE3TO9ngcLNvxKKseKvmF+F2Odiegby+Bv049mwpNh/ndjmoLcuPXszd09hNZaFnQu2W83IiGTXtv5M5bT4/wbBhXnEuPQPBGfF/kajvTqxFlQWcbO9jMKSN1zLpnDVpHK1yx+ZxObmopjgjzdc06Mex2ynEv1IvrvJGr8rvOd3NmjHy9mPJ15x+xjV1RfL56xZGlsTPhLLN9nFm+osrvQS18VrGxVf+jWVDXRl7M9B8TYN+nIZWH26ng4Vlw2vsl1R5Od7WS3f/IA3WBd2JyNP0TsbZNfprF0S6H86EFE9b7/gzfUAv5mZYQ/PQGp/xbKgtZTA0/c3XNOjHOdLio64iH5dz+I9miTWT+vWeMxgzsXw+DJVs6kw/c+yZ/tqFMyjoW1t2luWPFvSHFxuosf3X80f59rMNaX/cwy09uF0OFpSNXJgZL1PN17RkM05Di4+V84pGHLcreH65I7Kp+VhlmWPR6p3MO9M9QG6OgwvmRPKuM6Fss73XT2l+zojJiK04L4cKr1sXaCVh56kuvvLrAxgDl9aXRVfIpsPhFh+LK71jVvDZSvLdLK3yTnsrEA36MQYGQ5zs6OPWtfF7xAytetxyrIPqIg9VE9wzN99tX8jVoJ8pZ7r7mVecR6HHRb7bOSPKNiMtGMau515U6R03vdPQ0sND2xoxZqgNSHVRLu+7sj5hc7DzTShs+Nyje6nweshxCJ/51V4e//CVo76Ypupwsy86g0/GhroyHt/dRDhsxizxTCdN78Q43t5L2CTOxxV4XMyzVuhOdJYPkJvjsDZHz/6KkfPV6a4B5pXkISJUF+XOiPROuy8waj7ftriyYNz0zneeO8J/vXCUn756kp++epL/fvkEX37iQFpr/I0xw15UsskDW06yu7Gbz9yygs+
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(dt_range_lc1,dt_accuracy_cum1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test et évaluation de la performance de l'arbre de décision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[15.  0.  0.]\n",
      " [ 0. 16.  1.]\n",
      " [ 0.  0. 13.]]\n",
      "\n",
      "Exactitude:\n",
      "0.9777777777777777\n",
      "\n",
      "Précision:\n",
      "[1.0, 1.0, 0.9285714285714286]\n",
      "\n",
      "Rappel:\n",
      "[1.0, 0.9411764705882353, 1.0]\n",
      "\n",
      "Calculé en:\n",
      "0.00035953521728515625s\n"
     ]
    }
   ],
   "source": [
    "dt1_testres = dt1.test(test1, test_labels1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Réseaux de neurones: Choix du nombre de neurones dans la couche cachée\n",
    "\n",
    "On doit identifier le nombre optimal de neurones dans la couche cachée dans l'intervalle $[4,50]$ pour chacun des 5 jeux de données. On itère sur chacun des jeux de validation croisée, pour chaque dimension, puis on calcule l'*accuracy* moyenne. La dimension qui a la meilleure *accuracy* moyenne est choisie pour construire le réseau de neurones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "choix_n_neurones = range(4,51)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pour faire la séparation du jeu de données en $k_{cv}=5$ jeux de validation croisée, on génère une permutation sur les indices du jeu d'entrainement, puis, on sépare cet ensemble en 5 groupes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_cv = 5\n",
    "all_indices = range(len(train_labels1))\n",
    "np.random.seed(12345)\n",
    "indices_cv_test = (\n",
    "    np.sort(np.array_split(np.random.permutation(all_indices),\n",
    "                           k_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_cv_train = (\n",
    "    [np.setdiff1d(all_indices,indices_cv_test[i]) for i in range(k_cv)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ce jeu de données a trois classes possibles comme variable de sortie. On utilisera donc un réseau de neurones avec 3 neurones dans la couche de sortie, une pour chacune des valeurs possibles. Les valeurs de sortie du jeu de données sont transformées à l'aide d'un encodage binaire où la valeur de sortie est convertie en rang dans un vecteur (on commence à 0), prenant la valeur 1. Par exemple, la valeur 2 devient $[0,0,1]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum = []\n",
    "for n_neurones in choix_n_neurones:\n",
    "    accuracy_cv=[]\n",
    "    for cv_set in range(k_cv):\n",
    "        nn1 = NeuralNet.NeuralNet(np.array([4,n_neurones,3]),range(3))\n",
    "        nn1.train(train1[indices_cv_train[cv_set]], \n",
    "                  train_labels1[indices_cv_train[cv_set]], 0.1, 1, \n",
    "                  verbose=False)\n",
    "        _,accuracy,_,_,_ = nn1.test(train1[indices_cv_test[cv_set]], \n",
    "                                    train_labels1[indices_cv_test[cv_set]], \n",
    "                                    verbose=False)\n",
    "        accuracy_cv.append(accuracy)\n",
    "    accuracy_cum.append(np.mean(np.array(accuracy_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5887f2400>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl0W9d16P/vxkCCE0BKHDQPlDXQoyzJiuOptuUkztA4vw6u3TTxFPu9tmmaNm1e0iFJ09dprffS9teXtHESx85LE9d1k8ZN3bqRbEd2EtsaPEqkLImaOY8AJxDDeX8AFwRJgLggQYAC9mctLZEXF9T1NbFxsM8++4gxBqWUUqXBUegLUEoplT8a9JVSqoRo0FdKqRKiQV8ppUqIBn2llCohGvSVUqqEaNBXSqkSokFfKaVKiAZ9pZQqIa5CX8BM9fX1ZsOGDYW+DKWUuqgcOnSozxjTkOm8JRf0N2zYwMGDBwt9GUopdVERkTN2ztP0jlJKlRAN+kopVUI06CulVAnRoK+UUiVEg75SSpUQW0FfRG4XkWMickJEPpPi8XUi8pyIvCoib4jI+5Ie+2z8ecdE5D25vHillFLZyViyKSJO4MvAu4DzwAERecoYczTptD8CnjDG/L2IXAo8DWyIf30XcBmwCtgrIluMMZFc/4copZTKzM5IfzdwwhjTboyZBB4H7phxjgG88a99QEf86zuAx40xQWPMKeBE/OcppVTJGZ+M8MTBcxRym1o7QX81cC7p+/PxY8m+APyaiJwnNsr/rSyei4g8JCIHReRgb2+vzUtXSqmLy7+93sGnn3yDY92Bgl2DnaAvKY7NfJu6G3jUGLMGeB/wf0XEYfO5GGMeNsbsMsbsamjIuIpYKaUuSif7RgAYGgsV7BrstGE4D6xN+n4NU+kbywPA7QDGmJ+JiAeot/lcpZQqCe29owAMjxcu6NsZ6R8ANovIRhEpIzYx+9SMc84CewBEpAXwAL3x8+4SkXIR2QhsBl7J1cUrpdTFpL03NtL3FzDoZxzpG2PCIvJx4BnACTxijDkiIl8EDhpjngI+BXxNRH6HWPrmXhObqTgiIk8AR4Ew8JtauaOUKkXhSJSzA2NAYUf6trpsGmOeJjZBm3zsc0lfHwWuT/PcPwP+bAHXqJRSF73zg+OEIrEpTf9EuGDXoStylVIqD071jSa+LmR6R4O+Ukrlwcl4Pr+m3LW0c/pKKaUW7lTfKL4KN6trK5Z+Tl8ppdTCtPeO0txQhcflXPIlm0oppRboVN8oG+ur8FW48U9o0FdKqaI1GgzT5Z9gU0M13gqXjvSVUqqYWZU7iZH+uJZsKqVU0WqPB/3mhiq8HjfjoQiT4WhBrkWDvlJKLbL23hFEYMPyKnyVboCC5fU16Cul1CI71TfKKl8FHrcTrycW9AuV19egr5RSi8wq1wTwVcRH+hr0lVKq+BhjONU3SnN9LOh7K2LLo3Skr5RSRag3EGQkGKa5oRpIGukXqOmaBn2llFpE7UnlmoDm9JVSqphZu2VZOX2v5vSVUqp4neobodzlYJWvAgCP20m5y6FBXymlilF7b6znjsMhiWPeAvbf0aCvlFKL6FTfVLmmxVfh1py+UkoVm1B8X1xrEtfi9bgK1n9Hg75SSi2ScwNjhKOG5vrqacd1pK+UUkXIqtzZOCO9ozl9pZQqQu19sX1xm+s1p69KWOfweKEvQam8ONU3yrKqMmory6Yd93rc+MdDGGPyfk0a9FVeHesK8M6/eJaX2/sLfSlKLbqTvaOzRvkQG+lHDYwE8z+Zq0Ff5dWJntjH3bc6/AW+EqUWX6pyTShs0zUN+iqvrNROe+9Iga9EqcUVmAjRGwiycUblDiS3V9aRvipyHUMTwFRVg1LF6lTf9J47yQrZdE2Dvsora6RvvSCUKlaJRmspcvqJpmsFKNvUoK/yqmM4NtLv8k8wWoBJLKXypb1vFIfAuuWVsx6z0jtLdqQvIreLyDEROSEin0nx+F+LyGvxP2+LyFDSY5Gkx57K5cWri0/n0HjiF15H+6qYtfeOsKauknKXc9ZjhWyvnDHoi4gT+DLwXuBS4G4RuTT5HGPM7xhjthtjtgN/B3wv6eFx6zFjzAdzeO3qIjMZjtI7EuSdzcuBqc0llCpG6Sp3AGrKXYgs0aAP7AZOGGPajTGTwOPAHXOcfzfw3VxcnCou3f4JjIF3blqOCJzSyVxVpKb2xZ1duQPgcAg15a6CbJloJ+ivBs4lfX8+fmwWEVkPbASeTTrsEZGDIvKSiHxo3leqLnqd8Xz+hvoqVvkqEkvUlSo2Xf4JxiYjs3ruJPMWqBWDy8Y5kuJYurXDdwFPGmMiScfWGWM6RKQZeFZE3jTGnJz2D4g8BDwEsG7dOhuXpC5GVuXOKp+H5oYqzemromV9it2UonLH4qtwL9n0znlgbdL3a4CONOfexYzUjjGmI/53O/A8cPXMJxljHjbG7DLG7GpoaLBxSepiZNXor6ytoLm+ivbe0YL0HlFqsZ3sS91dM1mhmq7ZCfoHgM0islFEyogF9llVOCKyFagDfpZ0rE5EyuNf1wPXA0dzceHq4tM5PE6Nx0V1uYvmhmpGgmF6R4KFviylcu5U7yiVZU5WeD1pz/F6CtNeOWPQN8aEgY8DzwCtwBPGmCMi8kURSa7GuRt43EwfurUAB0XkdeA54C+NMRr0S1TH0ERic2hrJyFdmauKUXvfCBvrqxBJlR2PKdRI305OH2PM08DTM459bsb3X0jxvJ8CVyzg+lQR6fKPs7I2NvKxStlO9Y1ybbyEU6licapvlCtW++Y8x1tRmC0TdUWuypvOoQlWxkf6q3wVlLsc2nhNFZ1gOMK5gbGU7ReS+SrcjIciTIajebqyGA36Ki8mQhH6RydZ5YuN9B0OYWN8MlepYnJuYIyogeaG1DX6lkL139Ggr/Kia3iqcseysV7LNlXxOdmbvrtmskL139Ggr/KiI6lG39LcUMXZgTFCkfx+vFVqMSU2Q8+Q3rHaK+e7Vl+DvsqLzqFUI/1qwlHDuYGxQl2WUjl3qm+EhppyauJBPR2vjvRVMbNW466cMdIHLdtUxaW9dzTjKB/AF98yMd/9dzToq7zoGJ6grtKNxz3VZtaqbihEXv9M/yjRqK4GVrl3qm+UTRny+aAjfVXkOofGE+WaltrKMpZVleW98dq5gTFu+V/P859HuvL676riNzwWon90Mm13zWSa01dFrXN4glW1s5ekNxegbPPNC8NEDZzo0TUCKresAYyd9I7H7aTc5dCgr4pTR4qRPsReHPneTKW10w9MzTMolSvtNss1LYVor6xBXy260WAY/0Q40YIhWXNDNb2BIIE8LlBp7QwAU/39lcqV9r4RXA5h7bLZ++Km4qvIf9M1Dfpq0U310U890of8Tua2dcVH+kMa9FVuneobZd2yStxOe6HV63HpSF8VH2tEnVyuadnUkN+g758IcX5wHKdDEgvGlMoVu+WalthGKlqyqYqMNaJeVTt7pL9ueSUOmVq6vtiOdcVSOzvX1xGYCDMSzH+XQ1WcolEz52boqWhOXxWljuFxRKApxYYS5S4na+oq5+y2ORmO8sv/8FP+9dULC76Wtvgk7q3bGoFYKalSudAxPE4wHM3YaC2Z5vRVUeocmqC+upwyV+pft0z75T71egcHTg/ynZfPLvhaWrsC+Crc7FhXB8QWjSmVC9bvcDbpHa8ntk9uPhcKatBXi65jeHxao7WZrG6bqfbLNcbw9RfaATh4ZoDB0ckFXUtbp59tK2oS8ws60le5km25JsRG+lEDo5P5SzNq0FeLrnN4ghVzBP3mhmrGJiN0+2fvl/vC8T7augJ85Nr1RA08/3bPvK8jGjW0dQVoWellhc+DiI70Ve60945QXe6iobrc9nMK0V5Zg75aVMaYlC0YkjUn9sudndf/2gvtNNaU84fvb6Ghppy9rfMP+ucGxxibjNCysga300FDdbmO9FXOtMcncefaF3cmr9V0LY8VPBr01aLyT4QZnYykbMFgSXTbnJHXP9rh54Xjfdx7/QY8bie3bm1k/7H
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(choix_n_neurones,accuracy_cum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de neurones qui maximise l'*accuracy* est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_neurones_optimal1 = (\n",
    "    choix_n_neurones[np.where(accuracy_cum==max(accuracy_cum))[0][0]])\n",
    "n_neurones_optimal1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choix du nombre de couches cachées\n",
    "\n",
    "On choisit un nombre de 1 à 5 couches cachées. Le nombre de couches ayant l'*accuracy* maximale sera sélectionné pour la construction du réseau. On effectue 10 époques étant donné la taille des réseaux à entrainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "choix_n_couches = range(1,6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum = []\n",
    "lc_cum = []\n",
    "for n_couches in choix_n_couches:\n",
    "    accuracy_cv=[]\n",
    "    nn1 = NeuralNet.NeuralNet(\n",
    "        np.hstack((4,\n",
    "                   np.repeat(n_neurones_optimal1,n_couches),\n",
    "                   3)),\n",
    "        range(3))\n",
    "    lc = nn1.train(train1, train_labels1, 0.1, 10, verbose=False)\n",
    "    lc_cum.append(lc)\n",
    "    _,accuracy,_,_,_ = nn1.test(train1, train_labels1, verbose=False)\n",
    "    accuracy_cv.append(accuracy)\n",
    "    accuracy_cum.append(np.mean(np.array(accuracy_cv)))\n",
    "lc_cum = np.array(lc_cum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L'*accuracy* pour les différentes profondeur est de :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 2.        , 3.        , 4.        , 5.        ],\n",
       "       [0.6952381 , 0.97142857, 0.88571429, 0.93333333, 0.79047619]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.vstack((choix_n_couches,accuracy_cum))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de couches cachées qui maximise l'*accuracy* est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_couches_optimal1 = choix_n_couches[np.where(accuracy_cum==max(accuracy_cum))[0][0]]\n",
    "n_couches_optimal1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Un problème rencontré avec un nombre de couches élevées est appelé le *Vanishing gradient problem*. Dans ces situations, le nombre de multiplications nécessaires ainsi que le fait que les valeurs $\\Delta$ sont suffisamment petites pour que le gradient tende vers 0 et ainsi ne permette pas de mettre les poids à jour. Dans ces situations, l'entrainement va stagner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Courbes d'apprentissage\n",
    "\n",
    "Ce graphique présente les courbes d'apprentissage pour chacun des niveaux de profondeur du réseau"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXd4FFX3xz93N9n03itp9ITeRSlBinSkSFFRpPjaEf1hb6+CXbAB8qJiF0QBQVFCkyI1gSTUkEBClpDes9k2vz82SEvIbrIpyHyehyfLzJ25d8ucuXPuOd8jJElCRkZGRubmQNHUA5CRkZGRaTxkoy8jIyNzEyEbfRkZGZmbCNnoy8jIyNxEyEZfRkZG5iZCNvoyMjIyNxGy0ZeRkZG5iZCNvoyMjMxNhGz0ZWRkZG4ibJp6AFfj7e0thYWFNfUwZGRkZG4oDh48mCtJkk9t7Zqd0Q8LC+PAgQNNPQwZGRmZGwohxFlz2snuHRkZGZmbCNnoy8jIyNxEyEZfRkZG5iZCNvoyMjIyNxGy0ZeRkZG5iZCNvoyMjMxNhGz0ZWRkZG4iZKMv06CkZJcSd+xCUw+j+XB0HRRlNvUoZG5iZKMv06DM/+kIM748wOK4U9z09ZgrCuDHu2HXB009EpmbGNnoyzQYGfnlHDhbQJC7A+/9eZKFvx2/uQ2/OqHqb3zTjkPmpkY2+jINxvojagC+m9mLab1CWbojled/ScJovEkN/0Vjn5UIBl3TjkXmpqXZae/I/HtYl6CmawsPQr0ceW10NM52tizZfppyrYG3x3fARnmTzTnUh0x/9RrIPgYBHZp2PDI3JTfZVSfTWBzPKuZ4VgmjOwUCIIRg/rA2PDWkNT/HZ/LQt4eo1BuaeJSNjDoBAjtXvZZdPDJXkXsKyvMbvBvZ6Ms0COsS1CgVgjtiAq7Y/tCAKF4a2Y5NyRd44MsDVGhvEsNfmgNFGdB+HNi5yUZf5kokCdY+DCuGml43ILLRl7E6kiSxNkFN3yhvvJ3trtl/3y3hvHVnB3al5HLPir0Ua24C//ZFIx/UBQI7XXL1yMgAnNkJGX9D9wdAiAbtSjb6MlbnUHoBmYUV/7h2qmNi9xAWT+5MfHohUz/bS0GZthFH2ASo4wEBAR1Nhv/CUdBpmnpUMs2FHW+Bsx90ubvBu5KNvozVWZugxs5GweD2/tdtN6JDIMvu6cqJCyVMWraH7OJ/sRFUx4N3K7BzMfn1jTrITm7qUck0B9L3QtoO6PMo2Do0eHey0ZexKjqDkQ1HzjOonR/OdrUHhw1s48cX93XnXEEFE5bu4VxBeSOMsglQx19axJUXc2UuZ8fb4OgF3e5rlO5koy9jVXal5JJXpmV0x5pdO1fTJ9Kbrx/oSUGZlglL9pCaU9qAI2wCitVQmmVy6wC4hYCjN2TenEZfZ9Tx9/m/ef3v15n5x0x2q3c39ZCaDnU8pPwJvR8ClVOjdCnH6ctYlXUJalztbejXutb6zFfQJdSD72f15u7/7WXi0j18NaMnbQNcG2iUjczFGf3FGb4Qptc30Uy/Ql/B7szdxKXHsf3cdoq1xdgr7XFVuTL7z9mMjBjJvO7z8LT3bOqhNi473gF7N+g+s9G6lI2+jNWo0BrYlJzFyI6B2NkoLT6+XaArP87pzbTle5m0dA9f3t+DzqEeDTDSRkYdD0IJftGXtgV2htNxoC0HlWPTja0BKaosYvu57cSdjWO3ejcagwZXlSv9Q/ozMGQgfYL6IBAsO7KMz5M+56/Mv3iq+1OMjBiJaOAIlmZBVhIc/xX6zQf7xpvgyEZfxmrEHb9AmdbAqOtE7dRGpI8zP87uzdTle5m2fC/L7+1O70gvK46yCVDHg2/bK417YGeQjCZJhtCeTTc2K3Oh7AJbMrYQlx7HgawDGCQDvo6+jIkaQ2yLWLr6dcVWYXvFMY92eZRh4cN4Zc8rPLfzOdadXseLvV4k1DW0id5FI/HXu6BygV5zGrVbs4y+EGIosAhQAsslSVp41f4WwArAB8gHpkmSdK5qnwFIrGqaLknSKCuNXaaZsTZBjZ+rHT3D62ekQzwdWVU145/++T6WTOvKgDa+VhplIyNJkHkI2gy/cvs/i7mHbnijn1qUypb0LcSdjSMpLwmAMNcwprefTmxoLO2926MQ118+bOnRkpXDVrLqxCo+OPQB49aNY07HOdzb/t5rbhL/CnJOQvLP0PcJcGjcp9lajb4QQgl8DNwOnAP2CyHWSZJ09LJm7wArJUn6UggxEFgAXAw4rZAkqZOVxy3TzCgq17HtRDb39g5Dqaj/o7mfqz0/zO7NPSv2MuurA3wwqTPDOwTUfmBzozAdKvIhsDPlWj0bjpynT5Q3Qe4B4BJwQ/r1JUkiOS+ZuPQ44tLjSCtKAyDaK5rHujzGwNCBRLhF1HqeEo2Ov07lMrS9PwqFQCEUTGozif4h/Vm4byGLDi1iY9pGXur9Eh19Ojb022pc/nrXFJ7Z+6FG79qcmX4PIEWSpFQAIcT3wGjgcqPfDnii6vVW4BdrDlKm+fNb0nl0BonRnYKsdk5PJxXfzuzFjC/288h3hyjXdmBCtxCrnb9RqDLq353z5q3ftlBQrqNTiDs/PdgH5Q20mKsz6jh44SBxZ+PYkrGF7PJslEJJN79u3NX6LgaGDsTf6fp5GVfz4tpkfo7P5L2JHRnXJfif7X5Ofrw/4H22pG/h9b2vc/fGu7mrzV082vlRnFXO1n5rjU9+KiSugl4PgpN3o3dvjtEPAjIu+/854Orn0cPAnZhcQGMBFyGElyRJeYC9EOIAoAcWSpIk3xD+haxNUBPh7UR0kHUXpFztbfny/h7M/uogT60+QrnWwL19wqzaR0ORXazh9M4/6SYpeWmvxK1tPGgf5MbiuFN8/fdZ7g3sAid+A01xoy7kmUuFvoLd6t1sSd/Ctoxt/0Tc9AnsQ2yXWG4Lug13e/c6nXvHyRx+js9EZaPg/c0nGdEhEJXNlS6ggaED6eHfgw/jP+S7498Rlx7Hsz2fJTY01hpvr+nY+T4obKDPI03SvTlGv7pn9asVgeYBHwkhpgM7gExMRh4gVJIktRAiAtgihEiUJOn0FR0IMQuYBRAa+i9fvPkXklWk4e+0PB6PbdUgUReOKhuW39uNR7+L56V1yZRW6nloQJTV+7EWZ/PKWLI9lZ8OnuMLZTznHaNY9+BA2vi7IkkSCRmFvPX7cUaMa4cXEpw/DOG3NvWwAVPEzY5zO4hLj2NX5i40Bg0uKhf6B/cnNjSW3oG9cbStX7RRuVbPc78kEuHjxPyhbZj11UG+35/OPb3DrmnrrHLmmZ7PMDxiOK/seYXHtz5ObGgsz/R4Bj8nv3qNo0kozICE76DrdHCx7MnIWphj9M8Blz9TBwPqyxtIkqQGxgEIIZyBOyVJKrpsH5IkpQohtgGdgdNXHb8MWAbQrVu3m7TCxo3Lr0fUSBL1itqpDTsbJR9P6cJTq4/w9qYTlFbqeXpI62YV2nfsfDGfbjvNr0fU2CgUTOgaSK/jGSiix4O/aSYvhOD1MdHc/v52Xjuk4gMwuXia0OhfL+JmYOhAuvl3s+pi6qLNp8jIr+CHWb3oEe5Jj3BPFselML5rMI6q6k1SB58OfD/ie1Ymr+TTw58yeu1oHuvyGBNbTUSpsDw8uMnYtcj0t+/jTTYEc4z+fqClECIc0wz+LmDK5Q2EEN5AviRJRuAZTJE8CCE8gHJJkiqr2twCvGXF8cs0A9YmqOkY7Ea4d8NmFNooFbw7oSOOKiWfbjtNeaWel0a2R2GFheP6cPBsPh9vPc2W49k4qZTMvDWCGX3D8dVlwpHiS5E6mGbS2BQzo78Ln2xL5T/ugTic2wPFYxt1zOX6cv7K/Ist6VtIzDUF11kacVMXkjKLWL4zjck9QugZYYry+r+hrbnz0z18vuvMdZ/gbBW2zIiZweAWg3n171d5Y+8b/Jr6Ky/1folWHq2sPlarU5IFh1ZCpyngFlx7+waiVqMvSZJeCPEwsAlTyOYKSZKShRCvAgckSVoH9AcWCCEkTO6di0vSbYGlQggjJsm
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "for i in choix_n_couches:\n",
    "    plt.plot(range(10),lc_cum[i-1], label=\"%d couches\"%(i,))\n",
    "leg = plt.legend(loc='best', ncol=2, mode=\"expand\", shadow=True, fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choix des poids initiaux\n",
    "Les poids initiaux ont été choisis à partir d'une distribution uniforme sur $[-1,1]$. On compare ici les courbes d'apprentissage en initialisant les poids à 0 et en initialisant les poids aléatoirement, pour le réseau de dimension et de profondeur optimale sélectionnées précédemment.\n",
    "\n",
    "- Réseau initialisé avec les poids à 0 $RN_{0}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn1_poidszero = NeuralNet.NeuralNet(\n",
    "    np.hstack((4,\n",
    "               np.repeat(n_neurones_optimal1,n_couches_optimal1),\n",
    "               3)),\n",
    "    range(3),\n",
    "    input_weights=0)\n",
    "lc_nn1_poidszero = (\n",
    "    nn1_poidszero.train(train1, train_labels1, 0.1, 10, verbose=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Réseau initialisé avec les poids uniformes $RN_{\\neg{0}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn1_poidsunif = NeuralNet.NeuralNet(\n",
    "    np.hstack((4,\n",
    "               np.repeat(n_neurones_optimal1,n_couches_optimal1),\n",
    "               3)),\n",
    "    range(3))\n",
    "np.random.seed(12345)\n",
    "start_time = time.time()\n",
    "lc_nn1_poidsunif = (\n",
    "    nn1_poidsunif.train(train1, train_labels1, 0.1, 10, verbose=False))\n",
    "nn1_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graphique des courbes d'apprentissage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VfWd//HXJyFhR7YguywisgnIVek4ti61BVvBKlhQq9YW2vmVLozz69Dq2N/IqDzGrU7H5YcVxWXArSpaRrvYqbUtNoksCghGRAkICbIYlpDtM398Q0ggkJtww0lO3s/H4z6459zvPfeTS+473/s953yPuTsiIhIvaVEXICIiqadwFxGJIYW7iEgMKdxFRGJI4S4iEkMKdxGRGFK4i4jEkMJdRCSGFO4iIjHUKqoX7t69uw8YMCCqlxcRaZZyc3O3u3tWXe0iC/cBAwaQk5MT1cuLiDRLZvZRMu00LCMiEkMKdxGRGKoz3M1sgZkVmNm7R3nczOw/zCzPzFaZ2ZmpL1NEROojmTH3x4D/BB4/yuMTgSGVt3OAByv/rbfy8nJ27NhBaWlpQ54uIhIbGRkZdO3alfT09AY9v85wd/c3zGzAMZpMBh73MDH8MjPrbGa93P2T+hazY8cO2rRpQ/fu3SkvL6e8vLy+mxARafbcnX379lFYWEjPnj0btI1UHC3TB9hUbTm/ct0R4W5mM4GZAP379z9iQ6WlpXTr1o2tW7fy0UcfHXxOCkoUEWle3J09e/ZQUFDAGWecUe/npyLca0vfWi/v5O7zgfkAiUSi1jZ79+5l48aNtGvXjrQ07e8VkZarpKSE119/nV69epGVVeeh7TWkIj3zgX7VlvsCWxq6sZKSEsxMwS4i8eMVsONDKDuQVPO0tDTMjKKionq/VCoSdAlwbeVRM+OB3Q0Zbz9I13QVkVgq2QNbVsBnm2H/jkZ/uTqHZcxsEXA+0N3M8oGfARkA7v4QsBS4BMgD9gHfbKxiT4QxY8YwZMgQysrK6NOnD7fffjudOnVi8+bNTJw4kTlz5nDVVVcBcPvttzNixAgmT558xHbKy8uZNm1ajXXbtm3jnHPO4c477zwhP4tIU5WqzxnAzTffzLJly1i6dCmZmZns3LmT6dOn8+qrr6ak1q1bt/L973+/xrr8/HyuvPJKZs+eXfcG3OGzfNj5MaRnwMkjoG2XqofvuusuzjvvPM45p0EHGR5VnT13d5/u7r3cPcPd+7r7I+7+UGWw48H33H2wu49y92Y9p0Dr1q159tlneeGFFzjppJNYvHhx1WNdu3blqaeeSupQzfT0dJ599tmq2wMPPEDr1q2ZOXNm0rWUlZU16GcQaepS9Tk7KC0tjRdeeKExSqVnz541Psu33XYbHTp04Jprrqn7yWXFsPUdyrd/CO26Qe+xNYId4KqrrmLBggUprzuyuWXq8nDOTjbsLEnpNgd1yWRGokvdDSuNHj2a9evXVy136dKFsWPH8tJLLzFlypSkt+Pu3HzzzVx//fUMGTIEgDVr1nDnnXeyb98+unTpwty5c8nKyuKGG25g9OjRrFixgvPPP5+LL76YW265hZ07d1a169WrV/I/tMgx9Fp5H212vZ/SbRZ3HsIno3+YdPtUfM6uueYannzySa644ooa692de+65hzfffBMzY+bMmUyYMIHs7GwefPBBOnfuTF5eHsOHD+eOO+6o8+i8AwcO8JOf/ISf/vSnVTs4//KXv/DAAw9QUlJCv379mDt3Lu3atWPCly7msgvP5q8r1jDt69MYOKwHc2dfT3FxMf369ePWW2+lU6dO9O7dm127drF9+3a6d++e9PtWF+21PIry8nLeeustzj///Brrb7jhBh5//PF6HYP/xBNPkJ6eXvU1s7S0lDvuuIO7776bp59+mssuu4xf/OIXVe2Liop49NFHue6667j99tu59NJLef755/nKV77CvHnzUvLziTQFqfqc9ezZk7Fjx/LKK6/UWP+73/2OdevW8dxzz/Hwww9zzz33UFhYCMB7773Hj3/8Y1588UXy8/NZvnx5na9z7733MmbMGC644AIAdu7cyfz585k/fz7PPPMMI0aM4PHHHoWCtVBRSma79ix8cjETv/Z1brrpJmbPns3zzz/PkCFDePDBB6u2O2zYsKRevz6abM+9Pj3sVDpw4ABTp05ly5YtDBs2jM997nM1Hu/bty8jR45k6dKlSW1v3bp1PPnkkyxatKiqV7Bx40by8vL4zne+A4Rf8OqHOU2YMKHq/qpVq7j33nsB+OpXv1p1XyQV6tPDTqVUf84Avv3tb/ODH/yA8847r2rd8uXLmThxIunp6XTr1o1EIsG7775Lhw4dGDlyZNUJQkOHDmXLli2ceebRZ0/505/+xLJly1i0aFHVulWrVrFhwwauu+46AEoPFDN6SF/YPwbSWjHhiuugVWuKioooKioikUgAMGnSJG688caq7XTt2rXqj06qNNlwj8rBscCioiJmzZrF4sWLufrqq2u0mTFjBv/4j//IuHHjjrmt4uJi5syZw0033US3bt2q1rs7gwcP5sknn6z1eW3btj3qNnVSl8RBKj9nB/Xv35+hQ4fym9/8pmrdsY6+y8zMrLqfnp5+zG8JO3bsYO7cudx33301Pp/uzvjx4/n3eXfAzg+haCtktIOsoZDWirbt2iVVe0lJCa1bt06qbbI0LHMUHTt2ZM6cOSxcuPCIHTsDBw5k8ODBvPHGG8fcxt13300ikeALX/jCEc/fuXMnK1euBMIwTV5eXq3bGD16dNVe/6VLlzJ27NiG/kgiTU4qPmfVzZgxg4ULF1Ytjxs3jtdee61q3qrc3FxGjRpV7zpvueUWpk+fzrBhw2qsP+OMM1ixfDkf574GRVvZn9mdjSVdILP9ET9np06dyM3NBeDll1+u6sVD+DZ/6qmn1ruuY1HP/RiGDRvGaaedxquvvnrE17UZM2Zw5ZVXHvW5BQUFPP300wwcOJCpU6dWrR88eDDz5s3j7rvvZt68eezZs4fy8nKuvvrqWv9z58yZwy233MJjjz1WtUNVjqKiHHZuhLRW4cOV2Q5atQV922nSjudzdrhTTz2VYcOGsXbtWgAuuugiVq5cyZQpUzAzZs+eTffu3fnwww+T3ubKlSt544032Lp1a41hovHjx3Pjt6Yyd9Y0/nneQ5R4Glg6s2bNYsDAgUds59/+7d+YO3cuxcXF9O3bt+qzXFpayqZNmxgxYkTSNSXDojppKJFI+OFXYtqyZQuZmZnk5eXRvn37ozxT5Cg++wR2fFBznaVBRlvIqAz7zPbha3Or1H4FlhamdD9sXw8HiqB9FnQbHDoVDfD73/+etWvXMmvWrCMe2717N7m5uVx66aUMGjQIADPLdffEEY0Po567xEfRJ5DZAXqdASX7oHQflOyF0r1QvAv2Fhxqm9YqhHxmu5rB38APqLQgRVthx4bQccgaGsL9OJSXl3PttdemqLhD9JucArfddhsrVqyose7qq6/msssui6iiFqj4sxDm3U4NH7rWHcKtuvLSysDfFwK/ZB/s3Q4VWw+1Sc881Ls/OLST0S5sUyJ1Ij5nP/rRj9i8efMR684991woL4HteWHqgDadofuQlHwD/NKXvnTc26hNkwv35ji3zE033RR1CVL0CaSlH7sXlZ4B6SdBm5Nqri87cKiXX7I33C/eFU4bB8Ago021wD84tNNG4/kn0In4nP385z+v/YF9n4Zg93LoOgg69mr0/3t3P648bFLhnpGRQXFxcdRlSHNTXgr7tkOHniHg66tV63Crflq4O5Ttrwz8akM8+z491MbSah/aSc888jWkeaooD0Mwe7aF/9vuQ8P/cyNzdyoqKti/f3+Dt9Gkwr1r166sX7+eoqIizasiydtbCMUOHTrC7t0p3ngmpGVC687QmjBla9mBMGdIWTGUFMO+HVB+2Hh+qzbhD0ZGm8r7bTS009yU7oPdm6CsFDpkQfuTYX8p7E/179iR3J39+/ezY8cOzKxBl9prUuGenp5Onz59+OMf/0hZWVnKD+qXGHKHlU9BZkc
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "plt.plot(range(10),lc_nn1_poidszero, label=\"RN_Zero\")\n",
    "plt.plot(range(10),lc_nn1_poidsunif, label=\"RN_Non_Zero)\")\n",
    "leg = plt.legend(loc='best', \n",
    "                 ncol=2, \n",
    "                 mode=\"expand\", \n",
    "                 shadow=True, \n",
    "                 fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque clairement que l'initialisation des poids à 0 ne permet pas de démarrer l'entrainement du réseau. Il est donc nécessaire d'initialiser les poids aléatoirement pour permettre l'apprentissage et éviter que les différentes fonctions pour calculer la rétropropagation des erreurs aient toutes des valeurs de 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entrainement et tests\n",
    "On reprend les résultats du dernier entrainement, puisqu'il utilise les poids aléatoires et les hyperparamètres optimaux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[15.  0.  0.]\n",
      " [ 0. 16.  1.]\n",
      " [ 0.  3. 10.]]\n",
      "\n",
      "Exactitude:\n",
      "0.9111111111111111\n",
      "\n",
      "Précision:\n",
      "[1.0, 0.8421052631578947, 0.9090909090909091]\n",
      "\n",
      "Rappel:\n",
      "[1.0, 0.9411764705882353, 0.7692307692307693]\n",
      "\n",
      "Calculé en:\n",
      "0.0332188606262207s\n"
     ]
    }
   ],
   "source": [
    "res_test1 = nn1_poidsunif.test(test1, test_labels1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MONKS Dataset\n",
    "\n",
    "- Chargement des jeux de données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "train2, train_labels2, test2, test_labels2 = (\n",
    "    ld.load_monks_dataset(1))\n",
    "train3, train_labels3, test3, test_labels3 = (\n",
    "    ld.load_monks_dataset(2))\n",
    "train4, train_labels4, test4, test_labels4 = (\n",
    "    ld.load_monks_dataset(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entrainement de l'arbre de décision\n",
    "\n",
    "Dans cette section, on entraine un arbre de décision basé sur la mesure d'entropie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt2 = DecisionTree.DecisionTree(attribute_type=\"discrete\")\n",
    "dt3 = DecisionTree.DecisionTree(attribute_type=\"discrete\")\n",
    "dt4 = DecisionTree.DecisionTree(attribute_type=\"discrete\")\n",
    "\n",
    "start_time = time.time()\n",
    "_ = dt2.train(train2, train_labels2,verbose=False)\n",
    "dt2_compute_time = time.time() - start_time\n",
    "start_time = time.time()\n",
    "_ = dt3.train(train3, train_labels3,verbose=False)\n",
    "dt3_compute_time = time.time() - start_time\n",
    "start_time = time.time()\n",
    "_ = dt4.train(train4, train_labels4,verbose=False)\n",
    "dt4_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Courbe d'apprentissage\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt_range_lc2,dt_accuracy_cum2 = courbe_apprentissage_dt(dt2,train2,train_labels2)\n",
    "dt_range_lc3,dt_accuracy_cum3 = courbe_apprentissage_dt(dt3,train3,train_labels3)\n",
    "dt_range_lc4,dt_accuracy_cum4 = courbe_apprentissage_dt(dt4,train4,train_labels4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voici les graphiques de la courbe d'apprentissage pour les 3 jeux de données\n",
    "- 1er jeu de données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa588631400>]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl421eZ6PHvkSxZsi15XxLvSZx9aZo0XWkL3Sm07LQD7bAMZeBCGWCGC3MvAwNDh21gWHpnWlq2spTCsIQSWrpB27RJk5BmX5w43vdN3i1bOvcP6SfLtmzLthZLfj/P06e2/LN0LDuvjt7znvcorTVCCCGSiyneAxBCCBF5EtyFECIJSXAXQogkJMFdCCGSkAR3IYRIQhLchRAiCUlwF0KIJCTBXQghkpAEdyGESEIp8XrgvLw8XVFREa+HF0KIhHTo0KFOrXX+XNfFLbhXVFRw8ODBeD28EEIkJKVUXTjXSVpGCCGSkAR3IYRIQhLchRAiCUlwF0KIJCTBXQghkpAEdyGESEIS3IUQIglJcI8SrTV/PNZCXddgvIcihFiGJLhHwciYh08+doQP/fSvfOkPp+I9HCHEMhS3HarJqq1vhHseOcSRhl5Kc+zsq+nC49WYTSreQxNCLCMyc48g1/AYt393L9Vt/fz3u3fwTzetp29knONNrngPTQixzEhwj6ATTS5a+0b45jsv4ubNRVyxOheAF891xnlkQojlRoJ7BDX1DgOwrtABQF5GKuuLHLx0XoK7ECK2JLhHUItrBICiTFvgtivX5HGgtoeRMU+8hiWEWIYkuEdQi2uYvAwrNos5cNtVa/Jwj3s5VNcTx5EJIZYbCe4R1NQ7wopM+6TbdlXmkGJSkncXQsSUBPcIaukdZkVQSgYgPTWF7WVZvCTBXQgRQxLcI0RrTXPvMCuz7NO+dsXqPI42uXANjcVhZEKI5UiCe4T0jYwz6PawMss27WtXVeWhNbxc08XouIcfv1zLxx49LIusQoiokR2qEdLi8pVBhpq5byvJIs1q5oHnz/PFx08GSibfvL2Ya9cVxHScQojlQWbuEdLS6yuDnLqgCmBNMXHZqlwO1/eS50jlwbt2kGJS7L/QHethCiGWCZm5R4gxGw+VlgH44ps284GuIS5blYNSii0lmbwiwV0IESUyc4+QFtcwZpOiwBE6uBdn2bl8dS5K+RqIXVqZy9HGXobdkncXQkSeBPcIae4dochpC7v746WVOYx5NIfrZXOTECLyJLhHSHOIGvfZ7KjIxqSQvLsQIiokuEdIi2skZKXMTJw2CxtXOtl/oSuKo0pMg6Pj/O7VJrTW8R6KEAlLgnsEeL2aFtcwK2ZYTJ3JpZW+CprRccm7B3v8aDMfe/RVzrYNxHsoQiQsCe4R0Dk4yphHUzyPmTv4+s6Mjns51iiHeQQzumvWdEhwF2KhJLhHwGw17rO5pCIHkLz7VG19owDUdMrh4kIsVFjBXSl1s1LqjFLqnFLq0yG+XqaUek4pdVgpdVQp9frID3XpavbXuM9nQRUgJ93KukKHBPcpOvp9L5YXJLgLsWBzBnellBm4H7gF2AjcqZTaOOWy/ws8prXeDtwB/L9ID3Qpa/anEeablgFfauZQbTfjHm+khxU3u48089iBhgV/f2DmLmkZIRYsnJn7LuCc1rpGa+0GHgVun3KNBpz+jzOB5sgNcelr6R3GZjGRlWaZ9/deuiqHQbeHE819URhZfHznmWo+t/sEruGFdcFs65OZuxCLFU5wLwaCp2GN/tuCfR54t1KqEdgDfDTUHSml7lFKHVRKHezo6FjAcJemZpev1a+x+3Q+dlX68u7JcpjHkHuc8x0DDI95+NWhxnl/v8er6RwYJSM1hZ6hMXoG3VEYpRDJL5zgHipiTS1AvhP4oda6BHg98IhSatp9a60f1Frv1FrvzM/Pn/9ol6jm3hFWznMx1VDgsLG1JJNnTrVFeFTxcaqlD68Gu8XMT/bV4fXOr1a9a2AUr4adFdkAXOiS2bsQCxFOcG8ESoM+L2F62uX9wGMAWuuXARuQF4kBJoL57k6d6voNhRxu6KWjfzSCo4qP402+9NLHrq/iQufgvN+RtPufg8tW5QJwoUOCuxALEU5wPwBUKaUqlVJWfAumu6dcUw9cB6CU2oAvuCdP3mUW7nEvHQOj89qdOtX1GwrRGp49nfiz92NNLvIyrLz3ygpy0638+OW6eX2/kW/fUZ5NiklR0ymLqkIsxJzBXWs9DnwEeBI4ha8q5oRS6gtKqdv8l30S+IBS6gjwc+A9epnsHW/rG0HrmVv9hmPDCgfFWXaeOtkewZHFx/EmF5tWZpKaYuaOXaU8e7qNxp6hsL/fqJQpzrJTlpMmi6pCLFBYde5a6z1a67Va69Va6y/5b/sXrfVu/8cntdZXaq23aa0v0lr/KZqDXkomatwXPnNXSnH9hgJePNeR0C2AR8Y8VLcPsKU4E4C/ubQcgJ/urw/7Ptr9Ne55GalU5qVTI2kZIRZEdqgukrFVfjFpGYDrNxYyMuZlbwJXzZxu7cfj1Wwu9lXFFmfZuX5DIb840BD2ebFtfaPkpluxppiozEuntmtw3ouyQggJ7os21wlM4bq0MpeM1BSeTuCqmeNNvh45m1ZmBm67+/IKugfd7DnWEtZ9dPSPUOD0PZeV+emMjHlp9efhhRDhk+C+SHVdg+Q7UkmzLu7EQmuKiWvW5fP0qfaEnameaHaRlWahJHviXcyVa3JZlZ8e9sJqW98oBY5UACrz0gEkNSPEAkhwX6TariEqctMicl83bCikc2CUI429Ebm/WDvW5GLzysxJm7mUUtx1WTmvNvRyNIyfq71/hEKnL7ivyssA4IJUzAgxbxLcF6m+a4iynPSI3Ne16/IxmxRPnUy81Ix73MuZ1n42FTunfe2tO0pIs5rnnL17vJqO/tHAObSFzlTSrGbpDinEAkhwX4Rht4fWvpGIzdyz0qxcUpGdkHn3s239jHl0oFImmNNm4c3bi/n9keZZ2wkYu1ONmbtSisq8dCmHFGIBJLgvQn23r367PC8yM3fwbWg62zZAfVf4teFLgbGYunnl9OAOvoXV0XEvjx2cuVuksTs13zGxOC3BXYiFkeC+CHX+viflOZGZuQPcsLEQIOFm78ebXThsKZTP8C5mXZGDXZU5/GR/HZ4ZFoyN3anGzB1gVV46Dd1DchShEPMkwX0R6vyz64rcyM3cy3PTWVuYkXjBvamPTSuds3bGvPvychq6h3n2dOiduMbMvdAZNHPPT8eroaE7sd7JCBFvEtwXobZrkKw0C5kL6OM+m+s3FLL/QjeuoYX1Q4+1MY+XUy19M6ZkDDdtKqI0x853nztHqO4Uxsw9LyN45u6rmDkv5ZBCzIsE90Wo7x6iPIKzdsP1GwvxeDV/PpsYvWZONPcxOu7l4vLsWa+zmE186Jo1HGno5YXq6Ttx2/sndqca1hRkYLOYeG6G2b5hZMzDYwcaZkz5CLHcSHBfhNquwYjm2w0XlWSRl2Hl6VOJEdwP1vrOgN05R3AHeOuOYlZk2vjOs9XTZu/tfSPkO1In3ZaemsJbLi7h14eb6BqYuSXyA3+p4VP/c5SXzidu+wYhIkmC+wK5x7009QxHrAwymMmkuG59IX8+0457fOmfrfrKhW7Kc9MCbQNmk5pi5u+vWc2B2h721Uw+GLytb3RSvt3wvisrcI97+dkMDchcQ2M89GINANVtsuFJCJDgvmBNvcN4NVFJy4AvNdM/Ms6B2u65L44jrTUH63rYWZ4T9ve885JS8h2pfOfZ6km3t/ePBFoPBFtT4OCatfn8eF9dyBe7771QQ//IOKkpJqrbJbgLARLcF6zWKIOMwswd4Ko1eaSmmJb8btWazkG6B93sqpw7JWOwWczc85pVvHS+i0N1vhcvY3dqqJk7wPuuqqSjf5THj04+BKxrYJTv773ArVtXsLUkk/MS3IUAJLgvWF2nEdyjM3O3W828piqPp062hawsWSoC+faK8GfuAO+6rIzcdCtfeeIMWmu6BifvTp3q6qo81hRk8PCLFyY9Hw8
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(dt_range_lc2,dt_accuracy_cum2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 2e jeu de données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5885cb630>]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXd4W+d59/+5sQFuipRISqK2LEuyLMuybMczTmzLdmKnTZs4SX+J06ZO2je7TV+7I8N9+3alSZs2b1tnt0ntjCaNndhxvPeQZNmalrUXJXFPbOD5/XEGD0CABElQAoHnc128CBycAzw8BL64z/e57/sRpRQajUajqQxc53oAGo1Gozl7aNHXaDSaCkKLvkaj0VQQWvQ1Go2mgtCir9FoNBWEFn2NRqOpILToazQaTQWhRV+j0WgqCC36Go1GU0F4zvUAsmlqalKLFy8+18PQaDSaWcW2bdu6lVLNE+1XcqK/ePFitm7deq6HodFoNLMKETlayH7a3tFoNJoKQou+RqPRVBBa9DUajaaC0KKv0Wg0FYQWfY1Go6kgChJ9EdksIvtE5ICI3JXj8XYReVJEtovIDhG52fHY3eZx+0TkxmIOXqPRaDSTY8KUTRFxA18HrgdOAFtE5AGl1B7Hbn8O/Egp9a8ishp4CFhs3r4dWAO0AY+JyEqlVKrYf4hGo9FoJqaQSH8TcEApdUgpFQfuB27L2kcBtebtOqDDvH0bcL9SKqaUOgwcMJ9vRnnhYDcHu4Zn+mU0Go1m1lGI6M8HjjvunzC3Ofki8DsicgIjyv/EJI5FRO4Uka0isrWrq6vAoefncz/ewb8+dXDaz6PRaDTlRiGiLzm2Za+m/j7gu0qpBcDNwH+KiKvAY1FK3auU2qiU2tjcPGEV8YSE40kice0gaTQaTTaFtGE4ASx03F/AqH1j8XvAZgCl1IsiEgCaCjy26EQTaaIJLfoajUaTTSGR/hZghYgsEREfxsTsA1n7HAPeBiAi5wMBoMvc73YR8YvIEmAF8EqxBp8LpRSxZIpYMj2TL6PRaDSzkgkjfaVUUkQ+DjwCuIFvK6V2i8g9wFal1APAHwHfEJHPYNg3dyilFLBbRH4E7AGSwP+a6cydREqRVhBL6khfo9Fosimoy6ZS6iGMCVrnts87bu8Brshz7F8BfzWNMU6KqCn20YSO9DUajSabsqvIjZliryN9jUajGUvZib41gas9fY1GoxlL2Yl+zLZ3dKSv0Wg02ZSd6Edte0dH+hqNRpNN2Ym+FenH9ESuRqPRjKHsRN+K9KPJFEbWqEaj0Wgsyk70rUhfKSNnX6PRaDSjlJ3oO/PzddqmRqPRZFKGop9y3Na+vkaj0TgpO9F3Zu3oSF+j0WgyKTvRd0b6Om1To9FoMilD0U87butIX6PRaJyUneg7LR0d6Ws0Gk0mZSf6Gdk7eiJXo9FoMihD0Xdk7+iJXI1Go8mg7EQ/I3tHR/oajUaTQfmJfkb2Tmak39Ef4cavPsOZwejZHpZGo9GUBOUn+sk0Qa/buJ0V6e/vHGbfmSEOdA6fi6FpNBrNOafsRD+aSFEX9AJjI/1EUq+qpdFoKpvyE/2kU/QzI/1EyhR97fVrNJoKpexEP5ZI26KfXZwVT422XdZoNJpKpOxEP5pMUR3wIJIr0jdaLetIX6PRVCrlJ/qJNAGvi4DHnd/e0ZW6Go2mQik70Y8lUwQ8bvxe1xh7xxJ93ZNHo9FUKgWJvohsFpF9InJARO7K8fhXReQ18+dNEel3PJZyPPZAMQefi2gijd/rxu9xjbFx4kkd6Ws0msrGM9EOIuIGvg5cD5wAtojIA0qpPdY+SqnPOPb/BHCR4ykiSqn1xRvy+EQTKfweFwGve0xqZjylUzY1Gk1lU0ikvwk4oJQ6pJSKA/cDt42z//uA+4oxuKkQS6YJmJF+9spZiaQxkatX1NJoNJVKIaI/HzjuuH/C3DYGEVkELAGecGwOiMhWEXlJRN415ZEWQDqtiCeNiVy/Z2ykn9CRvkajqXAmtHcAybFN5dn3duAnSimnqrYrpTpEZCnwhIjsVEodzHgBkTuBOwHa29sLGFJuLK/e73ET8LryZu/oSF+j0VQqhUT6J4CFjvsLgI48+95OlrWjlOowfx8CniLT77f2uVcptVEptbG5ubmAIeXGiuCtSD9fcZaeyNVoNJVKIaK/BVghIktExIch7GOycETkPKABeNGxrUFE/ObtJuAKYE/2scXCiuAtTz9/GwZt72g0mspkQntHKZUUkY8DjwBu4NtKqd0icg+wVSllfQG8D7hfKeW0fs4H/l1E0hhfMH/jzPopNlZkP5q9k2ciV0f6Go2mQinE00cp9RDwUNa2z2fd/2KO414ALpjG+CaFJfKj2Tt5JnJ1pK/RaCqUsqrItUQ+4HXhzzGRqz19jUZT6ZSl6Ps9biNlM3siN6nbMGg0msqmrER/1N4xIv1s796yd+I60tdoNBVKWYl+dqQfT6ZxzivbrZW16Gs0mgqlvETfEekHvMaf5hT4uO6yqdFoKpyyEv1YVqRvbBsVfd1PX6PRVDplJfrRrJRNyOyzo3vvaDSaSqesRN+O9L1GcRZkRvVWcVYipUil87UP0mg0mvKlvETfivQ9o5G+07+3In1jXx3tazSayqOsRD+aSOES8LrFYe+MncgFvTi6RqOpTMpO9P0eNyLisHcyI30xG0VHdaSv0WgqkLIS/Zi5gArgsHcckX4yTbXPaDekI32NRlOJlJXoRxMpO8L354z0FTUBU/R12qZGo6lAykz003aEbxdnJTI9/WpT9HWBlkajqUTKSvRjSUekbxZnWd69UopEKk21X0f6Go2mcikr0Y8m0ratY2fvmJF+Kq1QCmoCXmO7nsjVaDQVSJmJfsph72QWZ1nN1kbtHR3pazSayqOsRN/I3smM9C3v3srRr7UncnWkr9FoKo+yEv1oIkXAk5myORrpG78te0dH+hqNphIpK9GPJUc9fY/bhccldkRvif7oRK6O9DUaTeVRXqLviPQBc3F0M9I3m63Zoq8jfY1GU4GUlehHHZ4+GAVaVkQft+0dcyJXR/oajaYCKS/Rd2TvAAQ8Ljuit9bF1ZG+RqOpZMpK9GM5Iv1o1kSuz+PC53Hp4iyNRlORlI3oJ1JpUmllt18Aw9O3FlaxRN/rdplev7Z3NBpN5VGQ6IvIZhHZJyIHROSuHI9/VUReM3/eFJF+x2MfEpH95s+Hijl4J1HH+rgWhqdv2jsO0Q84tms0Gk0l4ZloBxFxA18HrgdOAFtE5AGl1B5rH6XUZxz7fwK4yLzdCHwB2AgoYJt5bF9R/wocq2Z5s7N3rEjfyN7xeYwFVnTKpkajqUQKifQ3AQeUUoeUUnHgfuC2cfZ/H3CfeftG4FGlVK8p9I8Cm6cz4HzUB7089tmrece6Nnub3+HdJ5KZ9o6eyNVoNJVIIaI/HzjuuH/C3DYGEVkELAGemOyx08XjdrF8bg0NVT57W8DrdkT6oxO5AUcq50Qopfjsj15j29He4g9ao9FozjKFiL7k2Kby7Hs78BOllKWoBR0rIneKyFYR2drV1VXAkAoj6BD9eNZEbqGe/lAsyU9fPckLB3qKNi6NRqM5VxQi+ieAhY77C4COPPvezqi1U/CxSql7lVIblVIbm5ubCxhSYVT53YzEszx9twu/x11w9s5QNAlkLqqu0Wg0s5VCRH8LsEJEloiID0PYH8jeSUTOAxqAFx2bHwFuEJEGEWkAbjC3nRVCPg+R+NiUzYC38Eh/WIu+RqMpIybM3lFKJUXk4xhi7Qa+rZTaLSL3AFuVUtYXwPuA+5VSynFsr4j8JcYXB8A9SqmzZo5X+dyMxJMopeyKXK9b8HvcBU/kDscSwGjvHo1Go5nNTCj6AEqph4CHsrZ9Puv+F/Mc+23g21Mc37QI+jwoZbRRtiN9jwu/11Vw751BM9JP6Ehfo9GUAWVTkZuLKr9RqDUST9r2jM/tIjCZSN+yd3Qxl0ajKQPKWvRDPuNCJhxL2faM121E+oWmbA7HdKSv0WjKh7IW/SrfaKSfSKVxCbhdktF
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(dt_range_lc3,dt_accuracy_cum3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 3e jeu de données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa58852a898>]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXmYXHd55/t5a+/q6n2V1N3qltSykbzJFsILNjaLbYbEBsLkMSQBJxnIRsg+gZtcMnFu5uaSeTLkzjhhCDjJzB1wgLA4xNgY8ILt2JaE5UWyZa1Wt6RW71vty+/+cepUV1XX1l171e/zPP1IdeqcqtPVVd96z/f3LqKUQqPRaDTNgaXaJ6DRaDSayqFFX6PRaJoILfoajUbTRGjR12g0miZCi75Go9E0EVr0NRqNponQoq/RaDRNhBZ9jUajaSK06Gs0Gk0TYav2CaTT29urRkdHq30aGo1GU1ccPnx4VinVl2+/mhP90dFRDh06VO3T0Gg0mrpCRN4sZD9t72g0Gk0ToUVfo9Fomggt+hqNRtNEaNHXaDSaJkKLvkaj0TQRWvQ1Go2midCir9FoNE2EFv0aIByN8eAL54hEY9U+FY1G0+Bo0a8Bnj01x6e/+QpPnZip9qloNJoGpyDRF5E7ReS4iJwUkU9n2ednReSYiBwVka8kbY+KyJH4z0OlOvFG4tJyAIATl1arfCYajabRyduGQUSswP3Ae4BJ4KCIPKSUOpa0zzjwGeAmpdSCiPQnPYRfKXVNic+7oZhdDQJwYlqLvkajKS+FRPoHgJNKqdNKqRDwIHB32j4fB+5XSi0AKKWmS3uajc3MihZ9jUZTGQoR/W3ARNLtyfi2ZHYDu0XkGRF5TkTuTLrPJSKH4tvfX+T5NiSm6J+8tIJSqspno9FoGplCumxKhm3pymQDxoFbgSHgxyJyhVJqERhRSl0QkR3Aj0TkFaXUqZQnEPkE8AmAkZGRDf4K9Y9p73hDUS4uBdja2VLlM9JoNI1KIZH+JDCcdHsIuJBhn+8opcJKqTPAcYwvAZRSF+L/ngaeAPalP4FS6otKqf1Kqf19fXnbQTccMytBej0OQFs8Go2mvBQi+geBcREZExEHcA+QnoXzbeA2ABHpxbB7TotIl4g4k7bfBBxDk8Lsaojrd/QAcOLSSpXPRqPRNDJ5RV8pFQE+CTwKvAZ8TSl1VETuE5G74rs9CsyJyDHgceAPlFJzwFuAQyLyUnz7XyRn/WggGImy5A9z2UAb3a0OTupIX6PRlJGCJmcppR4GHk7b9tmk/yvgd+M/yfs8C1xZ/Gk2LnOrIQB625zs6vdo0ddoNGVFV+RWGTNzp8/jZLzfw4npVZ3Bo9FoyoYW/Spjin5vmyH6S/4wM/FsHo1Goyk1WvSrjJmu2dfmZHygDYCTuh2DRqMpE1r0q4wZ6fe0Ohjv9wA6bVOj0ZSPghZyNeVjdjVIu8uGy27FabPQ7rJxYlqnbWo0mvKgI/0qM7MapK/NCYCIMD7QprttajSasqFFv8rMroTo9TgTt8d12qZGoykjWvSrTHKkD7Cr38OcN8S8N1TFs9JoNI2KFv0qY/TdSRV9QEf7Go2mLGjRLwHBSJQFbyjxEy5w1q0/FGU1GEmJ9M20Tb2Yq9FoyoHO3ikB7/wvT3J+0Z+4fc1wJ9/+jZvyHpeco2+ytcNFq8OqF3M1Gk1Z0KJfJOFojPOLft51eT83j/fy3Ol5Hjk6xXIgTLvLnvNYs/K2L8neERF29ns4NaNFX6PRlB5t7xSJLxQF4IadPdx70xg/d70xBOaVyaW8xyb67iRF+gA7+zyc0p6+RqMpA1r0i8QfF/0WhxWAq7Z1AnBkYjHvsaa9k7yQC8Zi7oWlAN5gpJSnqtFoNFr0i8UfNkTfHRf9DredHb2tvFSA6CdaMMSnZpns7GsF4PSMt5SnqtFoNFr0i8UXMqLxFvva8sjVw528NFlYpN/d6sBuTf0z7Owz0ja1r6/RaEqNFv0iMe0dM9IHuHqog0vLQaaWAjmPTZ6Nm8z2nlasFtG5+hqNpuRo0S8SXybRHy7M159ZCa5bxAVw2Cxs73brSF+j0ZQcLfpFYoq+y74m+m/Z0o7dKnktntnV0LpFXJOdugePRqMpA1r0iyQQXh/pu+xWLh9sz7uYO7MSTMnRT2Znn4ezc14iBVb3ajQaTSFo0S+SNXsntc7t6uEOXp5cIhbLPO/WG4zgD0cz2jtgZPCEo4pz877SnrBGo2lqtOgXSSJ7JynSB7h6qJPVYITTs5ktmsRs3CyRvtl47ZRO29RoNCVEi36RJIqz7Kmif01iMTdzZW6mvjvJ7OzXaZsajab0FCT6InKniBwXkZMi8uks+/ysiBwTkaMi8pWk7R8TkRPxn4+V6sRrBX84is0iOGypL+WOPg8epy2rr58v0m932elvc+rFXI1GU1LyNlwTEStwP/AeYBI4KCIPKaWOJe0zDnwGuEkptSAi/fHt3cCfAPsBBRyOH7tQ+l+lOvhC0XXWDoDVIly5rYOXs2TwzOSJ9CHeg0dH+hqNpoQUEukfAE4qpU4rpULAg8Ddaft8HLjfFHOl1HR8+x3AY0qp+fh9jwF3lubUawN/KJqSuZPM1cOdHLu4TDASXXff7EoQi0B36/riLJOd/a2cnF5FqcyLwRqNRrNRChH9bcBE0u3J+LZkdgO7ReQZEXlORO7cwLF1jS8cXefnm1wz3Ek4qng5Q8fNc/M+ej1OrBbJ+ti7+jysBCKJqwKNRqMplkJEP5MqpYeeNmAcuBX4MPAlEeks8FhE5BMickhEDs3MzBRwSrWDPxSlxZHZJbtxVw92q/DYsUsp2yPRGE++McNNu3pzPnZiMXdaZ/BoNJrSUIjoTwLDSbeHgAsZ9vmOUiqslDoDHMf4EijkWJRSX1RK7VdK7e/r69vI+VcdfziS1d5pd9m5YWcvjx6dSrFoDp5dYMEX5vY9AzkfOzEvV/v6Go2mRBQi+geBcREZExEHcA/wUNo+3wZuAxCRXgy75zTwKHC7iHSJSBdwe3xbw+DL4ekD3LF3gDfnfLyRNP7w0aNTOG0W3nFZ7i+4wXYXbodVD1TRaDQlI6/oK6UiwCcxxPo14GtKqaMicp+I3BXf7VFgTkSOAY8Df6CUmlNKzQN/hvHFcRC4L76tYfCHsnv6AO/ZM4CIIfQASikeO3aJm8f71lXxpiMiOoNHo9GUlIJm5CqlHgYeTtv22aT/K+B34z/pxz4APFDcadYu2VI2TfrbXFw70sWjR6f41LvGefX8MucX/fz2u8cLevxd/R6eOz1XqtPVaDRNjq7ILRJ/OLe9A4bFc/TCMhPzPh49OoVF4N1vye3nm+zq93BxKcCqHp1YEyilePrEbNaeShpNraNFv0gMeyf3BdPtewYB+P6xS3z/2BQHxrrpypGfn8x4fDH3xKWV4k5UUxIOvbnAz3/5eR48OJF/Z42mBtGiXwRKKXyh7Nk7JqO9rVw20MY/PnuWNy6tcsfewYKfY3ygDYATejG3Jjgza6TPPvDMGV00p6lLtOgXQTASI6bWd9jMxB17BxJtkm/fgOiPdLtx2iw60q8RJhf8AJycXuXJN+qrpkSjAS36RWEOUMmVvWNiCv2V2zrY1tlS8HNYLUYGj470a4PJeR99bU4G2p18+ekz1T4djWbDaNEvgkzzcbOxd2s777q8n3tvHN3w84wPeDhxSYt+LTC54Gesp5WP3jDKj0/McnxKX4Fp6gst+kVgin4h9o6I8OV738rPXDe04efZPdDG+UW/zuCpASYXfAx1tfCRAyO47BYe0NG+ps7Qol8E2QaolJpEOwZt8VSVUCTGxeUAQ91uulodfPDaIb515HxiII5GUw9o0S8Cc1RivsraYtkdz+B5Qy/mVpWLS36UgqEuY03ml24aIxSJ8dXnz1X5zDSawtGiXwT+cOH2TjGMdLtx2Cw60q8yZuaOKfq7+j3sG+nk8ePTuQ7TaGoKLfpF4N/AQm4xmBk8OtKvLhPxlNvhLndi29vGenh5cinxXtBoap2GEX2lFKFIjHA0VrHn3Ej2TrHs1hk
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(dt_range_lc4,dt_accuracy_cum4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut facilement observer que dans les trois cas, il y a très peu d'apprentissage de réalisé par l'algorithme avec les exemples fournis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test et évaluation de la performance de l'arbre de décision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[144.  74.]\n",
      " [131.  85.]]\n",
      "\n",
      "Exactitude:\n",
      "0.5254629629629629\n",
      "\n",
      "Précision:\n",
      "[0.5236363636363637, 0.5345911949685535]\n",
      "\n",
      "Rappel:\n",
      "[0.6605504587155964, 0.39351851851851855]\n",
      "\n",
      "Calculé en:\n",
      "0.0012059211730957031s\n"
     ]
    }
   ],
   "source": [
    "dt2_testres = dt2.test(test2, test_labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[219.  72.]\n",
      " [ 88.  56.]]\n",
      "\n",
      "Exactitude:\n",
      "0.6296296296296297\n",
      "\n",
      "Précision:\n",
      "[0.7133550488599348, 0.4375]\n",
      "\n",
      "Rappel:\n",
      "[0.7525773195876289, 0.3888888888888889]\n",
      "\n",
      "Calculé en:\n",
      "0.001070261001586914s\n"
     ]
    }
   ],
   "source": [
    "dt3_testres = dt3.test(test3, test_labels3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[ 30. 175.]\n",
      " [ 25. 207.]]\n",
      "\n",
      "Exactitude:\n",
      "0.5370370370370371\n",
      "\n",
      "Précision:\n",
      "[0.5454545454545454, 0.5418848167539267]\n",
      "\n",
      "Rappel:\n",
      "[0.14634146341463414, 0.8922413793103449]\n",
      "\n",
      "Calculé en:\n",
      "0.001104116439819336s\n"
     ]
    }
   ],
   "source": [
    "dt4_testres = dt4.test(test4, test_labels4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Réseaux de neurones: choix du nombre de neurones dans la couche cachée"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pour faire la séparation du jeu de données en $k_{cv}=5$ jeux de validation croisée, on génère une permutation sur les indices du jeu d'entrainement, puis, on sépare cet ensemble en 5 groupes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_indices2 = range(len(train_labels2))\n",
    "all_indices3 = range(len(train_labels3))\n",
    "all_indices4 = range(len(train_labels4))\n",
    "np.random.seed(12345)\n",
    "indices_cv_test2 = (\n",
    "    [np.sort(x) for x in np.array_split(np.random.permutation(all_indices2),k_cv)])\n",
    "np.random.seed(12345)\n",
    "indices_cv_test3 = (\n",
    "    [np.sort(x) for x in np.array_split(np.random.permutation(all_indices3),k_cv)])\n",
    "np.random.seed(12345)\n",
    "indices_cv_test4 = (\n",
    "    [np.sort(x) for x in np.array_split(np.random.permutation(all_indices4),k_cv)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_cv_train2 = (\n",
    "    [np.setdiff1d(all_indices,indices_cv_test2[i]) for i in range(k_cv)])\n",
    "indices_cv_train3 = (\n",
    "    [np.setdiff1d(all_indices,indices_cv_test3[i]) for i in range(k_cv)])\n",
    "indices_cv_train4 = (\n",
    "    [np.setdiff1d(all_indices,indices_cv_test4[i]) for i in range(k_cv)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ces jeux de données ont deux classes possibles comme variable de sortie. On utilisera donc un réseau de neurones avec deux neurone dans la couche de sortie, une pour chacune des valeurs possibles. Les valeurs de sortie du jeu de données sont transformées à l'aide d'un encodage binaire où la valeur de sortie est convertie en rang dans un vecteur (on commence à 0), prenant la valeur 1. Par exemple, la valeur 1 devient $[0,1]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum2 = []\n",
    "for n_neurones in choix_n_neurones:\n",
    "    accuracy_cv=[]\n",
    "    for cv_set in range(k_cv):\n",
    "        nn2 = NeuralNet.NeuralNet(np.array([6,n_neurones,2]),range(2))\n",
    "        nn2.train(train2[indices_cv_train2[cv_set]], \n",
    "                  train_labels2[indices_cv_train2[cv_set]], 0.1, 1, \n",
    "                  verbose=False)\n",
    "        _,accuracy,_,_,_ = nn2.test(train2[indices_cv_test2[cv_set]], \n",
    "                                    train_labels2[indices_cv_test2[cv_set]], \n",
    "                                    verbose=False)\n",
    "        accuracy_cv.append(accuracy)\n",
    "    accuracy_cum2.append(np.mean(np.array(accuracy_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5885823c8>]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXt0W+d1p/1sAATBOymSEilRlGSLsmVdLNmKcnHj2Elsy4lrp83USdq1UrdNPN+0biaTNtO4k8lkPNNpM2mnnU4939ckk5WknTRNc3HkRrZi535xUsuWdb9blkRSvBMkSBD3/f0BHBAkQeKQxOWAfJ+1uCwcHhAvYGBjn9+792+LqmIwGAyG1YGr1AswGAwGQ/EwQd9gMBhWESboGwwGwyrCBH2DwWBYRZigbzAYDKsIE/QNBoNhFWGCvsFgMKwiTNA3GAyGVYQJ+gaDwbCK8JR6AbNpaWnRzZs3l3oZBoPBUFa89NJLQ6ramus8xwX9zZs3c+TIkVIvw2AwGMoKEbli5zwj7xgMBsMqwgR9g8FgWEWYoG8wGAyrCBP0DQaDYRVhgr7BYDCsIkzQNxgMhlWECfoGg8GwijBB35CTp4/1MjIZKfUyDAZDHjBB37AgPf4pfv8fjvKNl7tLvRSDwZAHTNA3LMj5/gAAY1PREq/EYDDkAxP0DQtysX8CgHET9A2GFYGtoC8iB0TknIhcFJGPzXPOwyJyWkROiciXM47/99SxMyLy1yIi+Vq8ofBcGEhm+uOhWIlXYjAY8kFOwzURcQNPAvcA3cCLInJQVU9nnNMFPA7coaqjIrI2dfxNwB3A7tSpPwHeAvwgn0/CUDguDJhM32BYSdjJ9PcDF1X1VVWNAF8BHpp1zgeBJ1V1FEBVB1LHFfABXqASqAD687FwQ+FR1Wl5J2SCvsGwErAT9DcA1zJud6eOZbIN2CYiPxWRn4vIAQBVfQH4PnA99XNYVc8sf9mGYtA/HiYQTso641NG3jEYVgJ2/PSzafCa5e90AXcBHcCPRWQn0AJsTx0DeE5E7lTVH814AJFHgUcBOjs7bS/eUFgsPX/jmiqT6RsMKwQ7mX43sDHjdgfQm+Wcb6lqVFUvA+dIfgn8CvBzVZ1Q1QngGeANsx9AVT+jqvtUdV9ra87BL4YicSEl7dze2WQ0fYNhhWAn6L8IdInIFhHxAu8FDs465yngbgARaSEp97wKXAXeIiIeEakguYlr5J0y4cLABE3VFWxuqWEyEicWT5R6SQaDYZnkDPqqGgMeAw6TDNhfVdVTIvKEiDyYOu0wMCwip0lq+B9V1WHga8Al4ARwDDimqk8X4HkYCsDFgQBda+uo91UAMBE2ur7BUO7YmpGrqoeAQ7OOfSLj3wp8JPWTeU4c+NfLX6ah2Kgq5/sneOfuduqrkkF/fCpGY7W3xCszGAzLwXTkGrIyNBFhbCpK19pa6n3J3MBs5hoM5Y+tTN+w+rAqd7rW1uFxJwu4zGauwVD+mEzfkJWLqU7crnW1aU3fZPoGQ/ljMn1DVi70T1Dv87C2rpJoqmrHNGgZDOWPyfQNWTnfH6BrXR0iMr2RazJ9g6HsMUHfkJWLAxN0ra0FoNbrQcRo+gbDSsAEfcMchifCDE9G2JoK+i6XUFvpMfbKBsMKwAR9wxymN3Hr0sfqfRVG3jEYVgAm6BvmYHnoW/IOQH1VhdnINRhWACboG+ZwcWCCGq+b9gZf+li9z2MyfYNhBWCCvmEOFwYCbE1V7lgkM30T9A2GcscEfcMcLvRPzJB2IKnpB8xGrsFQ9pigb5jBWDDKQCA8N+hXeUymbzCsAEzQN8zg4mDKc2ddlkw/HCOemD00zWAwlBMm6BtmYE3L6lpbN+O41ZU7YSQeg6GsMUHfMIMLAxP4KlxsaKyacbzO2CsbDCsCW0FfRA6IyDkRuSgiH5vnnIdF5LSInBKRL2cc7xSR74jImdTvN+dn6YZCcGFggq1ra3G5ZMZx47RpMKwMcrpsiogbeBK4h+QA9BdF5KCqns44pwt4HLhDVUdFZG3Gn/gS8Ceq+pyI1AJm0KqDudgf4A03NM85Xl+VyvRNg5bBUNbYyfT3AxdV9VVVjQBfAR6adc4HgSdVdRRAVQcAROQWwKOqz6WOT6hqMG+rN+SVQChK71iIrbM2ccFk+gbDSsFO0N8AXMu43Z06lsk2YJuI/FREfi4iBzKO+0XkGyJyVEQ+nbpyMDiQtOfOrE1cgIb0nFwT9A2GcsZO0Jcsx2bX7XmALuAu4H3A50SkMXX8zcAfAq8DbgAemfMAIo+KyBEROTI4OGh78Yb8ks1zx2I60zfyjsFQztgJ+t3AxozbHUBvlnO+papRVb0MnCP5JdANHE1JQzHgKeC22Q+gqp9R1X2quq+1tXUpz8OQBy4OTOD1uNi4pnrO72qt6h2T6RsMZY2doP8i0CUiW0TEC7wXODjrnKeAuwFEpIWkrPNq6r5NImJF8rcCpzE4kgv9AW5srcXtmntx53YJdZXGdM1gKHdyBv1Uhv4YcBg4A3xVVU+JyBMi8mDqtMPAsIicBr4PfFRVh1U1TlLa+a6InCApFX22EE/EsHwuDMz13MnE2CsbDOWPrcHoqnoIODTr2Ccy/q3AR1I/s+/7HLB7ecs0FJpgJEb36BTv2bdx3nPqfB4CJtM3GMoa05FrAODSwCQw13MnEzM9y+AUxoJR+sZCpV5GWWKCvgFIeugDbM1SrmmRdNo08k45krwYXzl8/FsneeB//dhceS4BE/SLRCKh9PinSr2MebkwMEGFW9jUPLdyx8Jk+uXJ//fDS7zjr39CYgU5pB55bYShiQif/dGrpV5K2WGCfpF45mQfd336+wyMO/OS9PLgJJuaa6hwz/+WMNOzypPj3X7OXB/n6DV/qZeSFwYCIa6Phaj2uvnsjy8zEHDmZ8qpmKBfJF4bniQa13QDlNMYCUZorvEueE69z0MgHFtRGeNqYGA8DMChE9dLvJL8cPzaGAD/+cEdROMJ/ufzF0q8ovLCBP0iMTwRAZLB34mMT0VprK5Y8Jz6qgpUYSJidP1yYnAiGfSfOXF9RXxhH+v243YJ79zdzq+/vpOvvHiNS4POTKaciAn6RWJkMvnBuzLsTL85fzCa9teZj7QVg5F4ygZVZWA8zNq6SnrHQrzSXf4Sz7HuMbrW1lLt9fCht3Xh87j49LPnSr2sssEE/SIxPJnK9IecmemPTUVprM4h7xh75bJjMhJnKhrn4X0b8bpdHDpe3hKPqnK828+ejY0AtNRW8uidN/LsqT5evjpa4tWVByboF4mhlLxzdcR5mX44lgwMdjN9UyZXPliFAze01vDmrhaeOdlX1uWbV0eC+INRdnc0po994M1baKmt5M8OnS3r51YsTNAvEsMpXfW14UnHvTHHUnJNrqBfZ5w2y47BQPJ9t7bOxzt2tdPjn+KVMq7iOdad3MTd3dGQPlZT6eHDb+/iX14b4btnBkq1tLLBBP0ikEgoI5MRGqoqCEUTDKQ+iE5hLGgv6E/LO6sz04/FE477ws6F9V5rravk7beso8ItZV3Fc+yan0qPi5vaZjYRvud1G7mhpYZPPXuWWLw8h/N965UenjraU/D3mAn6RWA8FCWWUG7rTF6S2tX1w7E4/+lbJwve1GVl+jmrd1bI9KxQNPm6do8uTmp74H/9hL/+7sUCraowTGf6lTRUVfBLW1s4dKJ8JZ7j3X52rK+f009S4Xbx0ftu4sLABF9/ubtEq1s6/eMhPv7USb78L1cp9P8aE/SLgKXn376pCbBfwXPs2hhffOEKf/fClYKtDZKVO2BH3lkZG7k/Oj/IF1+4wjdf7rF9n17/FGf7Ao4tuZ2PgUCYCrek/99aEo8lk5QTsXiCkz3j3LqxMevvD+xsY29nI//jufMMjIfwByNzfkLReJFXnRtV5T988ySRWIJPvXs3rizW5vnElsumYXlYev6ujkY8LrEdOM71J/1wDp/q448O3IRIYd4MdjV9j9tFjddd9pn+c6f7AXhpEdUeVmXIZLi8vvAGA2FaaivTgeTeW9r4Y/cJDp24nq6
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(choix_n_neurones,accuracy_cum2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de neurones qui maximise l'accuracy est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_neurones_optimal2 = (\n",
    "    choix_n_neurones[np.where(accuracy_cum2==max(accuracy_cum2))[0][0]])\n",
    "n_neurones_optimal2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum3 = []\n",
    "for n_neurones in choix_n_neurones:\n",
    "    accuracy_cv=[]\n",
    "    for cv_set in range(k_cv):\n",
    "        nn3 = NeuralNet.NeuralNet(np.array([6,n_neurones,2]),range(2))\n",
    "        nn3.train(train3[indices_cv_train3[cv_set]], \n",
    "                  train_labels3[indices_cv_train3[cv_set]], 0.1, 1, \n",
    "                  verbose=False)\n",
    "        _,accuracy,_,_,_ = nn3.test(train3[indices_cv_test3[cv_set]], \n",
    "                                    train_labels3[indices_cv_test3[cv_set]], \n",
    "                                    verbose=False)\n",
    "        accuracy_cv.append(accuracy)\n",
    "    accuracy_cum3.append(np.mean(np.array(accuracy_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5884c6b38>]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXmUJHd15/u9ua+1r129VG9C6gW3rFarxSJ2aAGWOGOMxZNZDIwMYz3ws2eeYY7Bz2DODPZ5g5+PGQyyWWQkZAaPjSxaEgIjEKClS1ZL6kVSV+9VXVvXlkvlnvf9EfGLjMyKzIjMyszIqvx9zqnTlZGRUb/Mzogbd/teYmZIJBKJROKwewESiUQiaQ2kQZBIJBIJAGkQJBKJRKIiDYJEIpFIAEiDIJFIJBIVaRAkEolEAkAaBIlEIpGoSIMgkUgkEgDSIEgkEolExWX3Aqqhr6+PR0dH7V6GRCKRrCueffbZq8zcb7bfujIIo6OjGBsbs3sZEolEsq4gootW9pMhI4lEIpEAkAZBIpFIJCrSIEgkEokEgDQIEolEIlGRBkEikUgkAKRBkEgkEomKNAgSiUQiASANgkTlZ6/M4dL8it3LkEgkNiINggQA8AcPPIdv/PK83cuQSCQ2Ig2CBAAQT+cQT2XtXoZEIrERaRAkYGaks3kks3m7lyKRSGxEGgQJUqohSGZyNq9EIpHYiTQIEqQy0iBIJBJpECQAklnFEAjDIJFI2hNLBoGIjhDRy0Q0TkSfNnj+w0Q0R0TH1Z+PqdsPENGTRHSSiF4got/WveZbRHRe95oD9XtbkmoQnoEwDBKJpD0xnYdARE4AXwHwNgATAI4R0YPMfKpk139k5rtLtq0A+CAznyGiTQCeJaJHmXlJff6/MPP31/geJGtE5hAkEglgzUM4BGCcmc8xcxrAAwBut3JwZn6Fmc+ov18BMAvAdGqPpLloHoIMGUkkbY0VgzAC4LLu8YS6rZTfVMNC3yeiLaVPEtEhAB4AZ3Wbv6i+5stE5K1m4ZL6ITyElAwZSSRtjRWDQAbbuOTxvwIYZeZXA/gxgG8XHYBoGMA/APhdZha3oZ8BcC2AGwH0APhjwz9OdBcRjRHR2NzcnIXlSqpFeggSiQSwZhAmAOjv+DcDuKLfgZnnmTmlPrwHwA3iOSLqAPBDAH/CzE/pXjPFCikA34QSmloFM3+dmQ8y88H+fhltagRJWXYqkUhgzSAcA7CbiLYTkQfAHQAe1O+gegCC2wCcVrd7APwzgHuZ+X8ZvYaICMB7AJyo9U1I1oYIFaWyeTCXOn8SiaRdMK0yYuYsEd0N4FEATgDfYOaTRPR5AGPM/CCATxLRbQCyABYAfFh9+fsA3AKgl4jEtg8z83EA9xFRP5SQ1HEAH6/f25JUgz5UlMrm4XM7bVyNRCKxC1ODAADMfBTA0ZJtn9P9/hkoOYHS130HwHfKHPPNVa1U0jD0yeRkJicNgkTSpshOZUmRhyATyxJJ+yINgmSVhyCRSNoTaRAkxR6C7EWQSNoWaRAkSOm8AilwJ5G0L9IgSLROZUCGjCSSdkYaBEmREZBT0ySS9kUaBIn0ECQSCQBpECRQjIDTQdrvEomkPZEGQYJUNo8uv1v5XSaVJZK2RRoECZKZHDpVgyDLTiWS9kUaBIliEAJu7XeJRNKeSIMgQSqbL3gIMmQkkbQt0iBIkMzkEPS44HKQnJomkbQx0iBIkMrm4XU74HM7pYcgkbQx0iBIkMzk4XU54XU5ZA5BImljpEGQIJXNwSc9BImk7ZEGQYJURpmS5nU7ZNmpRNLGSIPQ5uTyjHQuD6/LAZ/LWaR8KpFI2gtpENqctKpj5HM74XM7ZMhIImljpEFoc0QS2esSOQTpIUgk7Yolg0BER4joZSIaJ6JPGzz/YSKaI6Lj6s/HdM99iIjOqD8f0m2/gYheVI/510RE9XlLkmpIFXkITplDkEjaGFODQEROAF8BcCuAPQDeT0R7DHb9R2Y+oP78nfraHgB/CuAmAIcA/CkRdav7fxXAXQB2qz9H1vpmJNVT7CHIkJFE0s5Y8RAOARhn5nPMnAbwAIDbLR7/HQAeY+YFZl4E8BiAI0Q0DKCDmZ9kZgZwL4D31LB+yRop8hBcTtmpLJG0MVYMwgiAy7rHE+q2Un6TiF4gou8T0RaT146ov5sdE0R0FxGNEdHY3NycheVKqkF4CD63Qyk7lR6CRNK2WDEIRrF9Lnn8rwBGmfnVAH4M4Nsmr7VyTGUj89eZ+SAzH+zv77ewXEk1FEJGTnhdMqkskbQzVgzCBIAtusebAVzR78DM88ycUh/eA+AGk9dOqL+XPaakORRCRkqVkRyQI5G0L1YMwjEAu4loOxF5ANwB4EH9DmpOQHAbgNPq748CeDsRdavJ5LcDeJSZpwBEieiwWl30QQA/WON7kdSA3kPwuR1I5/LI5Q2dNUkb8dALV/C5H5ywexmSJmNqEJg5C+BuKBf30wC+x8wniejzRHSbutsniegkET0P4JMAPqy+dgHAF6AYlWMAPq9uA4BPAPg7AOMAzgJ4uG7vSmKZUg9B2SbDRu3Oj07O4L6nLyGTkx5jO+GyshMzHwVwtGTb53S/fwbAZ8q89hsAvmGwfQzAvmoWK6k/RR6Cy6FuyyPgsXNVErtZXEkjl2dMLiYw2he0ezmSJiE7ldsc4SF4dR6CTCxL5mNpAMCF+bjNK5E0E2kQ2pxC2alTGgSJxkJcMQgX51dsXomkmUiD0OZoHoLaqazfJmlPmFkzCNJDaC+kQWhzUpkciACP0wGvS3oIEiCWyiKtJpOlh9A87rp3DA+9YG/1vTQIbU4yq8xCICJ43YWksqR9WYxnAABE0kNoFvk847HTM3jq3Lyt65AGoc1JZXJa7kDLIciy07ZmPq70mF4zEMblhRXZl9IEVjI5MANLKxlb1yENQpuTzCgeAgD41JCRnJrW3oj8wa9v60Imx7iylLB5RRufWDILAFhOSIMgsZFUVu8hyJCRBJhXDcL1WxWleplHaDyxlGIIpEGQ2Eoyk9c8A1l2KgGAReEhqAZB5hEaT1T1EGTISGIryWxOSyZLgyABlJCRx+XAjr4gfG4HLkqD0HBiKRkykrQAqSIPQQ0ZyT6EtmY+nkZv0AOHg7CtJ4gLMmTUcOKqQYgkM8jbmMSXBqHN0XsIXi2pLA1CO7MQT6MnqIhZbesNSA+hCYiQEXPhdzuQBqHNSWXymiFwOghuJ8my0zZnXmcQRvuCuDi/YutdazsgQkYAsJRI27YOaRDaHL2HACilpzKH0N4slngIqWwe05Gkzava2MR0XoGdiWVpENocfQ4BALxupyw7bXP0IaPRXkX6WlYaNRa9h2BnYlkahDZH6UPQeQhuh2xMa2NS2RxiqSx6dR4CIHsRGk20KGQkDYLEJpK6HAKglJ7KHEL7IrqUu1WDMNzph8fpkB5Cg4mnsgh6lPNQeggS2zDyEGTIqH0RBkF4CE4HYUuPHxevblwPIZ3N4w1/+VM8/OKUbWuIJbMY6fYDAJZXZFJZYgO5PCOT42IPQSaV2xphEHqCXm3baG9wQ3sISytpXJxfwbELi7atIZrKoifogd/tbP2kMhEdIaKXiWiciD5dYb/3EhET0UH18Z1EdFz3kyeiA+pzj6vHFM8N1OctSaySyoppaXoPQRqEdqZgEApDtbf1KqWnzBuz9DSiVvhMLtnnBcWSWYS8bnT63a0dMiIiJ4CvALgVwB4A7yeiPQb7hQF8EsDTYhsz38fMB5j5AIAPALjAzMd1L7tTPM/Ms2t8L5IqEaEhoXYKqEll2anctohZyr06gzDaF0Aik8NcNGXXshpKJKlcgK8s2VdaG0tlEfa50BVwt3xS+RCAcWY+x8xpAA8AuN1gvy8A+AsA5T7V9wP4bk2rlDSEgoegKzuVIaO2ZiGehoOATr9b27ZNKz3dmHmEqOYh2CfzHUtlEfQ6W99DADAC4LLu8YS6TYOIrgewhZkfqnCc38Zqg/BNNVz0WSIiKwuW1A/hIRQZBJlUbmsWVtLoDig6RoJRtfR0o+YRIuoFeCGexkr
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(choix_n_neurones,accuracy_cum3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de neurones qui maximise l'accuracy est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_neurones_optimal3 = (\n",
    "    choix_n_neurones[np.where(accuracy_cum3==max(accuracy_cum3))[0][0]])\n",
    "n_neurones_optimal3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum4 = []\n",
    "for n_neurones in choix_n_neurones:\n",
    "    accuracy_cv=[]\n",
    "    for cv_set in range(k_cv):\n",
    "        nn4 = NeuralNet.NeuralNet(np.array([6,n_neurones,2]),range(2))\n",
    "        nn4.train(train4[indices_cv_train4[cv_set]], \n",
    "                  train_labels4[indices_cv_train4[cv_set]], 0.1, 1, \n",
    "                  verbose=False)\n",
    "        _,accuracy,_,_,_ = nn4.test(train4[indices_cv_test4[cv_set]], \n",
    "                                    train_labels4[indices_cv_test4[cv_set]], \n",
    "                                    verbose=False)\n",
    "        accuracy_cv.append(accuracy)\n",
    "    accuracy_cum4.append(np.mean(np.array(accuracy_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5884a2fd0>]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztvXl8W3ed9/v5arcleZG32M5ix3GapEmTtGmSbnRhaQolpTAzt+0wtMxAmZnLLTPMcuF58cBzO8M8w+U+bM+UGTqlwDDQsgzQAIVQuoc2K0mapGkSx3Zix6sWL5Ks/Xf/OOfIx7KWI1u7vu/Xyy9LR0fSz8fS53zPdyUhBBiGYZjqQFfsBTAMwzCFg0WfYRimimDRZxiGqSJY9BmGYaoIFn2GYZgqgkWfYRimimDRZxiGqSJY9BmGYaoIFn2GYZgqwlDsBSTS3Nwsurq6ir0MhmGYsuLYsWNOIURLpv1KTvS7urpw9OjRYi+DYRimrCCiS1r2Y/cOwzBMFcGizzAMU0Ww6DMMw1QRLPoMwzBVBIs+wzBMFcGizzAMU0Ww6DMMw1QRLPoMU2K4fSHsOzlS7GWUNL5gBG5fqNjLKEtY9BmmxPjR0SE88tRxjEzNFXspJcs/PXsWD33rcLGXUZaw6DNMiTE6HQAAXJjwFnklpcvI1BwGnb5iL6MsYdFnmBJjYlYW/fHZIq+kdPEGI5gJRBCKxIq9lLKDRZ9hSozxmSAAoI8t/ZTMBiIAwH79JcCizzAlxvgMu3cyoYi+yxcs8krKDxZ9hikhhBCYkC39C+OzEEIUeUWliTcoi76XLf1sYdFnmGUihMBbYzM5ea0pfxihaAzdzVbMBCKYnGVLNhEhRFz02b2TPZpEn4j2ENE5Iuojok8lefzLRHRC/jlPRFOqxx4kogvyz4O5XDzDlAKv97uw5yuv4uTQVOadMzAuB3FvWtcEgF08yZgLRxGNSVdATi+fFLMlo+gTkR7AYwDuArAJwP1EtEm9jxDir4UQ24QQ2wD8bwA/kZ/rAPA5ALsA7ATwOSJqzO2fwDDFpX9SSh08PTK97NdSgrg3r2sGwBk8yfDK/nyALf2loMXS3wmgTwjRL4QIAXgawD1p9r8fwFPy7TsBPCeEcAshPACeA7BnOQtmmFJjdFoqorowvnyrXAnibmqvR0OtkS39JMwG50WfffrZo0X0OwEMqe4Py9sWQURrAHQDeCHb5zJMuaIUU53PgVU+IYt+a50Zva02Fv0kqC19zt7JHi2iT0m2pUopuA/Aj4UQ0WyeS0QPE9FRIjo6OTmpYUkMUzqMTuUuxXJ8JoiGWiMsRj3Wtdo5Vz8JShDXbNDBxe6drNEi+sMAVqnurwSQqhvUfZh37Wh+rhDicSHEDiHEjpaWjMPcGaakGJOt88nZIKb8yxOh8ZkA2uwWAEBvqw1uXwguDlYuYDYQBgCsaapl984S0CL6RwD0ElE3EZkgCfu+xJ2I6CoAjQBeV23eD+BdRNQoB3DfJW9jlslzb47jkaeOF3sZVY8QAiNTc+httQEAzi/Trz8+G0RrnRkA0NsmvSa7eBaiFGatdlg5kLsEMoq+ECIC4OOQxPosgB8KIc4Q0aNEtFe16/0AnhaqahIhhBvAP0A6cRwB8Ki8jVkmL5+fwL6TI/HUNaY4ePxhBCMx3LpeukJdrl9/YiaAtjrF0rcDYNFPRHHvdDXVwhuMIBCOZngGo8agZSchxLMAnk3Y9tmE+/8jxXOfBPDkEtfHpMDjky5xZwNhNNSairya6kXJ3LluTSOeOnx5WSmWsZjAxGwQbbKl31Znht1sQB+nbS5ACeSuabYCkNI2OxpqirmksoIrcssU5bJ2Zi6SYU8mnyhB3PaGGqxrsy/LvePyhRCNibilT0RY18YZPInMBiOwGHVos0snR/brZweLfpnikQOGM3JQiykOo3IQt6PegvWtNlyYWLpVruTot8qBXEAK5i43TlBpzAYisJmNaLLJos9pm1nBol+mzFv6LPrFZHRqDgYdodlmxvo2O5ze0JKDi0offcW9A0h+fac3CA8HLON4gxHYLQY0WSW3Jlv62cGiX4YIITDll8SeLf3iMjYtBV51OprPtlmiD15pwaC4dwBgnfyafZNs7St4A2HYzAY02WTRZ0s/K1j0C8SJoSns/ZcD8IeW74P3haIIRaWJQezTLy4j03PoaJCzbdqkbJvzS/TBK+6dFrva0ldOJCz6CoqlbzMbYNJzgVa2sOgXiKODbrwxPI1Bp3/Zr6W+1GdLv7iMTgewol7KHOmot8BmNizL0m+2mWDUz38tO+prUGPULytWUGlIPn0DiAhNNhO7d7KERb9AKNbIZA6qK9U+Y/bpFw8hBEanA+ioV2XbtNqWnKs/MRNYEMQFAJ1Oek1uxzDPbCACm0XKNndYTVyglSUs+gVCKaXPxVAMt19t6bN7p1i4fSGEIjGsqJ8X6vVttiW7YsZnAwuCuAq9rUt/zUrEG4zAbpZEv8lm5jYVWcKiXyAUayQXoq/0d9ERMM2WftFQumu2188XBq1vs8O1xH454zPBBUFchXVtNozNBNiVh/mpWXaLEQDQbDWxTz9LWPQLhNObO9F3y9W4HQ017N4pIvOir8qrb1ta64RINAanN4jWJKKvtGNgFw8QCMcQjYkF7h326WcHi36BcOfQp+/xhaAjoLOhhq2/IjImt2Bob1jo3gGyT9t0ekMQAindOwDQxy6eeIdNm8q9MxeO5iQrrlpg0S8QyuW+MiRjObj9ITTWmtBQa+SUzSIyMh2AUU9ots4L9Yo6C+xmQ9ZVtEq6Zpt9saW/ylELk0HHGTyYn5plly19LtDKHhb9AhAIR+ELSZ0Ac2XpN1pNqLMY2dIvIqNTc/HCLAWlX062GTxx0U/i3tHrCD0t3IMHmG+2Nm/pKwVaLPpaYdEvAMoHstakz4lP3+MPwVFrQn2NkX36RURK11zc3XF9qz1rgR6fVapxF7t3AM7gUfDGLX0pkOuQLX03V+VqhkW/ALjlS8/1bXbMBpbf/9vjC6PRakRdjRG+UBQRuTqXKSxSYVaSwGubNPHKmcVV3cRMADpCvInYotdsteHK1Bx8wep25yX69Jvl4+Vk945mWPQLgFO2QjaskLIwlmvtKz79OtmvOcu5+gVHCIGx6cCCIK7CeiWDJwvLfHwmgBa7GXpdsrHS81O0LlZ5Dx7lsx736dsUS59FXyss+gVAsfTjor8Mv74QYt6nXyNd4rJfv/C4fCGEojG0J/HBx0U/i8Brqhx9hXWt2Z9ICk0kGsMX97+Fkam5vL2H4t5RLP1akwEWo44LtLKARb8AKF0AN7TXAViepT8bjCASE3DUSoFcgJuuFYOx6fnhKYm01ZlhtxiyCuaOJ2nBoGZNUy2MeirpYO6xSx489uJFPH1kKG/vEQ/kWuaH/jVZzZy9kwUs+gXA5QvBZNChWx7vthzRV5qtqS19rsotPIo1myyQS0RZDz9Rj0lMhlEvfX76Sjht89CANP76UL8rb+/hladmqZvSNdm4KjcbWPQLgMsbQpPVhCarCUTLFH25j77DakRdjWTtsHun8CjVuMkCuYDk4rkwPgshMg+uD0aicPtCad07gFSZW8qW/kFZ7I8PTeVtWPmMPDVLTZPVxD31s4BFvwC4fSE02Uww6HVospowkQtLf4F7h0W/0IxOB2CS/5/J6G2zw+MPa8oqmcyQrqmwrtWGy25/3gR1OYQiMfz+sgfdzVaEIjGcGJrKy/sovfTVOKzmeNyMyQyLfgFweYNwyFWbzTbzsix9t1r0OZBbNEan59BWb15QmKUm3o5BgztGmZiVrO+Omt42G4QozQyeN4anEAjH8Be39YAIONTvzsv7KFOz1DTbTHD6QpquqhgW/YLg8oXQLFuELXbzsrJ3lIHojVYTrCY9dMSB3GIwOh1Y0F0zkWzSNidnU7dgUFPKjdcU1847N7ZhU3td/H6uSWbpN9lMCEVi8ap3Jj0s+gXA5Q3FKwdb7GY4l2np63WEOos0OaiuhlsxFIPR6bkF3TUTabWbUacxg2d+Nm56905Xcy3
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(choix_n_neurones,accuracy_cum4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de neurones qui maximise l'*accuracy* est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_neurones_optimal4 = (\n",
    "    choix_n_neurones[np.where(accuracy_cum4==max(accuracy_cum4))[0][0]])\n",
    "n_neurones_optimal4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choix du nombre de couches cachées"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy_couches(train,train_labels,n_neurones_optimal):\n",
    "    accuracy_cum = []\n",
    "    lc_cum = []\n",
    "    for n_couches in choix_n_couches:\n",
    "        accuracy_cv=[]\n",
    "        nn = NeuralNet.NeuralNet(\n",
    "            np.hstack((6,\n",
    "                       np.repeat(n_neurones_optimal,n_couches),\n",
    "                       2)),\n",
    "            range(2))\n",
    "        lc = nn.train(train, train_labels, 0.1, 10, verbose=False)\n",
    "        lc_cum.append(lc)\n",
    "        _,accuracy,_,_,_ = nn.test(train, train_labels, verbose=False)\n",
    "        accuracy_cv.append(accuracy)\n",
    "        accuracy_cum.append(np.mean(np.array(accuracy_cv)))\n",
    "    lc_cum = np.array(lc_cum)\n",
    "    return accuracy_cum, lc_cum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum2, lc_cum2 = accuracy_couches(train2,train_labels2,n_neurones_optimal2)\n",
    "accuracy_cum3, lc_cum3 = accuracy_couches(train3,train_labels3,n_neurones_optimal3)\n",
    "accuracy_cum4, lc_cum4 = accuracy_couches(train4,train_labels4,n_neurones_optimal4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L'*accuracy* pour les différentes profondeur est respectivement (un jeu de données par ligne) de :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 2.        , 3.        , 4.        , 5.        ],\n",
       "       [0.54032258, 0.5       , 0.48387097, 0.58064516, 0.5       ],\n",
       "       [0.62130178, 0.62130178, 0.62130178, 0.62130178, 0.62130178],\n",
       "       [0.62295082, 0.69672131, 0.87704918, 0.51639344, 0.50819672]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.vstack((choix_n_couches,accuracy_cum2,accuracy_cum3,accuracy_cum4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de couches cachées qui maximise l'*accuracy* est, pour chacun des 3 jeux de données, respectivement de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 1, 3)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_couches_optimal2 = choix_n_couches[np.where(accuracy_cum2==max(accuracy_cum2))[0][0]]\n",
    "n_couches_optimal3 = choix_n_couches[np.where(accuracy_cum3==max(accuracy_cum3))[0][0]]\n",
    "n_couches_optimal4 = choix_n_couches[np.where(accuracy_cum4==max(accuracy_cum4))[0][0]]\n",
    "(n_couches_optimal2,n_couches_optimal3,n_couches_optimal4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Courbes d'apprentissage\n",
    "\n",
    "Ce graphique présente les courbes d'apprentissage pour chacun des niveaux de profondeur du réseau\n",
    "\n",
    "- MONKS1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xtc1FX6wPHPYYDhjnJTQUUsJAXvpLWtyWqZ7baWaaZWattF2+y+pbVt21Za230v1WbqqmXZPd02ddfK6qdlXlBEFDFE5Sb34TIw1/P7YwYWcdABBmYGzvv14iV853s5g8w88z3nOc8RUkoURVEUxcfdDVAURVE8gwoIiqIoCqACgqIoimKnAoKiKIoCqICgKIqi2KmAoCiKogAqICiKoih2KiAoiqIogAoIiqIoip2vuxvQFlFRUXLQoEHuboaiKIpX2bt3b5mUMvp8+3lVQBg0aBB79uxxdzMURVG8ihDihDP7qS4jRVEUBVABQVEURbFTAUFRFEUBvGwMwRGLxUJFRQUmk8ndTVEURXErPz8/IiIi0Gg07Tre6wNCRUUFAQEBREZGYrFYsFqt7m6SoihKl5NSotfrKSsro0+fPu06h9cHBJPJREREBD/99BPl5eUIIdzdJEVRFLdoDApSSvr27dvm470+IACcPn2asrIygoODVUBQFKVHa2hoYOPGjdx+++1t7jrqFoPK9fX1+Pr6qmCgKN2A2SopqW7AYFbdv+3h6+tLQ0MD9fX1bT62WwQEd68L/cQTTzBx4kSmT5/uluvv3r2bxYsXu+XaiuIqRouVkxV60k9WkltWR2FV29/Q2spisTBr1iy3vH42btzI8uXLO+Xc7f1w3C0CgrtNmzaNN954w93NUBSv1GCykFtWx/6TVRRW1RMe6EeQv4Y6g7nTr71+/XoSEhI6/TreQgUEF0hNTSU8PPyc+5SXl3P//fczc+ZMZs6cyf79+wFYt24d06dPZ/r06bz99tsAFBQUnHG3sWbNGl5//XUATp48yR133MHMmTOZNWsWp06dAkCv1/Pggw8ybdo0li5d2nTXlJWVxa233sqNN97IokWLKC0tBWwvhOuuu44ZM2bwyCOPuPYXoihOqDOYyTldw4FTVZTVGIgK8WfkgF4M6RNKeKAfeqOlU+/+i4uL+fbbb7n++utb3cfR601KyUsvvcT06dO5/vrr2bJlC3D2nfry5cvZuHEjAJmZmdxyyy3MnDmTuXPnUldXB0BpaSmLFi3immuu4eWXX246dufOndx8883MmjWLhx56CL1eD8Crr77a9Lp98cUXXf476RaDyo3e2lNJbqXRpecc3NufO1J7d/g8zz33HGPHjuXVV1/FYrGg1+vJysris88+Y/369QDcdNNNpKamEhYW1up5li5dym233cbkyZMxGAxYrVaKi4s5cuQIn3zyCTExMcybN4/09HSGDx/Os88+y1/+8hciIiLYsmULf/vb33jqqadYvXo1mzdvxt/fn+rq6g4/P0Vxlq7eRGFVPbp6ExofQcrWTwgpysenWTdHf4uVKJMFofVF047uDzloENZbF5xzn+eff54HH3yw6c3ZEUevt23btpGdnc1HH31EVVUVc+bMYezYsa2ew2Qy8fDDD/PCCy+QkpJCbW0tWq0WgCNHjvDBBx/g7+/PtGnTmDt3LlqtlhUrVrBixQqCgoJYvXo169atY86cOXz55Zds2rQJIUSnvG67VUDwZD/++CPLli0DQKPREBoayr59+5g8eTJBQUEATJ48mX379pGWlubwHHV1dZSUlDB58mSApj8qgJSUlKY0s6SkJAoLCwkNDeXYsWMsXLgQsPWXRkfbCh4mJiaydOlSJk2axKRJkzrlOStKIymhUm+ksKqeWoMZP40PAyKC6BMWgH+A31l93hof288Wq0SjcX2yyDfffENERATDhg1j9+7dDvdp7fWWnp7O1VdfjUajITIyktTUVDIzMwkJCXF4nuPHjxMdHU1KSgrAGfuNHz+e0NBQAAYPHkxhYSE1NTXk5uYyf/58wBZQRo4cSXBwMFqtlieffJIJEyYwceJE1/wymulWAcEVn+S7Umu3wxqN5ozHjEbjOfcH8Pf3P+N4i8V2u33BBRfwzjvvnLX/a6+9xt69e9m+fTtvvvkmn376Kb6+3erPQfEAVikpqzVSVFVPvclCgK+GhKhgokO1TXcEjj7JSyk5lFdJnzAt8ZHBLm/X/v372b59O//3f/+HwWCgrq6ORx99lGefffaMNjhyrtdt84mxBoOh6fvWBnlbe91ecsklPP/882ft/+6777Jr1y42b97Me++9x6pVq879RNtIjSF0kfHjx/P+++8Dtk/qtbW1jB07lq+++or6+nr0ej1ffvklY8aMITIykoqKCqqqqjAajXzzzTeA7ZNFnz59+OqrrwBboDhXallCQgKVlZUcOHAAsH3SOHbsWFM307hx43jggQeoqalp6qNUFFewWCVFVfXsP1lFbmktPkJwYUwIIweE0ycs4IzuIUeEEPaBZUuntO++++5j27ZtbNmyheeff55x48adEQyg9dfb2LFj2bp1a1PZnL179zJ8+HBiY2PJzc3FaDRSU1PDrl27ANvrsKSkhMzMTMB252E2tz5gPmLECPbv38/JkycBW1p9Xl4eer2empoaJkyYwJIlS8jOznb578Wpj4RCiKnAXwANsFJK+ZyDfWYBTwISOCClnCuE+AXwSrPdLgJmSyk/E0KsASYCOvtjC6SU+9v7RNzpkUceYc+ePVRVVXHFFVfw29/+9qyBqiVLlvCnP/2JTz/9FI1Gw+OPP87IkSO59tprmTt3LgDXX389Q4cOBWDhwoXcdNNNxMXFnZEFsXz5cp566ilee+01fH19eemll1ptl5+fHy+99BLPPfcctbW1WCwWbrrpJuLj43nssceoqalBSsktt9xyznELRXGWyWKlWNfA6eoGzFZJWIAfg6ND6BXk1+ZzBWt9Kas1ICW4a4qRo9fb5MmTOXDgADNnzkQIwQMPPEBUVBQAU6ZMYcaMGcTHxze9lv38/HjhhRd49tlnMRgMaLVa3nrrrVavGRERwdNPP82SJUuaegcWL15McHAw9913HwaDASklDz/8sMufrzjfKL4QQgMcBa4E8oHdwBwpZVazfRKBD4BJUspKIUSMlLKkxXkigGNAfyml3h4QPpdSfuRsY1NTU2XLBXIKCwvR6/VUVlYSEBDg7KkURXEhg8lCoa6B0hoDVimJCPYnNjyQkID2d0OWVDeQW1bHqAG9CPBrX7G2nkin07Fv3z5+85vfNI1XCCH2SilTz3esM/9b44BjUspc+4k3ANcCWc32uQN4TUpZCdAyGNjNBDZLKVXfhKJ0E3qjmcKqBsprDSAgKkRLbHgggf4dfwMP1trenuoMFhUQuogzYwhxwKlmP+fbtzU3BBgihNghhPjB3sXU0mzgvRbblgkhMoQQrwghtA6OURTFA1U3mDhSXE1Gvo7KOiN9wwMYPaA3F0SHuCQYAAT6axAC6oydP0FNsXHmDsFR713LfiZfIBFIA/oD3wkhUqSUVQBCiH7AcGBrs2MeBYoBf2AFsAR46qyLC3EncCfAwIEDnWiuoiidQUqo0hsp1NVT02BPHe0dRJ8wLb4a1+en+AhBkJ9vl8xYVmycCQj5wIBmP/cHCh3s84OU0gQcF0JkYwsQjQm+s4BP7Y8DIKUssn9rEEL8E/ido4tLKVdgCxikpqa6t2iRovRA0p46WmhPHdX6+jAoMpiYUC0+Pp072huk1VClV4tfdRVnwvpuIFEIkSCE8MfW9bOpxT6fAb8AEEJEYetCym32+BxadBfZ7xoQtgTd64DM9jwBRVE6h8UqKdI1sP9UFT+V1iIEXBgTwqgBvegbHtDpwQBs4wgmixWjqnzaJc57hyClNAshFmPr7tEAq6WUh4QQTwF7pJSb7I9NEUJkARbgYSllOYAQYhC2O4xvWpx6vRAiGluX1H5gkWuekqIoHWG2WCmubqBYZ0sdDQ3wIyEqmF5B/uc/2MWC/RsHls34+3b99Xsap3LCpJRfAF+
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "for i in choix_n_couches:\n",
    "    plt.plot(range(10),lc_cum2[i-1], label=\"%d couches\"%(i,))\n",
    "leg = plt.legend(loc='best', ncol=2, mode=\"expand\", shadow=True, fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MONKS2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xlc1VX++PHX4bIjqCxuoGyi4oYLuZRrmmkJpplTNm3TtHybpqbFpX7VlO1ZM83SLLZM40zNNNNogZmWkqbmDm6AqIAim6CAsl3g3nt+f1wkVJDtst738/HgAdx7Pp9z7tX75vM5y/sorTVCCCHsg0N7N0AIIUTbkaAvhBB2RIK+EELYEQn6QghhRyToCyGEHZGgL4QQdkSCvhBC2BEJ+kIIYUck6AshhB1xbO8GXM7X11cHBQW1dzOEEKJT2b9//1mttV9D5Tpc0A8KCmLfvn3t3QwhhOhUlFKnGlNOuneEEMKOSNAXQgg7IkFfCCHsiAR9IYSwIxL0hRDCjkjQF0IIOyJBXwgh7IgEfRsrNBby2dHPSClIae+mtLsKcwXfnvqWTac2IdtygsliIjY1lm2Z2zBZTO3dHGGnOtzirM5sb+5eln+/nLzyPADCeoYRFRLFzSE308u9Vzu3rm1orUnISyA2LZaN6RsprioGYMaAGbx07Ut0d+nezi1sH7mluSz7fhnxefEA+Lj6cFPITUSFRDHEewhKqXZuobAXqqNdgUVGRurOtiLXZDHxl4N/YdWhVQR6BfLCxBdIK0ojJi2GQ/mHcFAOjO8znqjQKGYMmIG7k3t7N9nmMi5kEJsWy7rUdWSWZOLm6MbMATOZGzqX44XHeTf+XXzdfHlz8puM6T2mvZvbpuIy4nh+x/OYLCaeHf8s3Zy7sS51HVsyt2CymBjYYyBRoVHcHHwzvT16t3dzRSellNqvtY5ssJwE/ZbJKclh2bZlJOQlMC90Hs+Of/aSoH7y/EnWpa1jXdo6skqycHN044bAG5gbMpdxfcZhcDC0Y+tb5nzFeTae3EhMagwH8w+iUIzvO57o0Ogr/rglnk1kyfdLyCrJ4uGIh3lwxIOd+rU3RoW5grf3vs2/U/5NuHc4K6euJNArsOb5ImMRG09uJDYttsH3T4iGSNBvA5tObeKFH17Aoi08N+E55obMrbesRVus3R6psXxz8huKq4rp5d6LuSFziQqJYmDPgW3Y8uarMlfxfdb3rEtdx9bMrVRZqmquVG8Kvok+Hn3qPbaksoRXdr/CV2lfEdk7ktcnv37V8p1ZWlEaS75fwrHCY9w99G4eH/M4zgbnesufunCKdWnriE2Nrbk4uHinNL7P+C7/B1K0nAT9VmQ0GXl739t8lvIZw3yGsXLKSvp79W/S8Vsyt7AudR3bs7Zj1mbCvcOJCo1iTvAcfN18W7H1Tae15vDZw8SkxrDx5EaKKorwdvXm5pCbm9wnrbUmJjWGV3e/irPBmZevfZnpA6a38itoO1pr1p5Yyxt73sDV4Mork15hSsCUJh2fkJdATGrMjxcHbr2s73VoFGE9w1qx9aIzk6DfSlKLUlny/RKOFx7nnqH38PiYx3EyODX7fOfKz7Hh5AZiUmNIOpeEQRm4tt+1RIdGM63/NFwdXW3Y+qbJKsliXaq1a+rkhZO4GFy4vv/1RIVGMbHfRBwdmj8PIP18Oku/X8rRgqMsHrKYJyOfxMXgYsPWt73iymJW7FzBhpMbGN9nPK9Nfq1FA/gV5gq2nt5KbGos27O2Y9ImhngPISokiptCbupwFweifUnQtzGtNf87/j/e3PMm7k7uvHLdK0wOmGzTOlKLUolNjWVd2jrOlJ2hm1M3ZgXNYm7IXMb2HouDav0ZtsWVxXxz8hti02LZf2Y/ANf0uYaokChuCLyBbs7dbFZXpbmS3+7/Lf9M/idDvIfw1pS3CO4ebLPzt6VD+YdY+v1Scktz+cWoX/Cz4T+zaZdMgbGAr9O/JjY1lsRziRiUgYn9JhIdGs30/tPb9eJAdAwS9G3oQuUFXvrhJb459Q0T+k7gtUmv4efe4F4FzWbRFvbm7iU2NZZvT31LmamMfh79am7xbR0YqyxV7MzeSUxqDFtOb6HCXEGQVxDRodHcHHIz/br1s2l9l9t6eivP7XiOCnMFz45/lnmh8zrNFEaLtvC3I3/jjwl/pJd7L96c8iajeo1q1TrTitKsM6XS1pFbmouHkwezAmcRFRrVZhcHouORoG8jB/IOsOz7ZeSV5fHo6Ee5b/h9bfqhKqsq47vT3xGbGsvOnJ1YtIURviOICo1idtBserr2bNZ5tdYkFyQTmxrL+vT1FBgL6OHSgznBc4gOjWaYz7A2DbxnSs/wzPZn2Ju7l5uCb+L5Cc/b9K6iNZwtP8uz255lZ85Obgi8gRevfREvZ682q9+iLezL3UdsmnVyQJmpjL4efZkbMpe5oXMJ6R7SZm0R7U+CfgtZtIWPjnzEHxP+SB+PPrw55U0i/CLatU35ZfmsT19PbGosKYUpOCpHJgdMJio0iqkBU686O+Si3NJcvkr7itjUWFLPp+Lk4MS0/tOIColikv+kFo1PtJTZYuaDwx/wp4N/wr+bP29NeYvhvsPbrT1XsyNrB89uf5bSqlKWjVvGwrCF7Xp3Um4q57uM74hJi2FntvXiYLjP8JrJAc29OBCdhwT9Fsgvy+eZ7c+wO2c3NwbdyAsTX2jTK7jGSClIYV3aOr5K+4r88ny8nL2YHTSbqNAoIvwiLglApVWlbDq1idi0WPbk7EGjGd1rNFGhUcwKnNXhVsnGn4ln2bZlnC07y+NjHufuYXd3mC6LKnMVv0/4PR8nfszAHgNZOWVlh5tue/HiYF3aOo4WHMVROTIpYBJRIVFM7T+10w+Yi7pJ0G+mbZnbeG7Hc5RVlbF83HIWhC3o0P3LZouZ3Tm7iUmLIS4jjnJTOf09+xMVEsVg78F8c+qbKx6fGzK3SVNM28P5ivP8+odfszljM9f1u45XJr3S7rNVTl84zdLvl3Lk3BEWDVrEkmuWdPgB1GOFx2pmYOWX5+Pp7MmNQTdyU7DM/qmtu0t3vF2927sZLSJBv4mqzFW8G/8uq5NWE9YzjJVTVhLaI7TN29ESdV3Rezp7MidoTp13AB2d1pr/Hvsvb+55E09nT16b/BrX9ru2XdqyPm09K3atwEE58NK1L3FD4A3t0o7mMlvM7M7dTWxqLJszNlNuKm/vJnUoF1dDR4VGMXPAzE65GlqCfhNkXMhgyfdLSDqXxE8G/4SnI5/u8FdwDcktzSX9fDpje49tVF9/R3as8BhLti4h/Xw69w2/j0dHP4qTQ9uMPZRVlfH6ntf54sQXjPIbxZtT3mz12UytrayqjF05uzCajO3dlA4j/UL6JXmjZgyYQVRoVKdaDS1Bv5FiU2N5ZdcrODo4suLaFcwInNFmdYvGKzeV8+aeN/nf8f8x0nckb055kwDPgFat82jBUZZsXcKpC6d4YOQD/F/E/7VoQZro2LTWHMg/ULPyvLjyx9XQc0PnMqjnoPZu4lVJ0G9AWVUZr+5+lZjUGMb0GsMbk9+gb7e+rV6vaJmNJzfy0g8vodH8euKvmR082+Z1aK359OinvLPvHXq69OT1ya8zru84m9cjOq6a1dBpsWzP/HE19NyQudwccnOHHA+RoH8VyeeSWfr9UjKKM3hw5IM8NPIhuYLrRLJKslj6/VIO5R/i1rBbWXrNUpv1wRYZi3j+h+fZcnoLUwOm8vJ1L8t0RztXYCxgQ/oGYlNjOXLuCA7KwboaOiSa6QOm4+bo1t5NBCTo10lrzSfJn/Cb/b+hp2tP3pj8Btf0uaZV6hKtq8pSxZ8O/IkPD39IUPcgVk5ZyWDvwS06597cvSzftpxCYyFPjn2SO8Pv7FQD36L1pZ1Pq5kNlVOag4eTBzcE3kBUSBSRfSLbdWqxBP3LFBoLeX7H82zN3Mq0gGmsuG6FXMF1AbtydvHMtme4UHGBp695mtsH397kQG2ymPjrob+y6tAq+nv2560pbzHUZ2grtVh0BRZtYf+Z/dZU6ae+obSqlL4
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "for i in choix_n_couches:\n",
    "    plt.plot(range(10),lc_cum3[i-1], label=\"%d couches\"%(i,))\n",
    "leg = plt.legend(loc='best', ncol=2, mode=\"expand\", shadow=True, fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MONKS3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XlcVNX/+PHXYdhkUxY3XAAR913ctUXNNM0V9y2t1MoW62tan35l9vlUn/blY5apmfuaaaVpWlnu4IKigiKLLKLIvi8z5/fHIIGADDAwg57n48EDmHvunfeg9z1nzj33fYSUEkVRFOX+YGHqABRFUZSao5K+oijKfUQlfUVRlPuISvqKoij3EZX0FUVR7iMq6SuKotxHVNJXFEW5j6ikryiKch9RSV9RFOU+YmnqAO7k5uYmPT09TR2GoihKrXLq1KlbUsr65bUzu6Tv6elJQECAqcNQFEWpVYQQkYa0U8M7iqIo9xGV9BVFUe4jKukriqLcR8xuTL80Wq2WxMRE8vLyTB2KoiiKSVlZWeHi4oJGo6nU/rUi6ScmJmJra4ubmxtSSvLz81HrACiKcr+RUpKZmcmNGzdo3LgxQogKH6NWJP28vDzc3NzIysoiODhY9fgVRbmvpaamcuLECYYPH461tXWF9q0VSf+2kJAQdDoddnZ2pg5FURTFZPLy8ggKCuLkyZP079+/QvvWmgu5Op2OnJycCr+rKYqi1AaJ2YkkZCUY3N7e3p74+PgKP0+tSfq3VWYMyxjefPNNHnzwQcaMGWOS5/f392f+/PkmeW5Fqe20Wi0TJkwwyTm0a9cu3n333bu2ycrPIiIlgviseCSGX6+szLXNWpf0TWXkyJEsX77c1GEoilIJGzZswMvLy9RhlCpfl09oUigWwgLvut4Iqrdjq5K+gXx9falbt+5d2yQkJPDSSy/h5+eHn58fZ8+eBWDt2rWMGTOGMWPGsG7dOgBiYmKKfWpYs2YNX331FQDXrl3j6aefxs/PjwkTJhAVFQVAZmYmL7/8MiNHjmTx4sWF7/IXL15k1qxZTJw4kXnz5hV+5NuwYQOjR49m3LhxvPrqq8b9gyhKLREXF8dff/3F2LFjy2xT2jknpeTjjz9mzJgxjB07ll9//RUo+an73XffZdeuXQAEBQUxffp0/Pz8mDJlChkZGQDEx8czb948RowYwSeffFK475GjR5g0ZRKvzX2N7977jrwc/SSVzz77rPDc/eijj4z696hVF3IBvg1IIiwp16jHbOFszdO+zlU+zvvvv0/37t357LPP0Gq1ZGZmcvHiRX788Uc2bNgAwNSpU/H19cXJyanM4yxevJgnn3ySQYMGkZOTg06nIy4ujuDgYH744QcaNGjAjBkzOHPmDB07duS9997j888/x8XFhV9//ZUvv/ySpUuXsnr1avbu3Yu1tTWpqalVfn2KUhUW361BREQY9ZjS0xPdrCfu2uaDDz7g5ZdfLkzApSntnDtw4AAhISFs376d5ORkJk+eTPfu3cs8Rl5eHgsXLuTDDz+kQ4cOpKenY2NjA0BwcDBbt27F2tqakSNHMmXKFGxsbFi2fBkvvfcSPg19+GXzL6xdu5bJkydz8OBBdu/ejRDC6OdurUv65uzkyZP85z//AUCj0eDo6Mjp06cZNGhQ4YyjQYMGcfr0aR566KFSj5GRkcHNmzcZNGgQQOF/GoAOHTrQqFEjAFq3bk1sbCyOjo6EhoYyd+5cQD92Wb++vtCej48PixcvZuDAgQwcOLBaXrOimLNDhw7h4uJCu3bt8Pf3L7VNWefcmTNnGDZsGBqNBldXV3x9fQkKCsLBwaHU44SHh1O/fn06dOgAUKxdr169cHR0BKBFixbExsYSlxhHRHgEH7zyATYaG/Ly8ujcuTP29vbY2NiwZMkSBgwYwIMPPmi0vwfUwqRvjB55TSrrQotGoym2LTc3967tgWIzlzQaDVqtFikl3t7erF+/vkT7ZcuWcerUKf7880+++eYbdu7ciaVlrfsnV+4R5fXIq8PZs2f5888/OXz4MDk5OWRkZPDaa6/x3nvvFbYp65y727mr0+kKf8/JySn8uayJJneeu9l52cRlxNGxe0eWf7YcC1F8pH3jxo2cOHGCvXv3smnTJlatWlX+izWQGtM3ol69erFlyxZA3+NOT0+ne/fu/P7772RlZZGZmcnBgwfp1q0brq6uJCYmkpycTG5uLocOHQL0vYOGDRvy+++/A/o3g6ysrDKf08vLi6SkJAIDAwH9R8zQ0NDCIaGePXuyYMEC0tLSyMzMrOa/gKKYlxdffJEDBw7w66+/8sEHH9CzZ89iCR/KPue6d+/Ovn37CsvAnDp1io4dO+Lu7k5YWBi5ubmkpaVx4sQJQH8u3rx5k6CgIED/CSI/P79ETFJKYtJi8G7rTdilMKKjogHIysoiIiKCzMxM0tLSGDBgAIsWLSIkJMSofxPV7TPQq6++SkBAAMnJyQwePJhnn322xIWhRYsW8fbbb7Nz5040Gg1vvPEGnTt3ZtSoUUyZMgWAsWPH0rZtWwDmzp3L1KlTadKkSbGZBe+++y5Lly5l2bJlWFpa8vHHH5cZl5WVFR9//DHvv/8+6enpaLVapk6dioeHB6+//jppaWlIKZk+ffpdryMoyv2stHNu0KBBBAYG4ufnhxCCBQsW4ObmBsCQIUMYN24cHh4eheezlZUVH374Ie+99x45OTnY2Njw7bfflniurPwscrQ5dPHowr/f+TeLFi0q/KQ/f/587O3tefHFF8nJyUFKycKFC436WoW51bDx9fWVdy6iEhsbS8OGDfH398fe3t5EkSmKolTNjcwbXEu9RhOHJrg7uFf6OCkpKVy8eBE3N7fCzqcQ4pSU0re8fdXwjqIoSg1Iy00jKjWKejb1aOzQ2GRxqKSvKIpSzfK0eVxNvoqNxgavul7VfgPW3RiU9IUQQ4UQIUKIUCHE4lK2NxdC/CGEOCOEOCeEeKzgcU8hRJYQ4mzB19fGfgGKoijmTEpJaEooOqmjpXNLLC1Meym13GcXQmiAZcAjQDTgL4TYLaW8WKTZG8BWKeVyIUQ7YA/gWbDtqpSyi3HDVhRFqR2i0qJIz03Hu543dSzrmDocg3r6PYFQKWWYlDIX2AyMuqONBG5PDakLxBovREVRlNopISuBG5k3aGjfEBdbF1OHAxiW9JsAUUV+jy54rKglwDQhRDT6Xv7zRbZ5FQz7HBJCDCjtCYQQc4QQAUKIgMqUClUURTE3mXmZRKRG4GjtSDOHZqYOp5AhSb+0Kw53zvOcDKyRUjYFHgPWCSEsgOtAcyllV+BlYKMQosRkcSnlCimlr5TS93YJAXMSFxfHk08+yahRoxgzZkypd79WN1VaWVEqZ+jQoYwdO5bx48czadKkGnnOfF0+ocmhWFpYknQlieeff778nWqIIVcUooGib1NNKTl88yQwFEBKeUwIYQu4SSlvAjkFj58SQlwFWgEB1CIajYZXXnmFdu3akZGRwaRJk+jTpw/e3t6mDk1RFAOsWrUKZ+eaKeEipSQsJYxcXS5tXNpwyeJSjTyvoQzp6fsDPkIILyGENTAJ2H1Hm2vAIAAhRFvAFogXQtQvuBCMEKIF4AOEGSv4mlK/fn3atWsH6FeruX279Z1UaWVFqZ2Mee7OfHImL816iXefe5ekuCTAvM7dcnv6Usp8IcR8YB+gAVZLKS8IIZYCAVLK3cArwLdCiAXoh36ekFJKIcQDwFIhRD6gBeZJKROrEnDjwM+xTb5SlUOUkF3Ph+udXzSobUxMDMHBwXTs2LHENlVaWVHKdvXvdNJvlaxFUxUObpZ4Dyi96mVRc+fORQjB+PHj8fPzK7HdWOfuwkULGThuIEMeGYK7rbtZnrsGTRiVUu5Bf4G26GNvFvn5ItCvlP12ADuqGKPZuP1u/eqrr5ZaXlWVVlYU87N27VoaNGhAQkICc+fOxdPTE1/f4tUKjHHuJqQkEHcjjv4P98fDyaNY5UxzOndrXcE1Q3vkxpaXl8fLL7/M8OHDGTx4sMH7qdLKiqJnSI+8OjRo0AA
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "for i in choix_n_couches:\n",
    "    plt.plot(range(10),lc_cum4[i-1], label=\"%d couches\"%(i,))\n",
    "leg = plt.legend(loc='best', ncol=2, mode=\"expand\", shadow=True, fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choix des poids initiaux\n",
    "Les poids initiaux ont été choisis à partir d'une distribution uniforme sur $[-1,1]$. On compare ici les courbes d'apprentissage en initialisant les poids à 0 et en initialisant les poids aléatoirement, pour le réseau de dimension et de profondeur optimale sélectionnées précédemment.\n",
    "\n",
    "- Réseau initialisé avec les poids à 0 $RN_{0}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn2_poidszero = NeuralNet.NeuralNet(\n",
    "    np.hstack((6,\n",
    "               np.repeat(n_neurones_optimal2,n_couches_optimal2),\n",
    "               2)),\n",
    "    range(2),\n",
    "    input_weights=0)\n",
    "lc_nn2_poidszero = (\n",
    "    nn2_poidszero.train(train2, train_labels2, 0.1, 10, verbose=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn3_poidszero = NeuralNet.NeuralNet(\n",
    "    np.hstack((6,\n",
    "               np.repeat(n_neurones_optimal3,n_couches_optimal3),\n",
    "               2)),\n",
    "    range(2),\n",
    "    input_weights=0)\n",
    "lc_nn3_poidszero = (\n",
    "    nn3_poidszero.train(train3, train_labels3, 0.1, 10, verbose=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn4_poidszero = NeuralNet.NeuralNet(\n",
    "    np.hstack((6,\n",
    "               np.repeat(n_neurones_optimal4,n_couches_optimal4),\n",
    "               2)),\n",
    "    range(2),\n",
    "    input_weights=0)\n",
    "lc_nn4_poidszero = (\n",
    "    nn4_poidszero.train(train4, train_labels4, 0.1, 10, verbose=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Réseau initialisé avec les poids uniformes $RN_{\\neg{0}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn2_poidsunif = NeuralNet.NeuralNet(\n",
    "    np.hstack((6,\n",
    "               np.repeat(n_neurones_optimal2,n_couches_optimal2),\n",
    "               2)),\n",
    "    range(2))\n",
    "np.random.seed(12345)\n",
    "start_time = time.time()\n",
    "lc_nn2_poidsunif = (\n",
    "    nn2_poidsunif.train(train2, train_labels2, 0.1, 10, verbose=False))\n",
    "nn2_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn3_poidsunif = NeuralNet.NeuralNet(\n",
    "    np.hstack((6,\n",
    "               np.repeat(n_neurones_optimal3,n_couches_optimal3),\n",
    "               2)),\n",
    "    range(2))\n",
    "np.random.seed(12345)\n",
    "start_time = time.time()\n",
    "lc_nn3_poidsunif = (\n",
    "    nn3_poidsunif.train(train3, train_labels3, 0.1, 10, verbose=False))\n",
    "nn3_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn4_poidsunif = NeuralNet.NeuralNet(\n",
    "    np.hstack((6,\n",
    "               np.repeat(n_neurones_optimal4,n_couches_optimal4),\n",
    "               2)),\n",
    "    range(2))\n",
    "np.random.seed(12345)\n",
    "start_time = time.time()\n",
    "lc_nn4_poidsunif = (\n",
    "    nn4_poidsunif.train(train4, train_labels4, 0.1, 10, verbose=False))\n",
    "nn4_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graphique des courbes d'apprentissage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MONKS1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4FGW2+PHvyUbY1wikwyayCAohYRiBUXFj4KqooIAsigs410FcrjM/9OLogOM6ZtRxmSG4IHgRRNxBkHFhRoMsQtCAEUQJEJZAFJAlIcn7+6O622ykO0knVdV9Ps+TJ+nq6uqTfrpP3px66z1ijEEppVRkiLI7AKWUUvVHk75SSkUQTfpKKRVBNOkrpVQE0aSvlFIRRJO+UkpFEE36SikVQTTpK6VUBNGkr5RSESTG7gDKa9OmjencubPdYSillKusX7/+gDEmIdB+jkv6nTt3Zt26dXaHoZRSriIiO4LZT8s7SikVQTTpK6VUBNGkr5RSEcRxNf3KFBcXk5+fz8mTJ+0ORSmlbBUbG0urVq2Ijo6u0eNdkfTz8/OJj4+nTZs2GGMoKipC+wAopSKNMYZjx46xb98+2rdvj4hU+xiuSPonT56kTZs2HD9+nG+++UZH/EqpiHb48GG++OILLr30UuLi4qr1WFckfZ/s7GxKSkpo1KiR3aEopZRtTp48yddff82aNWv4zW9+U63HuuZEbklJCQUFBdX+q6YUACcOQeHPdkehVMg0btyYvLy8aj/ONUnfpyY1LKXIy4aD2+2OQqmQqsm5Tdclfbv06dOHe++913+7qKiI888/n6lTp/q3ffTRR4waNYoRI0YwcuRIPvroI/99M2bM4OKLL6awsBCAH3/8kWHDhgGwe/durrrqKv++ixcvZvTo0Rw+fJjMzEzGjRvHNddcwxVXXMFzzz0HwPfff8+ECRNITU3l5ZdfDup3mDFjBsOGDeOaa67h6quvZvXq1f77brzxRsaOHeu/nZWVxY033njKYz399NNcc801/q/LL7+c5ORkjh07FlQs9aroBBQXWiP9MJ4AoO/RstauXUufPn345JNP/NumTp3K2rVrg4olGPfdd1+Zz8GwYcMYMmRISI797bffMmPGjJAcqzRX1fTt1LBhQ7Zt28aJEyeIj48nIyOD0047zX9/dnY2TzzxBP/85z9JSkpi165d3HLLLSQlJdG9e3cAoqKiePPNNxkzZswpn+fdd99lwYIFzJkzh2bNmjFjxgz++te/0qNHD4qLi/nhhx8AaNasGdOnTy/zoQ3GXXfdxdChQ1mzZg0zZ87kvffe89+Xn5/Pv//9b84999yAx5k2bRrTpk3z354+fTpDhw4N+nxLUVERMTH19PY7cdj6bkrg5DGIa1w/z1vP9D1aUdu2bUlPTw9ZIi5v1qxZ/p9LSkq48cYbufzyy4N+fFWfg+7du7Nv3z727NlD+/btax2rj+uSfvq6H9n+Y2FIj3l6yzgm928ZcL/BgwezatUqhg4dyrJlyxg+fDhffvklAHPnzuXmm28mKSkJgKSkJG666SZefvllHnroIQAmTJjA/PnzGTVqVKXHX758OS+++CLp6em0bGnFk5+fT5s2bQCIjo6ma9euALRu3ZrWrVuzatWqGv3Offv2Zf/+/WW2TZo0ifT09KA/UD7vvfceOTk5PPjggwAcO3aMRx55hK1bt1JcXMx///d/c8EFF/D222+zatUqCgoKOH78OHPmzCEtLY3//Oc/iAhTpkzxjyxDquAwIICBgiN1nvTbZz5F/E9bQ3rMEy26safv7QH30/doWT169KCoqIiMjAwGDhxY5r7Vq1eTlpZGUVERZ511FjNmzCAuLo5hw4Zx+eWX8+mnn1JUVMQTTzxBly5dAj7XnDlzaNmypf+1y8/PZ9asWezduxeAP/7xj/Tr14/nnnuOvLw8cnNzadGiBTNnzuTBBx8kKyuLmJgY7r77bgYMGADA+eefz7Jly6r8j6a6tLxTDcOHD+eDDz6goKCArVu3cvbZZ/vv27ZtG7169Sqzf+/evdm2bZv/drt27ejXr1+ZkYvPnj17eOihh/jHP/7h/wABTJw4kREjRnDHHXfw+uuvU1BQEDDOW2+9tcKHpbzPPvuMCy64oMy2vn37Ehsby5o1awI+h8/u3bt58skneeSRR/wjlvT0dAYMGOAfDaalpfnLPpmZmfzlL3/hhRdeYOXKlWRnZ7N48WLS09NJS0ur0YmpgE4choYtICom7E/m6nu0osmTJzN79uwy2woKCrjvvvt47LHHWLJkCUVFRSxcuNB/f8uWLVm0aBGjR48OqjT11VdfsWTJEh544AH/tkcffZSJEyeyYMEC0tLSyty3efNmnnrqKR599FFee+01AJYsWcKjjz7KjBkz/K9h7969/X+0Q8V1I/1gRuR1pXv37uTm5rJs2bJKp0mVP8lsjKmw7eabb2batGkVRiotW7akefPmrFixgokTJ/q3/+53v+PSSy/l888/Z+nSpSxbtowXX3yxyjh9NdXKpKWl8be//Y38/Hzmz59f4f4pU6Ywe/Zs7rzzziqfA6wrpe+9916mTp1Kx44d/dszMjL45JNPmDt3LmB9wHyjnYEDB9K8eXMANmzYwPDhw4mOjqZ169b079+fr7/+usIHvVaKT1olnSYJVj2/4Ejojn0KwYzI64q+RytKTU0FYP369f5tP/zwAx6PB98y7iNGjGDhwoX+3+uiiy4CoFevXvzrX/+q8vjHjh3j3nvv5YEHHvC/t8H6T2L79l8mDxw9epSjR48CMGTIEOLj4wHrc3DttdcC0KVLF9q3b8+OHTvo3r07rVq1CvlASEf61TRkyBCeeOIJhg8fXmZ7165dycrKKrNty5Yt/n91fTp27EiPHj1YsWJFme3x8fE899xzLFq0iPfff7/MfR06dGDMmDGkp6eTnZ3NTz/9VOP477rrLt5//32mTp1a6UmiX//61xQWFrJp06aAx5o9ezZt2rThyiuvLLPdGENaWhqvv/46r7/+OitWrOD0008HrLpz6f3qXIG3nt+gGTRoAoXHrNp+GNP3aEWTJ08mPT3dfzvQe883NTw6Opri4uIq93344YcZMmQI55xzTpntxhjmzZvn/xysXLmSxo2t0mKwn4OCggL/H4dQ0aRfTVdeeSW33HKL/8SXz/XXX88LL7zA7t27AavsMWfOHK677roKx5g8ebJ/FFxaq1ateP7553n66af57LPPAFi1apX/TZGTk0N0dDRNmzat1e8QFRXFhAkTMMb4n6d8fC+99FKVx8jMzOTtt9/m/vvvr3DfoEGDWLBggT/uLVu2VHqM1NRUli9f7l9baf369WXKESFx4jCIQIOm1hcm7Es8+h6taNCgQRw+fJjs7GzAGlHn5uaSk5MDWOelfP8RVMeKFSvIzs4uM6nBZ+DAgSxYsMB/+5tvvqn0GKmpqSxduhSw/gPZu3ev/z+QHTt2cMYZZ1Q7rqq4rrxjt3bt2jFhwoQK23v27Mkdd9zBbbfd5j8jf+edd9KzZ88K+55xxhmceeaZlSbDpKQknn76aX7/+9+TlpbGe++9x+OPP058fDzR0dE8/PDDREdHc+DAAcaOHcvRo0eJiopi/vz5vPXWWzRp0oRbb72VBx54oMzMjfJExP/BGTx4cJn7zj33XP9JulN5/vnnOXHiBDfddFOZ7Wlpadxyyy089thjjBo1CmMMHo+HZ555psIxLrroIjIzM7n66qsREe68884yteKQKDgMcU1BoqzvAAU/WyP/MKXv0cpNnjyZ22+3Sm8NGjRg5syZ3H333f4TuaNHj67W8QD+/ve/c+LECcaNG1dm+7x585g+fToPPfQQo0aNori4mNTUVO67774KxxgzZgyzZs1i5MiRxMTEMGvWLP9/GmvXrq32xIpAxGkLl/Xv39+U75yVm5tL27ZtWbt2rf/fI6UCMiWwIwOae6BlZ2vbzjUQ3xwSetgamlKBFBYWcsMNNzB37twK0zoPHTrE5s2badOmDSNHjgRARNYbY/oHOq6Wd1T4KjgCmLKj+gZNrZG+Ug6
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "plt.plot(range(10),lc_nn2_poidszero, label=\"MONKS1: RN_Zero\")\n",
    "plt.plot(range(10),lc_nn2_poidsunif, label=\"MONKS1: RN_Non_Zero)\")\n",
    "leg = plt.legend(loc='best', \n",
    "                 ncol=2, \n",
    "                 mode=\"expand\", \n",
    "                 shadow=True, \n",
    "                 fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MONKS2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VOXZ+PHvnZ19DYsZlqiAgghIQBYJoNbCz1e0oii+WFewVapiW1+12FpU3Kna4gK4tSpudUUQa0WCCAqoiSIiSFhCJhAIe/bk+f3xzIQkBDIJM3NmuT/XlSuZM2fOuZPM3HPmfjYxxqCUUio6xDgdgFJKqeDRpK+UUlFEk75SSkURTfpKKRVFNOkrpVQU0aSvlFJRRJO+UkpFEU36SikVRTTpK6VUFIlzOoDa2rdvb7p37+50GEopFVbWrFmzyxiTXN9+IZf0u3fvzurVq50OQymlwoqIbPFlPy3vKKVUFNGkr5RSUUSTvlJKRRFN+kopFUU06SulVBTRpK+UUlFEk75SSkWRkOun32ilh1j6wp/IP1DidCSUSBKLml1EmSQ4HYrjupRl07ZiN5lJaU6HolTI631CS/5yQZ+AniNykn5ZEenuFwmFFX9jMBiED5pf6nQojooxFUzbcz9tKgu4tuNbGNEPlko5zaekLyJjgCeAWGCeMebBOvaZANwDGCDTGHOFiPQHngZaAhXA/caY1/0Ue03N2iP37EUCcvAGenk8k7a/w6Sr/wqJLZyOxjnfzod3cwB47eJ20OEUhwNSStV76SUiscBsYCzQG5goIr1r7dMDuBMYbozpA9zquasQ+LVn2xjgcRFp7cf4Q9Pou6CoAFY+43Qkzqkog6UPQssUezvnK2fjUUoBvjXkDgY2GmM2GWNKgdeAC2vtMxmYbYzZA2CM2en5/pMxZoPn51xgJ1DvhEBhL2Ug9Dofvvg7FO1xOhpnfPsK7NkM5z8GSa0hZ5XTESml8C3ppwDbqt3O8WyrrifQU0SWi8hKTzmoBhEZDCQAPzc22LAy+i4o2QcrZjsdSfCVl8DSRyAlDXqOAdcgyNFJ9JQKBb4k/brK5LXbS+OAHsAoYCIwr3oZR0Q6A/8CrjHGVB5xApEpIrJaRFbn5+f7Gnto63Qa9L4IVj4Nh3Y7HU1wff1P2J8DZ/8JRKDLYNi5Dor3OR2ZUlHPl6SfA3SpdtsF5Naxz3vGmDJjTDawHvsmgIi0BD4EphtjVtZ1AmPMHGNMmjEmLTk5gqo/o++CskJY/rjTkQRPWRFkPApdh8GJo+02VxpgYPvXjoamlPIt6a8CeohIqogkAJcD79fa511gNICItMeWezZ59n8H+Kcx5k3/hR0mkntB30vhq7lwYIfT0QTHqufgYB6cPd1e5YNt40C0xKNUCKg36RtjyoGpwGJgHfCGMWatiMwQkXGe3RYDu0XkB2AJ8EdjzG5gApAOXC0i33q++gfkNwlVI/8PKkrh81lORxJ4JQft73niKOg+/PD2pFb2DVB78CjlOJ/66RtjFgILa237c7WfDXCb56v6Pi8DLx9/mGGs3UnQ/wpY/TwMuxla1W4DjyBfPQuFu2H09CPvcw2CHxeAMYc/ASilgk6HSAbDyNttslv2qNORBE7xPlj+JPT4JXQZdOT9rkG2+2rBpuDHppSqokk/GFp3hYFX2V4tezY7HU1grHgKivfaxuu6uDxvBNu0xKOUkzTpB8uI34PE2v7rkaawAFY+BadeACccpckm+RRIbKmDtJRymCb9YGl5Agy6HjJfhV0bnY7Gv754EkoOwKijXOUDxMRAyhma9JVymCb9YDprGsQl2TlpIsXBfPjyWThtPHTsfex9XYNgx1ooPRSc2JRSR9CkH0zNk2HwFPjuLTtCNRIsfxzKi2HUHfXv6xoMpgJyvwl8XEqpOmnSD7bht0BCc1gy0+lIjt9+N6yaB6dfDu171L+/y7OQipZ4lHKMJv1ga9oWht4I694Hd6bT0RyfZY9BZbntkuqLpm2h7Uk6MlcpB2nSd8KQG+10w+F8tb93K6x5EQZMgrapvj+uy2DbbdOEwhpnSkUfTfpOaNIahv0OfvoofK96Mx6xI2vT/9iwx7nS4NBO+6ahlAo6TfpOOfM30LQdLLnf6UgabvfP8M0rMPAaaOVq2GO9g7S0rq+UIzTpOyWxue3C+fOnsOULp6NpmKUPQ2w8jLit/n1r69AH4ptq0leqtjUvwpdzAn4aTfpOSrsOmneET+8Pnxp3/k/w3RsweDK06NTwx8fGwQk6SEupI6x5EX78IOCn0aTvpISmdnqGLZ9D9lKno/HNZw9AXBMYfmvjj+FKA3cWlBX7Ly6lwllFGez4ATqdHvBTadJ32sCroaULPr0v9K/2876HtW/DkN9Cs/aNP45rEFSWhX+XVaX8ZddPUFECnfsF/FSa9J0Wlwjpf7Dljg3/cTqaY/vsAUhsBcOmHt9xtDFXqZrcWfa7Jv0oMWAStOkOS0L4an/713YRlKE3QZM2x3esFh3tdNOa9JWy3Jm2g0O7kwN+Kk36oSA23i6r6M60iTUULZlpk/2Q3/rneK5B4TtGQSl/y8uCjn0gJjbgp9KkHyr6TrDv8ktmQmWl09HUtPVL2PgfO29QUkv/HNM1GPbnwP5c/xxPqXBVWQl53wWlERc06YeO2DgYdSfs/ME2loaSJfdBM88Mof6idX2lrD3ZULI/KPV80KQfWvpcDB16w2cPQkW509FY2Rn266zbIKGZ/47bqS/EJmrSVyrP24irV/rRJybGrjG7e4MdAOU0Y+zAsRadIe1a/x47LsEurbhNk76Kcu4siImzF3xBoEk/1JzyP/Zj3tKH7IANJ/38X9i20nYpjU/y//Fdg8D9LZSX+v/YSoWLvCxIPtV23w4CTfqhRgRG/wn2bIZvX3EuDu9VfquuMODXgTmHK82uurXj+8AcX6lQZ4zttRek0g5o0g9NPc6zV8FLH4HyEmdiWL8Icr+2C6TEJQTmHNqYq6LdgTw4lB+0njugST80ea/29+fAmpeCf/7KStt1tO2J0G9i4M7TygUtTtCkr6JXkBtxQZN+6DpxFHQ7C5Y9CqWFwT33uvdgx3cw8g7blTSQXGma9FX08k6/0Klv0E6pST9UicDZf4KDO2D1c8E7b2UFLHkA2veCvpcE/nyuQbb94mB+4M+lVKhxf2vXjU5sEbRTatIPZd2GwYmj4fO/QcnB4Jzz+3/DrvUw+s6gDAmny2D7Xa/2VTTKywpqaQd8TPoiMkZE1ovIRhG54yj7TBCRH0RkrYi8Wm37RyKyV0RCdFKZEHf2dCjcDV8+E/hzVZTbmTQ79oVTLwz8+cB2T42J06Svok/RHrtWdBAbccGHpC8iscBsYCzQG5goIr1r7dMDuBMYbozpA1RfYeMR4Eq/RRxtXGnQcwx88SQU7Q3suTLnQ8EmO0AsJkgfAuOb2HqmJn0VbfK+s9+DNP2Cly+v7MHARmPMJmNMKfAaUPsycDIw2xizB8AYs9N7hzHmv8ABP8UbnUbfBcX7YOVTgTtHeald+/aEM6DX2MCdpy6uwXbq5lCZekKpYPAuIhSCST8F2Fbtdo5nW3U9gZ4islxEVorImIYEISJTRGS1iKzOz9cGvSN07genjoMVT0FhQWDO8c0/Yd9W23gsEphzHI1rEJQdgvx1wT2vUk5yZ9kuy8ezCl0j+JL068oAtVf6iAN6AKOAicA8EWntaxDGmDnGmDRjTFpycrKvD4suo++C0oO2zONvZUWQ8Sh0GQInneP/49fHlWa/a4lHRZO8rKBf5YNvST8H6FLttguoPQl6DvCeMabMGJMNrMe+CSh/6XCq7UL55bNwcGf9+zfE6hfggNs2Ggf7Kh/sqmFN2+vkayp6lBbadXGD3HMHfEv6q4AeIpIqIgnA5cD7tfZ5FxgNICLtseWeTf4MVGEHS5UXw+eP+++YpYfg81mQmg6pI/x33IYQsV039UpfRYsda8FUBr3nDviQ9I0x5cBUYDGwDnjDGLNWRGaIyDjPbouB3SLyA7AE+KMxZjeAiCwD3gTOEZEcEfllIH6
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "plt.plot(range(10),lc_nn3_poidszero, label=\"MONKS2: RN_Zero\")\n",
    "plt.plot(range(10),lc_nn3_poidsunif, label=\"MONKS2: RN_Non_Zero)\")\n",
    "leg = plt.legend(loc='best', \n",
    "                 ncol=2, \n",
    "                 mode=\"expand\", \n",
    "                 shadow=True, \n",
    "                 fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MONKS3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VNX5+PHPk50dIoEgkUKVRRBEiai1Ku5xKaKIgBvuy0+k2upXaHEpuNFWba2oZVHcUXBDhYJbtVWxBEGRhGAEhMgykQRECEuS5/fHvRMmIclM1jvL83698srMuefee2YY8sw959zziKpijDHGxHndAGOMMeHBAoIxxhjAAoIxxhiXBQRjjDGABQRjjDEuCwjGGGMACwjGGGNcFhCMMcYAFhCMMca4ErxuQF107NhRu3fv7nUzjDEmoixduvRHVU0LVi+iAkL37t3Jzs72uhnGGBNRROT7UOpZl5ExxhggxIAgIlkikici+SIyvprtj4rIcvdntYhsC9g2RkS+dX/GBJQPEpEV7jEfExFpnJdkjDGmPoJ2GYlIPDAVOAMoAJaIyDxVzfHXUdXbAurfAhzlPk4F7gEyAQWWuvsWA08C1wOLgflAFrCgkV6XMcaYOgrlCmEwkK+qa1R1LzAbOL+W+qOBl93HZwHvqWqRGwTeA7JEpAvQVlU/V2f97eeAYfV+FcYYYxoslIDQFdgQ8LzALTuAiPwC6AF8GGTfru7joMc0xhjTPEIJCNX17deUVWcUMFdVy4LsG/IxReR6EckWkezCwsKgjTXGGFM/oQSEAuCQgOcZwMYa6o5if3dRbfsWuI+DHlNVp6lqpqpmpqUFnUZrjDGmnkIJCEuAniLSQ0SScP7oz6taSUR6Ax2AzwOKFwJnikgHEekAnAksVNVNwA4ROc6dXXQF8FYDX4sxdbdzK3w9x+tWGBMWggYEVS0FxuL8cc8FXlXVlSIySUSGBlQdDczWgCTNqloETMYJKkuASW4ZwE3ADCAf+A6bYWS8kP00vH4tFId0344xUS2kO5VVdT7O1NDAsrurPL+3hn2fBp6upjwbOCLUhhrTJHwrnd+bV0CHX3jbFmM8Zncqm9jmy3V+b17hbTuMCQMWEEzsKt0DW/OdxxYQjLGAYGLY1nwoL4XElhYQjMECgoll/u6iPufB9vVQUuxte4zxmAUEE7t8uRCXAEdc6Dzf/I237THGYxYQTOzy5cJBh0HXQc5z6zYyMc4CgoldvhzodDi07gStO1tAMDHPAoKJTXt3QfE66NTXeZ7e3wKCiXkWEExs+jEPUEjr4zxP7w+Fq6B0r6fNMsZLFhBMbPLPMPJfIXQ+Asr3OUHBmBhlAcHEJl8OxCdDag/nefoA57d1G5kYZgHBxCZfLqT1hrh45/lBh0JCCwsIJqZZQDCxyZfrzDDyi4uHzv0sIJiYZgHBxJ7d2+GnHyoHBNg/00hrSghoTHSzgGBij88dOPYPKPul94c922Hb+uZvkzFhwAKCiT2+HOf3AVcINrBsYpsFBBN7fLmQ1BraHVK5vHNfQCwgmJgVUkAQkSwRyRORfBEZX0Odi0UkR0RWishLbtkpIrI84Ge3iAxzt80SkbUB2wY23ssypha+HOeGNJHK5UmtnLWNLCCYGBU0haaIxANTgTOAAmCJiMxT1ZyAOj2BCcAJqlosIp0AVPUjYKBbJxUnf/KigMPfoapzG+vFGBOSwlXQK6v6ben9oSC7edtjTJgI5QphMJCvqmtUdS8wGzi/Sp3rgKmqWgygqr5qjnMRsEBVdzWkwcY0yM+FsLPwwAFlv/T+lhvBxKxQAkJXYEPA8wK3LFAvoJeIfCoii0Wkuq9fo4CXq5TdLyJfi8ijIpIccquNqa9C/5IVh1e/vWJg2XIjmNgTSkCQasqqTtROAHoCQ4DRwAwRaV9xAJEuQH9gYcA+E4A+wDFAKnBntScXuV5EskUku7CwMITmGlOLqmsYVZXe3/lt4wgmBoUSEAqAwOkYGcDGauq8par7VHUtkIcTIPwuBt5Q1X3+AlXdpI49wDM4XVMHUNVpqpqpqplpaWkhNNeYWvhyoEUHJwdCddp0hladLCCYmBRKQFgC9BSRHiKShNP1M69KnTeBUwBEpCNOF9KagO2jqdJd5F41ICICDAPsGt00Pd8q5+qg6gyjQJYbwcSooAFBVUuBsTjdPbnAq6q6UkQmichQt9pCYKuI5AAf4cwe2gogIt1xrjA+rnLoF0VkBbAC6Ajc1/CXY0wtVA9cw6g6lhvBxKig004BVHU+ML9K2d0BjxX4nftTdd91HDgIjaqeWse2GtMwP210lqYIJSD4cyN0GdA8bTMmDNidyiZ2BBtQ9rMlLEyMsoBgYod/DSN/2syaWG4EE6MsIJjYUbgKWqdDy9Ta6/lzI2yxeQ4mtlhAMLHDlxN8/MAvvT9s/tpyI5iYYgHBxIby8v1TTkOR3t9JpLN9Q/C6xkQJCwgmNmxbB6UldbhCsIFlE3ssIJjY4AuyhlFVlhvBxCALCCY2+ANCWu/Q6ltuBBODLCCY2ODLhfbdILlN6Pv4B5aNiREWEExs8OWGPqDsl94ftq2Hkm1N0yZjwowFBBP9yvbBj6tDHz/w8w8s2/0IJkZYQDDRb+t3ztpEaXUNCJYbwcQWCwgm+gXLklYTy41gYowFBBP9fLkgcdCxV933tYFlE0MsIJjo58uB1EMhMaXu+6b3d+5wttwIJgZYQDDRz5cLnYKscFoTf26EH/Mat03GhCELCCa67dsNRWvqPuXUz5awMDHEAoKJbj+uBi2v+4Cyn+VGMDEkpIAgIlkikici+SIyvoY6F4tIjoisFJGXAsrLRGS5+zMvoLyHiHwhIt+KyCsiktTwl2NMFaFmSauJPzeCBQQTA4IGBBGJB6YCZwN9gdEi0rdKnZ7ABOAEVe0H3BqwuURVB7o/QwPKpwCPqmpPoBi4pmEvxZhq+HIgPglSf1n/Y1huBBMjQrlCGAzkq+oaVd0LzAbOr1LnOmCqqhYDqKqvtgOKiACnAnPdomeBYXVpuDEh8eXCQT0hPrH+x0g/wnIjmJgQSkDoCgT+TyhwywL1AnqJyKcislhEsgK2pYhItlvu/6N/ELBNVUtrOSYAInK9u392YWFhCM01JoAvt/7jB342sGxiRCgBQaopq3rtnAD0BIYAo4EZItLe3dZNVTOBS4C/icihIR7TKVSdpqqZqpqZlpYWQnONce3ZAdvXNzwgdLLcCCY2hBIQCoBDAp5nABurqfOWqu5T1bVAHk6AQFU3ur/XAP8GjgJ+BNqLSEItxzSmYQrdewfqO6Dsl9zamW1kAcFEuVACwhKgpzsrKAkYBcyrUudN4BQAEemI04W0RkQ6iEhyQPkJQI6qKvARcJG7/xjgrYa+GGMq8eU4vxt6hQC2hIWJCUEDgtvPPxZYCOQCr6rqShGZJCL+WUMLga0ikoPzh/4OVd0KHA5ki8hXbvlDqur+L+VO4Hciko8zpjCzMV+YMfhyIbEltP9Fw49luRFMDEgIXgVUdT4wv0rZ3QGPFfid+xNY5zOgfw3HXIMzg8mYpuHLcVJmxjXC/ZeBuRG6/7rhxzMmDNmdyiZ6+VY1fPzAz3IjmBhgAcFEp11F8PPmxhk/AGjdGVqlWUAwUc0CgolOvnomxamJiA0sm6hnAcFEJ/8Mo7qmzayN5UYwUc4CgolOvlxIbgdtD268Y6YPsNwIJqpZQDDRqXCV010k1d0UX082sGyinAUEE31UnS6jxho/8DvoMDc3wjeNe1xjwoQFBBN9ft4CJcWNN+XULy4eOve1gWUTtSwgmOhTsWRFPfMo1ya9v9NlZLkRTBSygGCiT0OzpNUmvT/s3gbbCxr/2MZ4zAKCiT6+XOcmslYdG//YlhvBRDELCCb6NEZSnJpYbgQTxSwgmOhSXu5OOW2C7iIIyI1gA8sm+lhAMNFl+wbY+zOkNcGAsp9/YNmYKGMBwUSXphxQ9kvvD9u+t9wIJupYQDDRpdAfEJryCsGfG2Fl053DGA9YQDDRxZcLbTMgpV3TncOWsDBRKqSAICJZIpI
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "plt.plot(range(10),lc_nn4_poidszero, label=\"MONKS3: RN_Zero\")\n",
    "plt.plot(range(10),lc_nn4_poidsunif, label=\"MONKS3: RN_Non_Zero)\")\n",
    "leg = plt.legend(loc='best', \n",
    "                 ncol=2, \n",
    "                 mode=\"expand\", \n",
    "                 shadow=True, \n",
    "                 fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque ici aussi que l'initialisation des poids à 0 ne permet pas de démarrer l'entrainement du réseau. Dans le second jeu de données, cependant, l'initialisation aléatoire ne permet pas non plus de démarrer l'apprentissage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entrainement et tests\n",
    "On reprend les résultats du dernier entrainement, puisqu'il utilise les poids aléatoires et les hyperparamètres optimaux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[  0. 216.]\n",
      " [  1. 215.]]\n",
      "\n",
      "Exactitude:\n",
      "0.4976851851851852\n",
      "\n",
      "Précision:\n",
      "[0.0, 0.4988399071925754]\n",
      "\n",
      "Rappel:\n",
      "[0.0, 0.9953703703703703]\n",
      "\n",
      "Calculé en:\n",
      "0.20783138275146484s\n"
     ]
    }
   ],
   "source": [
    "res_test2 = nn2_poidsunif.test(test2, test_labels2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[255.  35.]\n",
      " [126.  16.]]\n",
      "\n",
      "Exactitude:\n",
      "0.6273148148148148\n",
      "\n",
      "Précision:\n",
      "[0.6692913385826772, 0.3137254901960784]\n",
      "\n",
      "Rappel:\n",
      "[0.8793103448275862, 0.11267605633802817]\n",
      "\n",
      "Calculé en:\n",
      "0.04411029815673828s\n"
     ]
    }
   ],
   "source": [
    "res_test3 = nn3_poidsunif.test(test3, test_labels3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[203.   1.]\n",
      " [199.  29.]]\n",
      "\n",
      "Exactitude:\n",
      "0.5370370370370371\n",
      "\n",
      "Précision:\n",
      "[0.5049751243781094, 0.9666666666666667]\n",
      "\n",
      "Rappel:\n",
      "[0.9950980392156863, 0.12719298245614036]\n",
      "\n",
      "Calculé en:\n",
      "0.2925841808319092s\n"
     ]
    }
   ],
   "source": [
    "res_test4 = nn4_poidsunif.test(test4, test_labels4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congressionnal Dataset\n",
    "\n",
    "- Chargement du jeu de données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "train5, train_labels5, test5, test_labels5 = (\n",
    "    ld.load_congressional_dataset(train_ratio = 0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entrainement de l'arbre de décision\n",
    "\n",
    "Dans cette section, on entraine un arbre de décision basé sur la mesure d'entropie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt5 = DecisionTree.DecisionTree(attribute_type=\"discrete\")\n",
    "start_time = time.time()\n",
    "_ = dt5.train(train5, train_labels5, verbose=False)\n",
    "dt5_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Courbe d'apprentissage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt_range_lc5,dt_accuracy_cum5 = courbe_apprentissage_dt(dt5,train5,train_labels5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voici le graphique de la courbe d'apprentissage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa58827f2e8>]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXe8I9V99p/fzKjdXrfd7csuy9IWWNZLx5hOwuISB147duJCnAQnduK84NixMe5xYr/YIbFJTNzBGBsMztqYsmDaNljY3uvddnuTrqQp5/1j5oxG0ow0KnvvlXS+n89+9t7RSBpdSc8885zf+R1ijEEgEAgEtYE02QcgEAgEgolDiL5AIBDUEEL0BQKBoIYQoi8QCAQ1hBB9gUAgqCGE6AsEAkENIURfIBAIaggh+gKBQFBDCNEXCASCGkKZ7APIpKOjg82fP3+yD0MgEAgqitdff72PMdaZb78pJ/rz58/Hpk2bJvswBAKBoKIgosN+9hPxjkAgENQQQvQFAoGghhCiLxAIBDWEEH2BQCCoIYToCwQCQQ0hRF8gEAhqCCH6AoFAUEMI0S+Q/rEEnnrr+GQfhkAgEBTFlJucNdX59K+24vc7TuGcrmYs6Kif7MMRCASCghBOv0DGEhoAYF/P2CQfiUAgEBSOEP0CmdkcAQDsOjEyyUciEAgEhSNEv0AMxgAAO4ToCwSCCkSIfoGMxlUAwE4h+gKBoAIRol8gI3Ez0z88EMNP1h1GQtMn+YgEAoHAP0L0C2Q0rqEpbBY9ffaJbfjFpu5JPiKBQCDwjxD9AhmNq7j2rOnYeu8NWDK9Ab96Q4i+QCCoHIToF8hoXENjWEFDSMG7LpyNN44M4VBfdLIPSyAQCHwhRL8AGGMYjatoDAcAALecOxMA8Ie9vZN5WDg5HMfeU6OTegwCgaAyEKJfANGkDoMBjVamP6slAomA3tHEpB7XV9bsxId/KJaYFAgE+RGiXwC8XJM7fVkitDeEJkX0R+Mqjg7EoBsMRwZiODIQs2cLCwQCgRei904BjFrlmtzpA0BHQwh9YxMr+owxvP1fX0DfWBIfvnwBTg7HAZitIZbPabH3IaIJPS6BQDD1EU6/AFJO3yn6wQl3+kndQN9YEgDw1tEh9Iyaor/HyvVf2N2DlV95DtuODU/ocQkEgqmPEP0CGLGdfsDe1tkYsgV4ooirhv3zlu5hGGZnCOzrGcOLe3px549fR+9oAkcGYhN6XAKBYOojRL8AeLzTHEk5/U4r02dWT56JIKGas4BnNoeR1FMngN9tO4mP/mgT2uqCAADVcZtAIBAAQvQLInMgFzCdflI37KuAiYA7/TNnNNrblkxvwJGBGBZ1NuDBD1wEAEhqQvQFAkE6QvQLYGTcfSAXmNiyzXHL6S+d0WRv+9BlC3Dlkk789CNvw7TGMABA1ct/9fHZJ7bif7ecKPvjCgSCiUGIfgGMxlXIEiESkO1tnY2m6E9kBU/cFn3T6UcCMv704jn40YdWoq0+iIBsVu1oRnmdvqYb+Mm6I/ibn71R1scVCAQThyjZLIBYUkddUE4rhZwMp89Fv7MxhPb6IJojgbRjCijmubzc8c7RwfGyPp5AIJh4hOgXQELTEXa4fGCSnL4l5uGAhGWzmhBS0o8pIJmiX+5450CvuUTk/Pa6sj6uQCCYOHzFO0R0IxHtJqJ9RHSPy+1ziWgtEW0moi1EdLPjtk9b99tNRDeU8+AnmoRqIKSk/8laIgFIBAxEJ65skzv9kCLjO3dcgH977/lpt/N4p9zVO/st0Z/XLhaEFwgqlbxOn4hkAA8AuA5AN4CNRPQkY2yHY7fPAniUMfafRLQMwBoA862fbwdwNoBZAJ4loiWMsYpceSShZ4u+JBEUWTotg6ZecNEPB2S0WOWZTmSJQFR+0T/Qa3YTbQiLC0SBoFLx4/RXAtjHGDvAGEsCeATA6ox9GABeStIM4Lj182oAjzDGEoyxgwD2WY9XkZhOX87aLhNBL/Ogab7jAMx4xw0iQkCW0mr4ywEX/YmckyAQCMqLH9HvAnDU8Xu3tc3JvQDeT0TdMF3+xwu4b8WQ0HSEXIRWkQiaMYFOX0s5fS+CsgSt3Jl+nxnv6BP4WgUCQXnxI/puXbsyv/V3APgBY2w2gJsB/JiIJJ/3BRHdSUSbiGhTb+/k9qbPRULLjncAQJYJxkSKvppf9AMylTXeGY2rdrsJMdFXIKhc/Ih+N4A5jt9nIxXfcD4M4FEAYIy9BiAMoMPnfcEYe5AxtoIxtqKzs9P/0U8wpui7xzsT4fRV3cDPNx6x20GEXU5AHHOcoXzqfHQgVa5piHhHIKhY/Ij+RgCLiWgBEQVhDsw+mbHPEQDvAAAiOgum6Pda+91ORCEiWgBgMYAN5Tr4iSah6u5OX6IJiTy++8J+3P3LrXh001Eo1gCyF0FZQlIr3zEdHTSbtxEJ0RcIKpm8os8Y0wDcBeBpADthVulsJ6L7iOhWa7d/APBRInoLwMMA/pyZbId5BbADwO8A/E2lVu4A5mSnkEukUqzodw/G8Njr/hdW/43V/iCuGjmjHaD88c5Rq2Pn/PZ6kekLBBWMr9o7xtgamAO0zm2fc/y8A8BlHvf9MoAvl3CMUwbPTL9I0f/l68fwrWf34Lbls3K6dgA40h/Dbqtf/vC4io6G7FJNJ4GyxzsxNIQUtNUHhdMXCCoY0XunABKae7yjSATdRQjHEhq+8fQuz3YIvAon33jAV9bsxJXfWJu2zW1swUmgzHMHjg6OY3ZrxCpPFaIvEFQqQvQLwKtOX/Io2Vy3vx8PrN2PLd1Dno8H5Bb953aewoN/OIDVy2fhwT+7CPVB8/m9avQ5AaX8Tn9uWx0kCRCaLxBULkL0CyChGZ51+rqLq+aTo7wWLE9YTt/tvpw/7OlFfVDGN95zPq4/ewaaI2Yv/7yZvlS+TJ8xhu7Bccxpq4NEE1ueKhAIyosQfZ8YBkPSpQ0DAMiS5Brv8FjHW/TN29Ucs3lVgyESlBG0nrfJr+iXMdPvHhzHuKpjTmvEHL8Qmb5AULEI0fcJd+2udfqS+yxVLvpRD9Hnt+fKyHWdQZZSc9y404/kE31FQrJMmf53nt+LgEx4+9JpwukLBBWOEH2f8Pw96OX0XYQwYZ0oRj2WUuTxTi5HrhkMipR6zlS8k/utC8oEtcR++obB8L0X9+MXr3fjg5fMx7z2esgSiUxfIKhghOj7hAu0a7xD7m5dtZ2++9SEhB+nbxhQ5Gyn7zZfwElAlkpeOeu5XT346m934e1nTsPfXbsYACB5vFaBQFAZCNH3CRdo95JNd4FNDeSq7o/po3pHM9zjnfAElGweHzJbL3zjPefZi8FLRKJOXyCoYITo+8R2+h4zct1MdWog18vpW3X6OcRZNxgUN9HPV7IpSyUvlzgUM09WfPAYmLiWEwKB4PQgRN8ncdXb6csSuTp9NU/JJr8SyBXDmE7fkenX+a3eKb1kc3hcRUNIQcAxW1iShNMXCCoZIfo+yRXveLnffNU7drxjOX3dYLjl2y/h6e0n7X1KcfrlEP1mh8sHeLxT0sMKBIJJRIi+T1IDudkO26sNAz9RjHlW76Rn+kOxJLYfH8HOEyP2PpmZftMEZvrD42patAN4D1oLBILKQIi+T2yn7+KwJYlcc3ke3/RFE1j55Wfxu20n0263Z+Ry0R83M3QeJQGAphseTj9fnX454p0kmiPpPfkkkekLBBWNEH2fJHJk+opHzs1LNg/2RdEzmsC+ntH0x+RO3xJnPnDKV8YCclTv5K3TL0+80xJJ7+Ypi+odgaCiEaLvk1zxjlfDNe70uUaOZmT7yYx4Z3g8aT1XSqx1g6UNpHa1RHDF4g5cOK815/EGZAkGKy2K8c70hegLBJWKr376gnx1+rkHcjmZA7qZk7O400/kcPrhgIwff/hteY+XT+hSdQOylDsK8mJ4XLWrhThmvFPUwwkEgimAcPo+yZXp56ve4TgHdDXdsO+jZsQ76U4/PdP3S9C6OkgWqdBxVUdcNbKcviyJ5RIFgkpGiL5PuPv2WhjdVfQzBNc5SSszwgG
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(dt_range_lc5,dt_accuracy_cum5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test et évaluation de la performance de l'arbre de décision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[26. 36.]\n",
      " [ 8. 67.]]\n",
      "\n",
      "Exactitude:\n",
      "0.6641221374045801\n",
      "\n",
      "Précision:\n",
      "[0.7647058823529411, 0.6504854368932039]\n",
      "\n",
      "Rappel:\n",
      "[0.41935483870967744, 0.8933333333333333]\n",
      "\n",
      "Calculé en:\n",
      "0.0005872249603271484s\n"
     ]
    }
   ],
   "source": [
    "dt5_testres = dt5.test(test5, test_labels5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Réseaux de neurones: Choix du nombre de neurones dans la couche cachée"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pour faire la séparation du jeu de données en $k_{cv}=5$ jeux de validation croisée, on génère une permutation sur les indices du jeu d'entrainement, puis, on sépare cet ensemble en 5 groupes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_indices5 = range(len(train_labels5))\n",
    "np.random.seed(12345)\n",
    "indices_cv_test5 = (\n",
    "    [np.sort(x) for x in (np.array_split(np.random.permutation(all_indices5),\n",
    "                           k_cv))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_cv_train5 = (\n",
    "    [np.setdiff1d(all_indices5,indices_cv_test5[i]) for i in range(k_cv)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ce jeux de données a trois classes possibles comme variable de sortie. On utilisera donc un réseau de neurones avec 3 neurones dans la couche de sortie, une pour chacune des valeurs possibles. Les valeurs de sortie du jeu de données sont transformées à l'aide d'un encodage binaire où la valeur de sortie est convertie en rang dans un vecteur (on commence à 0), prenant la valeur 1. Par exemple, la valeur 2 devient $[0,0,1]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum5 = []\n",
    "for n_neurones in choix_n_neurones:\n",
    "    accuracy_cv=[]\n",
    "    for cv_set in range(k_cv):\n",
    "        accuracy=0\n",
    "        try:\n",
    "            nn5 = NeuralNet.NeuralNet(np.array([16,n_neurones,3]),range(3))\n",
    "            nn5.train(train5[indices_cv_train5[cv_set]], \n",
    "                      train_labels5[indices_cv_train5[cv_set]], 0.1, 1, \n",
    "                      verbose=False)\n",
    "            _,accuracy,_,_,_ = nn5.test(train5[indices_cv_test5[cv_set]], \n",
    "                                        train_labels5[indices_cv_test5[cv_set]], \n",
    "                                        verbose=False)\n",
    "        except:\n",
    "            pass\n",
    "        accuracy_cv.append(accuracy)\n",
    "    accuracy_cum5.append(np.mean(np.array(accuracy_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa5886f1e10>]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXmYW3d577+v9m0041k8i+3xeI/XxMFJnI2sTkziJEBbSCiF3hJyoYRCE9ZethvK0962F7ht05bQpqUsCSEEMCYQnAVIiJPYjnc7ju3JzHg8+6bRjHbpd/8450gajZYj6Ug6I72f5/HjmaMj6WdZ59Wr7/v+vi8JIcAwDMPUBoZKL4BhGIYpHxz0GYZhaggO+gzDMDUEB32GYZgagoM+wzBMDcFBn2EYpobgoM8wDFNDcNBnGIapITjoMwzD1BCmSi8glebmZtHV1VXpZTAMwywoDh48OCaEaMl1nu6CfldXFw4cOFDpZTAMwywoiKhXzXks7zAMw9QQHPQZhmFqCA76DMMwNQQHfYZhmBqCgz7DMEwNwUGfYRimhuCgzzAMU0Nw0GdqDiEEnjhwHt5AuNJLYZiyw0GfqTmO9HvwmSeP4gev9lV6KQxTdjjoMzXHob5JAMC+7vEKr4Rhyg8HfabmeL1vCgDw2lsTCEdjFV4Nw5QXDvpMzXGobxJ1NhN8oSiO9k9VejkMU1Y46DM1xYg3gP5JP/5k+3IAwMtnWeJhagsO+kxNcViWdm5avxjr292s6zM1Bwd9pqZ4vW8KZiNhY0c9rlrVhAO9kwiEo5VeFsOUDQ76TE1xqG8SG9rdsJmNuGpVE0KRGF6Xu3kYphbgoM/UDJFoDEf7PdjauQgAcPmKRhgNhH3nFo7E84WfHsPnnzrKXUdMwehuchbDlIrTw174w1Fs7WwAANTZzNi0pH7BBH1fKIIf7j+PcFTA4w/j/929FWYj521MfvA7hqkZlP78S+VMHwCuWtWEw+enMBuMVGpZqjnQM4lwVOAdm9rw9LEh/MVjhzjjZ/KGgz5TMxzqm0Szy4Kli+zxY1etakIkJrC/Z6KCK1PHvu5xmAyEf/iji/GF29fjl8eHcP8PXkcowoGfUQ8HfaZmONw3ha2di0BE8WPbljfCbFwYuv6+c+O4eFkDnFYT7r12Jb60awOeOTHMgX8BMRuM4Huv9CIWExVbg6qgT0Q7ieg0EZ0los+luX05ET1HREeJ6DdEtDTptg8S0Rn5zwe1XDzDqGVyNoTusdm4nq9gtxixtXOR7vv1Z4IRHLvgwZUrm+LH/uyaFfjKHRvw65PD+BgH/gXBc2+M4As/PY6jFzwVW0POoE9ERgAPA3gHgA0A7iGiDSmn/QOA/xZCbAHwEIC/ke/bCODLAK4AcDmALxPRIjBMmTks2y1sXTb/7XflyiYcv+CBx6dfq+X9b00gGhO4clXTnON/evUK/O87N2LvyWH8+fcPssavc2YCUu2ob8JXsTWoyfQvB3BWCNEthAgBeBzAXSnnbADwnPzzC0m33wpgrxBiQggxCWAvgJ3FL5th8uNQ7yQMBGxZWj/vtqtWNSEmgFff0m+2v697HBajAW9bPv9D64NXdeFLuzbg2VMjePbkcAVWx6jFF5KC/nmdB/0lAM4n/d4vH0vmCIA/kH9+F4A6ImpSeV8Q0X1EdICIDoyOjqpdO8Oo5tD5Kaxrc8Npnd+lfElnA2xmA17Wsa6/79y4vE5j2tv/eHsnTAaqqGxQa0wHwugencnrPv6QtPu7b1zfQZ/SHEutQnwKwHVEdAjAdQAuAIiovC+EEI8IIbYJIba1tLSoWBLDqCcWEzjcN4VLU/R8BavJiMu6GvGKTnV9jy+M4wNz9fxUrCYj1rTW4cTAdBlXVtv8zdNv4J5vv5LXfXyy5Yfe5Z1+AMuSfl8KYCD5BCHEgBDi3UKIrQD+l3zMo+a+DFNqzo3OwBuMxHfipuPKVU14Y8iLsZlgGVemjlffGocQkgyVjY0dbpy44IEQlesMqSVePDOK8ZlQXvfxBReGpr8fwBoiWkFEFgB3A9idfAIRNROR8lifB/Co/PMzAG4hokVyAfcW+RjDlA3FWye1cycZJYvWY7a/r3scVpMBl2RZPwBs6nBjfDaE4Wn9fXBVG+cnfOif9CMSE3l1TflkeWfQ469Yt1XOoC+EiAC4H1KwPgXgCSHECSJ6iIjulE+7HsBpInoTQCuAr8n3nQDwVUgfHPsBPCQfY5iycahvCvV2M1Y0OTOes3lJPVxWky51/X3nxrGtaxGspvR6vsLGJVKR+sQA6/qlJjk5UIqzalDknZgABqb8mq9LDaq8d4QQTwN4OuXYl5J+fhLAkxnu+ygSmT/DlJ1DfVPY2tkAgyFdiUnCZDTgihWNeEVnQX9iNoQ3hrz41C1rc567vt0NIuD4hWnctL61DKurXfbNCfpRNDjU3U8p5AJA74QPXc2ZE5FSwTtymarGGwjjzRFv2v78VK5c1YTusVkMeiqTgaXjVTm4pPbnp8NlNWFFk5Mz/RIjhMAr58ZhMUnhM59MfzYYQWej9AlRKV2fgz5T1Rw574EQ2fV8hatWNQOAriwZ9nWPw2ExYsvS3OsHgA0dbu7gKYAv/+w4vvXbc6rO7ZvwYcATiBfWfSH1Q3j84Si6mp2wmAwV69XnoM9UNYfkIu7Fy3IHzYva6rDIYdaVri/p+Y2qLZQ3LanHhSk/Jmfz6yqpZfaeHMZ39vXiX397TtWOZiUpUCS02aD6oO8LReG0GLFskb1ivfoc9Jmq5tD5Kaxe7EK93ZzzXIOBsH1lE154YwQ/PXQBMxW2Wx7xBnBmZCZnq2YyGzvcAICTg/rJ9rtHZ/DNZ9/EQz8/WVGjsXT4QhF8ZfcJuKwmTPnC+P3ZsZz32dc9jpY6KzbLhfN85B1/KAqHxYTORkfF5B0eosJULUIIHOqbxI4N6ouaH7pmBY6cn8Inf3gYNrMBN13Uijsu7sD161oy7oYtFa90S41u2TZlpbKxQwpExy94cPXq5pKsSw1DngD2HB3A7iMDONqfqDFs61qE2za3V2xdqfzjc2dxYcqP7997BT7y3YP4xdFBXL9uccbzhRDYd24c21c2wWmR3g/5yDuzoQgcFiNcVgcO9ExCCDHH9bUccNBnqpaecR8mfeGsm7JS2dbViJc+eyNe75vE7iMDePrYIH5xbBB1VhNu2diG27e04erVzTnbJ7Vg37lx1FlN8exdDY1OCzrqbap1/d1HBtA9OgMDEQgAEUBEIAKanBa8Z9uyvILSU6/340cH+vGKvKFs85J6fOH29bhtczs++Ohr+PreN3HrxjYYs3RSlYs3h7349xe78UdvW4qrVzdjx8ZWPHNiCF971+Z4kTaV7rFZjHiDuHJlExyypUdeLZuhKBwWIxa7bfAGI5jyhbHIadHk36MWDvpM1XJIxaasdBgMhG1djdjW1Ygv7dqAfd3j+PmRAfzy+BB+/Ho/XFYTrl/Xgp2b2nD9usVwpfHz0YJXusdx+YpGmPIcibiho15VB8+oN4hPPH4I2TbwblnagPXt6j50esdn8cATR9DZ6MBf3LgGd17SgVUtrvjtD+xYi49+/3XsPnIB79q6NMsjlR4hBL7wk+Nw2Uz4/G3rAQC7trTjqdcv4KWzo7jxovTfDhU9/8pVTXCY88v0o/JGLrvFOKeDh4N+DXL4vGT7e4mKYmMxCCFwtN+jqqhZDRzqm4LLasKaxXUFP4bJaMC1a1pw7ZoW/PU7N+Plc2N45sQQ9p4cxp6jg7CYDLh2dTNu29yOd25dolkGO+QJ4K2xWfzxFZ1533fTEjeee2MYs8FIWoM5hedODUMIYM/Hr8H6djdiQkAIQEDg5MA03vUvL6NnbFZ10O8emwUAfP09F2NbV+O822/d2IaNHW5889kz2LWlo6LzfZ882I/Xeibwt+/ejEY56F6zugVumwl7jg5mDvrd42hz29DV5EBILvqqDfrKNwKnrOkDUtAv9/XIhVwd8NDPT+A9/7av5La4z5wYxl0P/16XVgOl4JXucVyyrEGzQGwxGXD9usX4m3dvwat
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(choix_n_neurones,accuracy_cum5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de neurones qui maximise l'*accuracy* est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_neurones_optimal5 = (\n",
    "    choix_n_neurones[np.where(accuracy_cum5==max(accuracy_cum5))[0][0]])\n",
    "n_neurones_optimal5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choix du nombre de couches cachées"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_cum5 = []\n",
    "lc_cum5 = []\n",
    "for n_couches in choix_n_couches:\n",
    "    accuracy_cv=[]\n",
    "    nn5 = NeuralNet.NeuralNet(\n",
    "        np.hstack((16,\n",
    "                   np.repeat(n_neurones_optimal5,n_couches),\n",
    "                   3)),\n",
    "        range(3))\n",
    "    lc = nn5.train(train5, train_labels5, 0.1, 10, verbose=False)\n",
    "    lc_cum5.append(lc)\n",
    "    _,accuracy,_,_,_ = nn5.test(train5, train_labels5, verbose=False)\n",
    "    accuracy_cv.append(accuracy)\n",
    "    accuracy_cum5.append(np.mean(np.array(accuracy_cv)))\n",
    "lc_cum5 = np.array(lc_cum5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L'*accuracy* pour les différentes profondeur est de :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 2.        , 3.        , 4.        , 5.        ],\n",
       "       [0.94736842, 0.92105263, 0.92763158, 0.90460526, 0.8125    ]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.vstack((choix_n_couches,accuracy_cum5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le nombre de couches cachées qui maximise l'*accuracy* est de:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_couches_optimal5 = choix_n_couches[np.where(accuracy_cum5==max(accuracy_cum5))[0][0]]\n",
    "n_couches_optimal5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Courbes d'apprentissage\n",
    "\n",
    "Ce graphique présente les courbes d'apprentissage pour chacun des niveaux de profondeur du réseau"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXd4VMX3h9+7yaZveiEhhE4CBBISiki3oQiogAVExa+KHfBnQWxgo1iwowgiFiyIhaaAqPRO6AmEUNJ7202y2Tq/P24ICQTSNgVy3+fZZ8udO3PuZvOZuWfOzJGEECgoKCgotAxUTW2AgoKCgkLjoYi+goKCQgtCEX0FBQWFFoQi+goKCgotCEX0FRQUFFoQiugrKCgotCAU0VdQUFBoQSiir6CgoNCCUERfQUFBoQVh39QGXIivr69o165dU5uhoKCgcEWxf//+HCGEX3Xlmp3ot2vXjn379jW1GQoKCgpXFJIkJdaknOLeUVBQUGhBKKKvoKCg0IJQRF9BQUGhBVEj0Zck6WZJkk5IkpQgSdKLVRxvK0nSP5IkHZYkaZMkScEVjlkkSTpY9lhlS+MVFBQUFGpHtRO5kiTZAZ8BNwIpwF5JklYJIWIrFHsP+FYI8Y0kSdcBc4D7yo7phRCRNrZbQUFBQaEO1GSk3xdIEEKcFkIYgZ+A2y4o0w34p+z1f1UcV1BQUFBoBtRE9FsDyRXep5R9VpFDwNiy13cAGkmSfMreO0mStE+SpF2SJN1eL2sVFBQUFOpFTeL0pSo+uzDH4nPAp5IkTQK2AKmAuexYiBAiTZKkDsC/kiQdEUKcqtSAJE0GJgOEhITUwnwFBQWFulFsMBOfqeN4ho70An1TmwNAKw9nJvRrWA2sieinAG0qvA8G0ioWEEKkAWMAJElyA8YKIQorHEMIcVqSpE1AL+DUBed/CXwJ0Lt3byVpr4KCgs2wWgVJeSUcz9ASl67jeIaW4xk6kvJKqJgiXKpqeNvIRLbxbBaivxfoLElSe+QR/D3AhIoFJEnyBfKEEFZgBrCk7HMvoEQIYSgrMwB4x4b2KygoXEHozXoyijNIL0onvTidvNI8XNQuaBw0uKndKj1rHDS4ql2xV9V844CCEiPHM3QcT5eFPS5DR3yGDr3JAsjC3t7Hle5B7oyNCiaslYauge609nRGpWoGqt8IVPttCiHMkiQ9BawH7IAlQohjkiS9AewTQqwChgJzJEkSyO6dJ8tO7woslCTJijx/MPeCqB8FBYWrBCEEBYYC0orTyCjKIK04jfTidNKL0uXPijPIK82rdb3O9s5yJ6DW4ObgJj/sNVgtjpQaHNCW2JOnU5FVIJFfbAcWJ4TVGXcHN7r4+XF3n9Z0C/QkLFBDZ38Nzg52DXD1Vw6SEM3Lm9K7d2+h7L1TP0wWE3sz95KoTaS7T3e6+nRFrVI3tVkKACc3gn9X8LgwFqL5Y7KayCrJKh+lpxenk1aUVv46ozgDvbmyb9zZ3plA10D54RZIkGsQrVxbEeQWRKBrID7OPuhNenRGHTqTjiJj0UWvM4sKSNXmkV1cSH6pFp1Rh8FaAqpSJJUeSWW5rN0SEm5qt/IOQ6OW7yLcHNxwU7vh7uCOn4sfkX6RdPHqgp3qyuwUJEnaL4ToXV25ZrfhmkLdyNXnsjV1K1tStrAjbQfFpuLyY872zvT060m0fzRRAVH09OuJs71zE1rbQsk6DsvGgndHeHgjuHg3tUWVKDYVl4/Kzwn7uRF6WlEa2fpsrMJa6RxvJ28CXQPp5NmJQa0HlYt7oKss8B6OHkjVOMsd7RzxdPJEb7QQn6njTJa2zEUj+9/zS0zlZQM9nOjZSkNYkDtdAzWEtXKntbc9pZZidMayjqJCh1FkKutELnidWZJJQkFC+WfnrstN7UaEf0T5/0q4bziOdo62/7KbEGWkf4UihCA+P57NKZvZnLKZI9lHEAj8nf0ZHDyYNk7RqMyB+HhnciT3IDGZMcTnxyMQ2Ev2dPPpRlRAFFH+UUQFROHh6NHUl3T1s2oKHPoJENCmH9z3O9g17h2YyWLiWO4xDmcfJqUopdz9kl6cjtaorVTWXrInwDWgfFQe6BpIkFvZSL1sxO5k71RnW/KKjfywO5HYdC3H03WcyS0un1h1VtsR2kpD2LlHoDthrTR4ujjU5/KrRAhBenE6MVkx7M/cT0xmDKcLTwPgoHIg3Dec6AC5E4j0i8TNwc3mNtiCmo70FdG/gjBYDOxJ31Mu9BnFGQCE+4QzuM1gOrv14/ApV1YeSiMxtwSQR0bP3NiFsVHBFJt1HMySO4CYrBiO5hzFZJVHUZ08OxHlH1X+427l2qrJrvOqpDgXPugGEfdASH/4/VGIngQjP2zQsJESUwkHs8//zY9kH6HUUgrIo9pzo/Jzgl7uinENxNfZt8FcHQeS8nlyWQzp2lLaeruUCfz50XuIt0uTTqzml+YTkxUjf2+ZMcTlxWERFlSSilCv0EoDJl9n3yazsyKK6F8lZJdksyVlC5tTNrMrfRd6sx5ne2f6B/ZnSJshhHv1Y8cJI78fTONQcgGSBNd29OG+jnralcax4ISG1RmedPJ35/nhodzYLaD8drvUXMrRnKPy6CYrhoNZBykxy51Fa7fW5T/qqIAo2ru3r/Y2XeEybH4X/nsLntwDfqGw8XXYNh+Gz4H+T9ismYpitT9zP8fzjlcSq3Odei//Xk0iVkIIvt+VyBtrYglwd+KLidGEt27+d5klphIOZR8q/24PZx8u7zzburct/1+J9o8mWBPcJP8riuhfoQghiM2LZUvyFjalbCI2Vw52CnQNZEjwEIa0GUJ3715sPl7AHwdT2XoyB4tV0C3QnTER/ox1PYTX0W8gcVt5nUa1B/utXdhU2gmtfx/GjryV3h0vHsmbrWZO5J8oH93EZMWUR1t4O3nTy79X+d1AqHdorULpWjRmA3zYA1r1gIm/yp9ZrbD8PjjxJ4z/GbrcVKeq04rSyjvtC90SPfx6lP+9IvwimtwtoTdaeOn3I/x+IJVhoX58cHdkg7hrGgOTxURsXmyl/5Vz7jF/Z//ywVKUfxSdvTqjkhp+Q2NF9K8g9GY9u9J2sTllM1tTtpKlz0JCoqdfT4a2Gcrg4MG013Rk+6lcVh5MY/2xDEqMFlp7OnNbZBBju9jRMelX2L8UdOng2Rb6PASdboD0w5C0A5G4Eyn3pNyecCDRuSu+3Yfi220oBPcFx4sFQQjBWe3Z8h/1/sz9pBalAuBi70Kkf2T5CKeHb496+Xevag7+CH88BhN/g07Xn//cWAxLboa8M/DQBgjodtlqrMLK6YLT533PWTHlLj6NWiP/PQJkke/u0x0Hu+YjqGdyinn8+/2cyNTxzA1deGpYJ1SmIjj+Jzh5gGcb8AiWX1+BWIWVhIKE8k5gf9Z+skqyANA4aCoNmLr7dEfdAHM5iug3czKKM9icLPvm92TswWAx4Kp25dqgaxnaZigDWw/Ey9GLwymF/HEwldWH0sgpMuLuZM+tPYO4IzKI3sSi2rcYjq8Bq0UW+b6PyM9V+WKLsjCc3s7x3etRp+4iVJzFThIIyQ4psCe0HSD7m0P6g6vPxeeX2X0g60C56JzMlzsSe5U94T7h5aIT6R+Ju4N7Q36FVwZCwMJBYDHDEzsv9t8XpsKi68DeAR75D1zPu1xMVhNxuXHlInIg6wCFhkIAfJ19yzvc3gG96eTZqdmGGq4/lsFzyw9hbyfx0T29GNzFD0q18P0YSNlbubCjhyz+nm3Ao80Fr9uAWwComn8aECEEqUWplVxtZ7VnATlaqYdvj3J3UIR/BK5q13q3qYh+M8MqrBzNOcqm5E1sSdnCifwTAAS7BTO0zVCGtBlCtH80ajs1ibnF/HEgjZUHUzmdU4yDvYrrw/y5vVdrhrZzwjF2BexZDNlx4OQJvSbKI3vvDjW2J7/YyOJ/DhG39x96S8e51f0MIfo4JItBLuAbCm37Q8i18rNn1UvDCw2FHMg6UC5MsTmxmIUZCYnOXp2J8o/ipnY30adVn3p/h1ckZ7bCNyNh1McQ/UDVZVL3w9c
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "for i in choix_n_couches:\n",
    "    plt.plot(range(10),lc_cum5[i-1], label=\"%d couches\"%(i,))\n",
    "leg = plt.legend(loc='best', ncol=2, mode=\"expand\", shadow=True, fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choix des poids initiaux\n",
    "Les poids initiaux ont été choisis à partir d'une distribution uniforme sur $[-1,1]$. On compare ici les courbes d'apprentissage en initialisant les poids à 0 et en initialisant les poids aléatoirement, pour le réseau de dimension et de profondeur optimale sélectionnées précédemment.\n",
    "\n",
    "- Réseau initialisé avec les poids à 0 $RN_{0}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn5_poidszero = NeuralNet.NeuralNet(\n",
    "    np.hstack((16,\n",
    "               np.repeat(n_neurones_optimal5,n_couches_optimal5),\n",
    "               3)),\n",
    "    range(3),\n",
    "    input_weights=0)\n",
    "lc_nn5_poidszero = (\n",
    "    nn5_poidszero.train(train5, train_labels5, 0.1, 10, verbose=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Réseau initialisé avec les poids uniformes $RN_{\\neg{0}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn5_poidsunif = NeuralNet.NeuralNet(\n",
    "    np.hstack((16,\n",
    "               np.repeat(n_neurones_optimal5,n_couches_optimal5),\n",
    "               3)),\n",
    "    range(3))\n",
    "np.random.seed(12345)\n",
    "start_time = time.time()\n",
    "lc_nn5_poidsunif = (\n",
    "    nn5_poidsunif.train(train5, train_labels5, 0.1, 10, verbose=False))\n",
    "nn5_compute_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graphique des courbes d'apprentissage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VFX6x/HPQ2gKIlVRiiAiAioisa+KIgo2UMAF1LWs4G8VXcuqWNbdBduufe2oKIqCig0RxS4WRIJgAcSNKBBiAUGkhyTP748TcIBAJjDJzeR+369XXmTunHvnmSHznTvnnnuuuTsiIhIPVaIuQEREyo9CX0QkRhT6IiIxotAXEYkRhb6ISIwo9EVEYkShLyISIwp9EZEYUeiLiMRI1agL2FjDhg29RYsWUZchIpJWpk6dusjdG5XUrsKFfosWLcjKyoq6DBGRtGJmc5Npp+4dEZEYUeiLiMSIQl9EJEYU+iIiMaLQFxGJEYW+iEiMKPRFRGJEoS8SN/l58NmTsOrXqCuRCCj0ReLmrX/A2EHwwgAoLIy6GilnCn2ROJk5Fj65HxrvC/97Az6+O+qKpJwp9EXiYvF38PIg2HV/OO8taNcT3h4KcydFXZmUo6RC38y6mdlsM8s2s8HF3L+bmb1tZl+Y2Xtm1jThvgIzm170MzaVxYtIkvLXwHNngwF9HoOqNeDke6DebjDmHFixKOoKpZyUGPpmlgHcB3QH2gH9zKzdRs1uA55w932BIcDNCfetcvf9in5OTlHdIlIaE66FH6ZDzwegXouwrGYd6DMCVi6GFwaqfz8mktnTPxDIdvc57p4HjAZ6bNSmHfB20e/vFnO/iETlqxdgysNwyCDY64QN79tlX+j+b/j2bfjw9mjqi7PCQlj2I+Rkhf+nWePK/CGTmVq5CTA/4XYOcNBGbT4HegF3A6cAO5hZA3f/BahpZllAPnCLu7+07WWLSFJ++RbGXgxND4Bj/ll8m05nw9yP4N2boNnB0PLwciywksvPg98WwNL58Ot8WJoDS+cV/T4fli6AgjW/t2+8L7Q9sUxLSib0rZhlvtHtvwH3mtnZwERgASHkAZq7e66Z7Q68Y2Zfuvu3GzyA2UBgIEDz5s1LUb6IbNba1fDcWZBRFXo/BhnVim9nBifeBbnT4fk/w/99CLV3Kt9a09Wa5QmBnhjmOeH3ZT+wSVzWbgx1m8Eu+8FeJ0Ld5rBjs7Bsx2ZlXnIyoZ8DJFbSFMhNbODuucCpAGZWG+jl7ksT7sPd55jZe0BH4NuN1h8GDAPIzMzc+ANFRLbG64Phxy+h/7MhULakRm04bQQ83AWePw/OfBGqZJRPnRWVezjAvXR+QrBvFPCrNzrBrUo12LFJCO9WR4V/d2z6e6Dv2DQcRI9QMqE/BWhtZi0Je/B9gf6JDcysIbDY3QuBq4HhRcvrASvdfU1Rm8OA/6SwfhEpzpdjYOpjcNhfYc/jkltn5/Zw/K3hxK2Jt0LnTQbqVU6rfoVvJmy4p76uKyZ/1YZtq9f+fa+86YG/h3nd5iHQazeGKhV7JHyJoe/u+WY2CJgAZADD3X2GmQ0Bstx9LNAZuNnMnNC9c2HR6m2Bh8yskHDQ+BZ3n1kGz0NE1ln4TejHb3YwHP330q3b8YzQv//eLdDsoLC3WpmtWASPnwgLZ4XbtRqFEN+pbfiwTOx2qdsMatYN3WFpzNwrVm9KZmam6xq5IlspbyU80gWW/wTnfxC6Gkq9jRXw8NGw8pfQv79D49TXWRGsXAwjToZf/heGru5+JFTbLuqqtpqZTXX3zJLaVezvISJSOq9dAT/PhFOGbV3gA1SvFUIwbwWM+TMU5Je8TrpZ9Ss8eQosmg19n4Y23dI68EtDoS9SWUwfBdNGwuF/g9bHbNu2dtoLTrwT5n4I791ccvt0svo3GNkLfpoBfxwJe3SJuqJypdAXqQx+/hpevQx2+wN0vjo12+zQFzqeCR/cDtlvpWabUVuzHJ4+LZydfNqI5A9yVyIKfZF0l7cijMevXgt6PxrG5afK8bfCTu3CNA1LF6Ruu1HIWwmj+sL8T6HXo5uenRwTCn2RdOYOr14OC2dDr0dSf9C12nZhjzh/TThxK13799euhtH9wsikUx6C9j2jrigyCv3KqrAwjE6Qym3aSPh8FBx5FezeuWweo2FrOOlumDcJ3hlaNo9RlvLXwDNnwJz3ocd9sG+fqCuKlEK/Mlr4DQw/Fm5vA9lvl9xe0tNPM2D836DlkXDklWX7WPv0hk7nwEd3hROZ0kV+Hjx7FmS/GT649utf8jqVnEK/MiksgI/uhgf/AL9kQ93dwh7O/ClRVyaptmZZCLOaO4ZunfKYMqHbLdB4H3jx/HDGakVXsBaePxe+eQ2Ovw06nRV1RRWCQr+y+PlreLQrvHk9tO4KF0yGs1+F2jvD033g51lRVyip4g7jLoXF34YDkuU1OVq1mmH8fkF+uPBKfl75PO7WKMgPH06zXoHjboYDB0RdUYWh0E93BflhSN1Dh4fL4fUeHsYe77Bz+PnTS5BRI5yIsmRu1NVKKkx9HL58DjpfU/7TIDdoBT3ugZwp8Pa/yvexk1VYAC9fCF89D12HwCEXRF1RhaLQT2c/zYRHj4G3h0Cb7nDhp7B3rw3nBqnXAs58AdauDMG/fGFk5UoK/PA5vHYVtDoaDr88mhranwIHDIBJ98LXr0ZTw+YUFsIrF8MXo+Ho68KEc7IBhX46KlgbZkF86IjQt9rncTjtCajdqPj2O7eH/s/Bb7kw8tRwRqKkn9W/hevcbt8ATn042tkcj7sxzAf/0l9gyffR1ZHIHcZfHkY0HXkVHHFF1BVVSAr9dPPjV2EyrHdugLYnwYWTw55XSZofBH98MszLMqpfGLcs6cMdxl4Uuuh6D4daDaOtp2qNMH7fgecqQP++e7h+QNZw+MOlqTsruRJS6KeL/Lww3e2wI8PVeE57Evo8Vro3f+uu4cSUuR/BmHPT90SbOJryCMx8Cbr8HXY7JOpqgnotoOd9kPsZvFnKKZxTyR3euA4mPxiuA9zlH2k//XFZUuingx++CHv3790c9uov/BTanbx129qnN3T/D8x+NfR9VrCptaUYudNgwjXQ+lg4tIL1Ubc9CQ6+IATuzJfL//HdwzGtSffCgQPh2BsU+CVI4SQdknL5efDBbWF0zvYNwhSwqZgv5KCBsGpx+BDZrp7eKBXZql/DePxaO4VvaRXxqkzH/AvmT4aXB4Vx/PV3L7/Hfv/f8OEd4eLu3f6tv+MkVMC/IAHC3t2wzuGPeu/ecMEnqZ0g6sirwp7RpHvDWZZS8biHoYe/LQhdedvXj7qi4lWtHgYTWJVwoLm8jhdNvC3suOx3OpxwZ8X8QKyA9CpVNPlrwtfVh7uEvfF+z8CpD6X+DW8W9oz27g1v/TOM/ZaKZfKD8PU4OOaf0OzAqKvZsrrN4ZQHw5DSCdeU/eN99N8wD9A+p8HJ9yjwS0HdOxXJgqnw0oXhep37nR6GxW1Xr+wer0oV6PkArF4azvDcrh6061F2jxeVxd+Fuebz10DmudD25LB3WpHlTIU3/g5tjg8HJ9NBm+5w6EXw8T3Q4rBwzkhZ+OTBcOC4/Snh77c8pqCoRPTxWBGsXR32th85JgTw6WOg5/1lG/jrVK0exvg3PQCePw/mvFf2j1mevngWHjw8hOhvuWF64Lv2hndvht9+iLq64q1cHLpJdtgl/B2kUz91l3+EC6qPvRgWZad++1Mehdevgr1ODOcqpPLaATGh0I/a/CnhJKsP7wx79xd+EoZWlqfq20P/Z6DBHjD69PCNI92tWQYvnA8vDICd28FfPoSLPoPTnw8nFb3/7xD+z50DcydVnFFM7vDSBWFYbp/Hy+eDP5UyqoXzCDKqhwu7rF2Vum1/9mT4xrZnN+j9WHgsKTWFflTWrgpji4cfG658dMbz0OPeMGtiFLarB2e8EEYJjewdpmdOVwumhr37L58NB6zPHh/6nKtUCdeOPf1ZuPgzOOj/4Nu34bFuof3UEeHqSlH6+J4wK+SxN0DTTtHWsrV2bAqnDoO
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(111)\n",
    "plt.plot(range(10),lc_nn5_poidszero, label=\"RN_Zero\")\n",
    "plt.plot(range(10),lc_nn5_poidsunif, label=\"RN_Non_Zero)\")\n",
    "leg = plt.legend(loc='best', \n",
    "                 ncol=2, \n",
    "                 mode=\"expand\", \n",
    "                 shadow=True, \n",
    "                 fancybox=True)\n",
    "leg.get_frame().set_alpha(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque ici aussi que l'initialisation des poids à 0 ne permet pas de démarrer l'entrainement du réseau."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entrainement et tests\n",
    "On reprend les résultats du dernier entrainement, puisqu'il utilise les poids aléatoires et les hyperparamètres optimaux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrice de confusion:\n",
      "[[57.  2.]\n",
      " [ 7. 65.]]\n",
      "\n",
      "Exactitude:\n",
      "0.9312977099236641\n",
      "\n",
      "Précision:\n",
      "[0.890625, 0.9701492537313433]\n",
      "\n",
      "Rappel:\n",
      "[0.9661016949152542, 0.9027777777777778]\n",
      "\n",
      "Calculé en:\n",
      "0.02217245101928711s\n"
     ]
    }
   ],
   "source": [
    "res_test5 = nn5_poidsunif.test(test5, test_labels5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparaison entre arbre de décision et réseau de neurones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Taux d'erreur sur l'ensemble de test. Sur la première ligne, les erreurs pour les arbres de décision, sur la seconde, pour les réseaux de neurones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.97777778, 0.52546296, 0.62962963, 0.53703704, 0.66412214],\n",
       "       [0.91111111, 0.49768519, 0.62731481, 0.53703704, 0.93129771]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([[dt1_testres[1],\n",
    "           dt2_testres[1],\n",
    "           dt3_testres[1],\n",
    "           dt4_testres[1],\n",
    "           dt5_testres[1]],\n",
    "          [res_test1[1],\n",
    "           res_test2[1],\n",
    "           res_test3[1],\n",
    "           res_test4[1],\n",
    "           res_test5[1]]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Temps de prédiction d’un seul exemple. Sur la première ligne, les temps de prédiction moyens pour un exemple de l'ensemble de test pour les arbres de décision, sur la seconde, pour les réseaux de neurones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[7.98967150e-06, 2.79148420e-06, 2.47745602e-06, 2.55582509e-06,\n",
       "        4.48263328e-06],\n",
       "       [7.38196903e-04, 4.81091164e-04, 1.02107172e-04, 6.77278196e-04,\n",
       "        1.69255351e-04]])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([[dt1_testres[4]/len(test_labels1),\n",
    "           dt2_testres[4]/len(test_labels2),\n",
    "           dt3_testres[4]/len(test_labels3),\n",
    "           dt4_testres[4]/len(test_labels4),\n",
    "           dt5_testres[4]/len(test_labels5)],\n",
    "          [res_test1[4]/len(test_labels1),\n",
    "           res_test2[4]/len(test_labels2),\n",
    "           res_test3[4]/len(test_labels3),\n",
    "           res_test4[4]/len(test_labels4),\n",
    "           res_test5[4]/len(test_labels5)]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Temps d’apprentissage du modèle. Sur la première ligne, les temps d'entrainement pour les arbres de décision, sur la seconde, pour les réseaux de neurones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.85799599e-02, 2.40612030e-03, 2.02822685e-03, 1.84917450e-03,\n",
       "        9.10449028e-03],\n",
       "       [4.40478587e+00, 3.11064458e+00, 8.43454361e-01, 4.58273411e+00,\n",
       "        2.55364871e+00]])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([\n",
    "    [dt1_compute_time,\n",
    "     dt2_compute_time,\n",
    "     dt3_compute_time,\n",
    "     dt4_compute_time,\n",
    "     dt5_compute_time],\n",
    "    [nn1_compute_time,\n",
    "     nn2_compute_time,\n",
    "     nn3_compute_time,\n",
    "     nn4_compute_time,\n",
    "     nn5_compute_time]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les arbres de décisions sont des algorithmes simples à mettre en oeuvre et démontrent une exactitude supérieure et comparable aux arbres de décisions lorsque le nombre d'attributs est limité. Cependant, lorsque le nombre d'attributs croît, les réseaux de neurones sont beaucoup plus performants. Ceci est démontré entre autres par la performance sur le jeu de données Congressionnal où le réseau de neurones est beaucoup plus performant. Pour les jeux de données plus difficiles tels que MONKS, on ne remarque pas une grande différence entre les deux algorithmes au niveau du taux d'erreur.\n",
    "\n",
    "Le temps de prédiction est deux ordres de magnitude plus élevé pour les réseaux de neurones que pour les arbres de décisions, car il faut effectuer toutes les multiplications des valeurs par les poids pour chacune des couches, ce qui est une opération $O(d^2)$ pour une dimension $d$. Pour l'arbre de décision, la prédiction est une recherche dans un arbre binaire, de complexité $O(log_2(d))$ pour une profondeur $d$.\n",
    "\n",
    "Les réseaux de neurones nécessitent beaucoup plus de temps d'entrainement que les arbres de décisions, et il faut aussi considérer que ce temps est multiplié lorsque l'on effectue une recherche des hyperparamètres optimaux. Ce besoin de capacités physique explique pourquoi il a fallu attendre l'apparition des processeurs graphiques très performants pour que la recherche avec les réseaux de neurones explose. La complexité est d'autant plus élevée, car le nombre de connexions croit de façon exponentielle avec l'ajout de nouvelles couches cachées."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}