{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Семинар 13. Понижение размерности данных (отбор признаков)\n", "\n", "![](http://i.imgur.com/9HiVD0m.png)\n", "\n", "\n", "Основная цель: каким-либо образом \"уменьшить\" число признаков.\n", "\n", "Почему это может быть важно:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Проклятие размерности\n", "\n", "Рассмотрим пример (аналогично пончикам с лекции): пусть есть треугольники, круги и квадратики. Будем считать что признаки принимают значения только из какого-то интервала. Если есть всего один признак, то объекты лежат \"примерно\" рядом. В случае двух признаков они находятся дальше друг от друга. При увеличении размерности объекты будут все дальше друг от друга. То есть чем больше пространство, тем объекты становятся более \"непохожими\" (лежат далеко друг от друга). Получается, нужно много данных, чтобы алгоритм мог с ними работать (что порой затруднительно).\n", "\n", "![](http://nikhilbuduma.com/img/dimension_sparsity.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Шумовые и просто плохие признаки\n", "\n", "Присутствие шума в признаках может ухудшать модель. Будем решать задачу классификации." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import pylab as plt\n", "\n", "%precision 3\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import sklearn.datasets\n", "from sklearn.cross_validation import train_test_split" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Рассмотрим некоторую случайную выборку, где все признаки \"хорошие\". Разделим ее на обучающую и тестовую:" ] }, { "cell_type": "code", "execution_count": 94, "metadata": { "collapsed": true }, "outputs": [], "source": [ "X, y = sklearn.datasets.make_classification(n_redundant=0, random_state=42)\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "В качестве модели будем использовать логистическую регрессию. Для оценки качества воспользуемся метрикой *Accuracy*." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn.linear_model import LogisticRegressionCV" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Обучим нашу модель и посчитаем ее качество:" ] }, { "cell_type": "code", "execution_count": 97, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.940" ] }, "execution_count": 97, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = LogisticRegressionCV(Cs=[0.01, 0.1, 1.0], random_state=42)\n", "lr.fit(X_train, y_train)\n", "accuracy_score(y_test, lr.predict(X_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Теперь к обучающим и тестовым данным добавим признак, не несущий никакой информации, а являющийся равномерным шумом на интервале от 0 до 1." ] }, { "cell_type": "code", "execution_count": 98, "metadata": { "collapsed": false }, "outputs": [], "source": [ "X_train_noise = np.concatenate([X_train, np.random.uniform(size=(X_train.shape[0], 1))], axis=1)\n", "X_test_noise = np.concatenate([X_test, np.random.uniform(size=(X_test.shape[0], 1))], axis=1)" ] }, { "cell_type": "code", "execution_count": 99, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.920" ] }, "execution_count": 99, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = LogisticRegressionCV(Cs=[0.01, 0.1, 1.0], random_state=42)\n", "lr.fit(X_train_noise, y_train)\n", "accuracy_score(y_test, lr.predict(X_test_noise))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Можно заметить, что качество стало чуть хуже.\n", "\n", "Аналогично и для \"плохих\" признаков (в данном случае — коррелирующих). Добавим к исходным данным первый признак умноженый на 2." ] }, { "cell_type": "code", "execution_count": 100, "metadata": { "collapsed": false }, "outputs": [], "source": [ "X_train_corr = np.concatenate([X_train, (X_train[:,0] * 2)[:, np.newaxis]], axis=1)\n", "X_test_corr = np.concatenate([X_test, (X_test[:, 0] * 2)[:, np.newaxis]], axis=1)" ] }, { "cell_type": "code", "execution_count": 101, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.920" ] }, "execution_count": 101, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = LogisticRegressionCV(Cs=[0.01, 0.1, 1.0], random_state=42)\n", "lr.fit(X_train_corr, y_train)\n", "accuracy_score(y_test, lr.predict(X_test_corr))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Качество снова стало чуть ниже.\n", "\n", "К сожалению, сходу не всегда понятно какие именно из признаков являются шумовыми." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Переобучение\n", "\n", "Чем больше признаков, тем сложнее становится модель (например, в случае линейных — нужно находить больше коэффициентов). Таким образом, если признаков очень много, а данных не совсем — можно столкнуться с переобучением." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Какие еще проблемы возникают при большом числе признаков?*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Отбор признаков" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " - Одномерные методы — оценивают важность признаков по отдельности\n", "\n", "Например, использую оценку дисперсии признака:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn.feature_selection import VarianceThreshold\n", "from sklearn.feature_selection import SelectKBest\n", "from sklearn.feature_selection import f_classif" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([ 0.139, 0.222, 0.25 ])" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = [[0, 0, 1], \n", " [0, 1, 0], \n", " [1, 0, 0], \n", " [0, 1, 1], \n", " [0, 1, 0], \n", " [0, 1, 1]]\n", "sel = VarianceThreshold(0.2)\n", "sel.fit(X)\n", "X_new = sel.fit_transform(X)\n", "sel.variances_" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([[0, 1],\n", " [1, 0],\n", " [0, 0],\n", " [1, 1],\n", " [1, 0],\n", " [1, 1]])" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sel = VarianceThreshold(0.2)\n", "X_new = sel.fit_transform(X)\n", "X_new" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Теперь попробуем поработать с реальными данными." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Рассмотрим датасет Титаник:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale2210A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female3810PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale2600STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female351011380353.1000C123S
4503Allen, Mr. William Henrymale35003734508.0500NaNS
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38 1 \n", "2 Heikkinen, Miss. Laina female 26 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35 1 \n", "4 Allen, Mr. William Henry male 35 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.2500 NaN S \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "4 0 373450 8.0500 NaN S " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = pd.read_csv('titanic.csv')\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Оставим только числовые признаки (а пол перекодируем в 0 и 1), выкинем объекты с пропусками:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PclassAgeSibSpParchFareSex
0322107.25001
11381071.28330
2326007.92500
31351053.10000
4335008.05001
\n", "
" ], "text/plain": [ " Pclass Age SibSp Parch Fare Sex\n", "0 3 22 1 0 7.2500 1\n", "1 1 38 1 0 71.2833 0\n", "2 3 26 0 0 7.9250 0\n", "3 1 35 1 0 53.1000 0\n", "4 3 35 0 0 8.0500 1" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = data[['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']]\n", "X['Sex'] = (data.Sex == 'male').astype('int')\n", "X = X.dropna(axis=0)\n", "\n", "y = data.Survived[X.index]\n", "\n", "X.head()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": true }, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Теперь посмотрим какие признаки с точки зрения одномерных методов важны.\n", "\n", "Функция *SelectKBest* позволяет выбрать *k* наиболее важных признаков из модели с помощью заданного метода, после чего оставить только их. Так как в данном случае нам интересно скорее посмотреть на сами признаки, просто рассмотрим какие \"важности\" мы получаем, используя этот метод. " ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "SelectKBest(k=2, score_func=)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "selector = SelectKBest(f_classif, k=2)\n", "selector.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Посмотрим на полученные значения:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([ 6.771e+01, 8.380e+00, 5.096e-03, 3.647e+00, 4.239e+01,\n", " 2.354e+02])" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "scores = selector.scores_\n", "scores" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEACAYAAACj0I2EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEpFJREFUeJzt3XuQpFV9xvHvIxs0kQSR0p3UrhEMiGC0NiQupqjEIbHw\nUimhvCBKFGNZmkJz8x8hRncSyxgqCYk3knitNWpw1RiQRAWE8RYVlCWs7Eo2KVdwdUcTiAXmxsIv\nf7zvQrv07MzOTHfPHr6fqq45ffp0v+dMv/3M6fO+3ZOqQpJ06HvQpDsgSVoZBrokNcJAl6RGGOiS\n1AgDXZIaYaBLUiMWDPQk65NcneSmJNuS/GZfvynJt5Jc31+ePnCfC5LsTLIjyemjHIAkqZOFzkNP\nMgVMVdUNSY4AvgqcATwfuKOqLtqv/YnAB4EnAeuBq4DjyxPeJWmkFpyhV9WeqrqhL98J7ADW9Tdn\nyF3OAC6pqr1VtQvYCWxcme5KkuZzUGvoSY4BNgBf7qteleSGJO9KcmRftw64deBuu7nvD4AkaUQW\nHej9cstHgN/uZ+oXA4+pqg3AHuDPRtNFSdJirFlMoyRr6ML8b6rqUoCq+t5Ak3cCH+/Lu4FHDdy2\nvq/b/zFdU5ekJaiqYcvdi56hvwfYXlVv3lfRHyzd59nA1/ryZcDZSQ5PcixwHHDtPJ066MumTZuW\ndL9D5eL4Du1Ly+NreWyH0vgOZMEZepJTgXOAbUm2AgX8HvDCJBuAe4BdwCv6kN6eZAuwHbgLOK8W\n6oUkadkWDPSq+gJw2JCbPnmA+7wJeNMy+iVJOkiH3CdFp6enJ92FkXJ8h7aWx9fy2KCN8S34waKR\nbThxJUaSDlISapkHRSVJq5yBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtqytT6KZKM7TK1\nfmrhTo2JHyyS1JQkMDPGDc6w4JdmrSQ/WCRJDwAGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqE\ngS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjo\nktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRELBnqS9UmuTnJTkm1JfquvPyrJFUluTvKpJEcO3OeC\nJDuT7Ehy+igHIEnqLGaGvhd4dVU9HvgF4JVJHgecD1xVVScAVwMXACQ5CTgLOBF4BnBxkoyi85Kk\n+ywY6FW1p6pu6Mt3AjuA9cAZwOa+2WbgzL78LOCSqtpbVbuAncDGFe63JGk/B7WGnuQYYAPwJWBt\nVc1BF/rAI/tm64BbB+62u6+TJI3QmsU2THIE8BHgt6vqziS1X5P9ry9oZmbm3vL09DTT09MH+xCS\n1LTZ2VlmZ2cX1TZVC+dwkjXA5cAnqurNfd0OYLqq5pJMAddU1YlJzgeqqi7s230S2FRVX97vMWsx\n25akg5EEZsa4wRkYZ5YloaqGHpdc7JLLe4Dt+8K8dxnwkr58LnDpQP3ZSQ5PcixwHHDtQfdaknRQ\nFlxySXIqcA6wLclWuqWV3wMuBLYkeSnwTbozW6iq7Um2ANuBu4DznIpL0ugtasllJBt2yUXSCLjk\nIkk65BnoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0\nSWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJek\nRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhqxYKAneXeSuSQ3\nDtRtSvKtJNf3l6cP3HZBkp1JdiQ5fVQdlyT9sMXM0N8LPG1I/UVVdXJ/+SRAkhOBs4ATgWcAFyfJ\nivVWkjSvBQO9qj4P3D7kpmFBfQZwSVXtrapdwE5g47J6KElalOWsob8qyQ1J3pXkyL5uHXDrQJvd\nfZ0kacSWGugXA4+pqg3AHuDPVq5LkqSlWLOUO1XV9wauvhP4eF/eDTxq4Lb1fd1QMzMz95anp6eZ\nnp5eSnckqVmzs7PMzs4uqm2qauFGyTHAx6vqCf31qara05d/F3hSVb0wyUnAB4BT6JZargSOryEb\nSTKsWpKWJQnMjHGDMzDOLEtCVQ092WTBGXqSDwLTwNFJbgE2Aacl2QDcA+wCXgFQVduTbAG2A3cB\n55nakjQei5qhj2TDztAljcADeYbuJ0UlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5J\njTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQI\nA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQ\nJakRBrokNcJAl6RGGOiS1IgFAz3Ju5PMJblxoO6oJFckuTnJp5IcOXDbBUl2JtmR5PRRdVyS9MMW\nM0N/L/C0/erOB66qqhOAq4ELAJKcBJwFnAg8A7g4SVauu5Kk+SwY6FX1eeD2/arPADb35c3AmX35\nWcAlVbW3qnYBO4GNK9NVSdKBLHUN/ZFVNQdQVXuAR/b164BbB9rt7uskSSO2UgdFa4UeR5K0RGuW\neL+5JGurai7JFPDdvn438KiBduv7uqFmZmbuLU9PTzM9Pb3E7khSm2ZnZ5mdnV1U21QtPLlOcgzw\n8ap6Qn/9QuC2qrowyWuAo6rq/P6g6AeAU+iWWq4Ejq8hG0kyrFqSliUJzIxxgzMwzixLQlUNPdlk\nwRl6kg8C08DRSW4BNgF/DHw4yUuBb9Kd2UJVbU+yBdgO3AWcZ2pL0ngsaoY+kg07Q5c0Ag/kGbqf\nFJWkRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0\nSWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDVizSQ3\nnmRs21q7bi17vrVnbNuTpHGbaKAzM75Nzc3MjW9jkjQBLrlIUiMMdElqhIEuSY0w0CWpEQa6JDXC\nQJekRhjoktQIA12SGmGgS1IjDHRJasSyPvqfZBfwfeAe4K6q2pjkKOBDwKOBXcBZVfX9ZfZTkrSA\n5c7Q7wGmq+pnq2pjX3c+cFVVnQBcDVywzG1IkhZhuYGeIY9xBrC5L28GzlzmNiRJi7DcQC/gyiTX\nJXlZX7e2quYAqmoP8MhlbkOStAjL/frcU6vqO0keAVyR5Ga6kB+0//X7XDNQPgY4dpm9kaTGzM7O\nMjs7u6i2qZo/bw9Gkk3AncDL6NbV55JMAddU1YlD2tc4vw+dGVipsUpavZKM9X8tjDtbklBVQ/87\n0JKXXJL8WJIj+vJDgdOBbcBlwEv6ZucCly51G5KkxVvOksta4GNJqn+cD1TVFUm+AmxJ8lLgm8BZ\nK9BPSdIClhzoVfUNYMOQ+tuApy6nU5Kkg+cnRSWpEQa6JDXCQJcegKbWT5FkLJep9VOTHu4DxnLP\nQ5d0CJrbPTe2U/vmZubGsyE5Q5ekVhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCX\npEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElq\nhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDPQRmVo/RZKxXabWT016yJImbM2kO9Cq\nud1zMDPG7c3MjW9jklYlZ+iS1IiRBXqSpyf5epJ/SfKaUW1HktQZSaAneRDwNuBpwOOBFyR53Io8\n+DdW5FFWr8bHNzs7O+kujFTT42t832xhfKOaoW8EdlbVN6vqLuAS4IwVeeRdK/Ioq9euSXdgtA6V\nwFvqQe3TTjut3YPauybdgRHbNekOLN+oDoquA24duP4tupCXDglLPqh9DXDaErbnQW2tAA+KSlIj\nUlUr/6DJk4GZqnp6f/18oKrqwoE2K79hSXoAqKoMqx9VoB8G3Az8CvAd4FrgBVW1Y8U3JkkCRrSG\nXlV3J3kVcAXdss67DXNJGq2RzNAlSeM3sYOiSe5Ocn2SbUk+lOQhB2i7Kcmrx9m/UUlyZpJ7kjx2\n0n1ZCUlem+RrSf65fz43JnnHvs8dJLljnvudkuRLSbYmuSnJ68fb84UdzD56EI95bpK3rkT/VsrA\nOLf2P39q0n0alSH765Mm3aeVNMnvcvlBVZ0MkOT9wG8AfzHB/ozL2cDlwAuAP5hwX5alP/j9TGBD\nVe1N8nDg8Kp6+UCz+d4CbgaeW1VfSxLghBF3dymWvI8meVBV3TPPzavtbfG94zwYSQ6rqrtH0aFR\nmG9/nXC3VtRqOW3xc8BxAEle3P/13Jpk8/4Nk7wsybX97R/eN2tK8rx+JrU1yWxfd1KSL/d/iW9I\n8tPjHNSQvj8UOAV4JV2wk87FSbYn+VSSf0jy7P62k5PMJrkuySeSrJ1g94f5SeDfq2ovQFXdVlV7\nklyTZF9AJMlF/azoyiRH9/WPAOb6+1VVfb1vvCnJ+5L8U5Kbk7xs3IOax+A++rH+Odk22L8kdyT5\n0yRbgScn+fkkX+j3vS/1zz/Auv75vDnJhUO2NW73O2MiyaOTfDbJV/rLk/v6p/T1lwI39XXnDLzO\n/rL/A70azbe/3u91luSwPmd+CSDJm5K8YaK9X4yqmsgFuKP/uQb4e+AVwEl0Z8cc1d/2sP7nJuDV\nffmogcd4A/DKvnwj8JN9+Sf6n2+hO7tm33YePKnx9n14IfBXffkzwM8CzwEu7+vWArcBz+77+wXg\n6P62s+gOLk+s/0PG81BgK/B14O3AL/X11wAn9+V7gLP78uuAtwyUbwM+Crx833PTP9db6WZORwO3\nAFOrZR/db798CLBtYH+9B3hOX/4R4N8Gfg9HAIcB5wL/2l9/MN3nE9dN+HncC1zf/94/OjC2w/vy\nccB1ffkpwB3AT/XXHwdcBhzWX3878GuT3jcXu78e6HVGl0c30Z2t91VgzaTHsNBlkksuP5rk+r78\nWeDddG9pt1TV7QBV9Z9D7vfE/i/lw+ieoE/19Z8HNifZAvxdX/dF4LVJ1gMfq6p/Hc1QFu0FwJ/3\n5Q/TBfyavkxVzSW5pr/9BOBngCv7Gc+DgG+Pt7sHVlU/6Gfivwj8MnBJkgv2a3Y3sKUvv58uwKmq\nN/TLGKfT/R7O7h8D4NKq+j/gP5JcTfcp48tGOpjhBvfRz9HtowC/k+TMvrweOJ7u1Ny93LfvnQB8\nu6quB6iqOwH6yeunB65vBx4N7B7tUA7ov+r+Sy6HA29LsoHuOTx+4LZrq+qWvvwrwMnAdf1++hD6\nd16rzbD9FXgj93+dfadvv73fRy8HTql+Zr+aTTLQ77cTLfKd2nuBZ1W39nou3YyBqjqvP8Dxq8BX\nk5xcVX+b5Et93T8meXlVza7oKBYpyVF0O9HPpPtQ1WF0a6kfm+8uwNeq6tQxdXFJqpvKfBb4bJJt\ndDPQA60R33tbVX0D+Osk7wK+1/+OfqgN3e9hUmvOw/bRp9A9j6dU1f/2f4D3HSz9n/73cW/zeR73\nfwfKd7M6/y/B7wJ7quqJ6T5X8t8Dt/1goBxgc1W9dqy9W6Ih++srOfDr7AnA7XTvnle9Sa6hD9vZ\nrwae1x+sYOAFPugIYE+SHwHOuffBksdU1XVVtQn4LvCoJMdW1Teq6q3ApcATV3wUi/c84H1VdWxV\nPaaqHk33/W63A8/p19LXAtN9+5uBRwysXa5JctIkOj6fJI9NctxA1Qbu/xVHhwHP7cvn0L2TIskz\nB9o8lm52u+8d2RlJDu/X258CXLfCXV+sYfvokcDtfZg/DnjyPO1vBqaS/BxAkiP6YFyN5hvnd/ry\ni+mex2E+DTw3ySOge81mlZ4lM8/+up15Xmf9sayj6JZm3pbkJ8bd54M1yZnB/WZd/VucNwKfSbKX\nbr3rpfs1ez3d29vvAl8Gfryv/5Mk+94WXlVVNyZ5TZIXAXfR7ZxvHME4Fuv5wP4HwD4KnEj35WU3\n0X2h2VeB71fVXUmeC7w1yZF0L6i/oNsBV4sjuK9/e+nWhl8OfGSgzZ3AxiSvo3sr/vy+/kVJLgL+\nq7/vC6uq+ndpNwKzdGvof1hVe8YwlmGGvTP4JPAbSW6iC+0vDmvfP3/PpwuCH6Ub51MXuY1xG9aH\ni4GPJnkx3Zh/MKQNVbUjye8DV6T72uz/o5v13jKs/YTNt7++g/1eZ0nmgD8Cfrmqvp3uVNM3A78+\nma4vjh8sWgWSPLRf33s43R+pU6vqu5Pu1yQk2UR3MPKiSfdFOtSsxrW7B6LLkzyM7syIP3yghrmk\n5XGGLkmNWC0fLJIkLZOBLkmNMNAlqREGuiQ1wkCXpEYY6JLUiP8HFUz98G4RcakAAAAASUVORK5C\nYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "idx = np.arange(scores.size)\n", "width = 0.5\n", "p1 = plt.bar(idx, scores, width, color='g')\n", "plt.xticks(idx + width/2, X.columns)[1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Итак, получили что пол, класс и цена билета хорошо связаны с целевой меткой (с точки зрения этого метода), в то время как SibSp имеет относительно малую значимость. Посмотрим, найдутся ли эти закономерности, если мы попробуем воспользоваться другой группой методов." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " - Отбор с помощью моделей\n", " \n", "Некоторые изученные модели позволяют оценить \"важность\" признаков.\n", "\n", "Например:\n", " - коэффициенты в линейных моделях можно интерпретировать как \"важности\" признаков в случае смасштабированных данных (а в случае *L1*-регуляризации коэффициенты при *неважных* признаках зануляются, например, в *Lasso*)\n", " - RandomForest (и просто решающее дерево) умеют оценивать насколько важен был признак\n", " \n", "Попробуем понять, являются ли эти празнаки также важными с точки зрения моделей." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Для начала рассмотрим знакомую нам линейную модель (логистическую регрессию):" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "LogisticRegressionCV(Cs=[0.1, 1.0, 10], class_weight=None, cv=None,\n", " dual=False, fit_intercept=True, intercept_scaling=1.0,\n", " max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2',\n", " random_state=42, refit=True, scoring=None, solver='lbfgs',\n", " tol=0.0001, verbose=0)" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.preprocessing import StandardScaler\n", "\n", "scaler = StandardScaler()\n", "X_train_scaled = scaler.fit_transform(X_train)\n", "\n", "lr = LogisticRegressionCV(Cs=[0.1, 1.0, 10], random_state=42)\n", "lr.fit(X_train_scaled, y_train)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([[-1.026, -0.727, -0.337, -0.081, 0.111, -1.282]])" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "coef = lr.coef_\n", "coef" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Построим коэффициенты на диаграмме для более удобного сравнения (в качестве \"важности\" считаем модуль коэффициента):" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAEACAYAAABF+UbAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE5dJREFUeJzt3Xuw3Gd93/H3R1YUUkwMNoxPIsWywdguBOO6IHsGileQ\niQ3TxgyXIpsaCkNwJjgzbf6om6FEUghNMqWEgrlYVOOByRBDcKkdCrYp0cZxikEB37AtXwDfZCyX\nYDLglEQW3/6xP4nlaM85K2nPnvNI79fMzvldnvPb73P2t5/z7PM7uydVhSSpTSuWugBJ0sEzxCWp\nYYa4JDXMEJekhhniktQwQ1ySGrZgiCfZmmRXktsWaPfiJLuTvGZy5UmS5jPOSPwK4Nz5GiRZAfwh\ncN0kipIkjWfBEK+qG4HHF2j2W8BngMcmUZQkaTyHPCee5BeBV1fVR4AcekmSpHFN4sLm+4FLh9YN\nckmakpUTOMaLgCuTBHgm8Moku6vqmtkNk/hBLZJ0EKpq5AB53JF4mGOEXVXP7m4nMZgX/81RAT7U\n/oBvGzduPKjva+Vm/9q+Hc79O5z71lL/5rPgSDzJJ4EecFySB4GNwKpBHteW2Rm90PEkSZOzYIhX\n1YXjHqyq3npo5UiSDkQT79js9XpLXcKisn9tO5z7dzj3DQ6P/mWh+ZaJ3llS07w/STocJKEO8cKm\nJGkZMsQlqWGGuCQ1zBCXpIYZ4pLUMENckhpmiEtSwwxxSWqYIS5JDTPEJalhhrgkNcwQl6SGGeKS\nmjezZoYkU7vNrJlZ6i7v46cYSmpeEtg0xTvcxIL/cWeS/BRDSTpMGeKS1DBDXJIaZohLUsMMcUlq\nmCEuSQ0zxCWpYQuGeJKtSXYluW2O/RcmubW73ZjkBZMvU5I0yjgj8SuAc+fZ/y3gZVX1QuD3gY9N\nojBJ0sJWLtSgqm5Msnae/TcNrd4ErJ5EYZKkhU16TvxtwBcmfExJ0hwWHImPK8l64C3AS+drt2nT\npn3LvV6PXq83qRIk6bDQ7/fp9/tjtR3rA7C66ZQ/r6rT59h/OnAVcF5VfXOe4/gBWJImzg/AGuMY\n3W3UwU9gEOAXzRfgkqTJW3A6JckngR5wXJIHgY3AKqCqagvwLuBY4MNJAuyuqnWLV7IkaS8/T1xS\n85xOkSQ1yRCXpIYZ4pLUMENckhpmiEtSwwxxSWqYIS5JDTPEJalhhrgkNcwQl6SGGeKS1DBDXJIa\nZohLUsMMcUlqmCEuSQ0zxCWpYYa4JDXMEJekhhniktQwQ1ySGmaIS1LDDHFJapghLkkNWzDEk2xN\nsivJbfO0+UCSe5PckuSMBY43tdvMmpmD+ZlIUjNWjtHmCuCDwCdG7UzySuA5VfXcJGcBHwXOnvNo\nmw68yIO1a9Ou6d2ZJC2BBUfiVXUj8Pg8Tc6nC/iq+gpwTJLjJ1OeJGk+k5gTXw08NLS+s9smSVpk\n40ynTNa2oeUTgZOmXoEkLWv9fp9+vz9W20mE+E7gl4bW13TbRls/gXuUpMNYr9ej1+vtW9+8efOc\nbcedTkl3G+Ua4E0ASc4Gvl9VXlGUpClYcCSe5JNADzguyYPARmAVUFW1pao+n+RVSe4DngDespgF\nS5J+YsEQr6oLx2hzyWTKkSQdCN+xKUkNM8QlqWGGuCQ1zBCXpIYZ4pLUMENckhpmiEtSwwxxSWqY\nIS5JDTPEJalhhrgkNcwQl6SGGeKS1DBDXJIaZohLUsMMcUlqmCEuSQ0zxCWpYYa4JDXMEJekhhni\nktQwQ1ySGmaIS1LDxgrxJOcl2ZHkniSXjth/XJIvJLklye1J/u3EK5Uk7WfBEE+yArgMOBd4PnBB\nktNmNbsEuKWqzgDWA/81ycpJFytJ+mnjjMTXAfdW1QNVtRu4Ejh/VptHgad1y08D/raqnpxcmZKk\nUcYZLa8GHhpaf5hBsA/7GPClJI8ARwNvmEx5kqT5TGrK43eAW6tqfZLnAF9McnpV/XC/ltuGlk8E\nTppQBZJ0mOj3+/T7/bHajhPiO4EThtbXdNuGvQR4D0BVfTPJt4HTgL/Z72jrx6pLko5YvV6PXq+3\nb33z5s1zth1nTnw7cHKStUlWARuAa2a1uQv4FYAkxwOnAN86oKolSQdswZF4Ve1JcglwPYPQ31pV\ndyW5eLC7tgB/AFyR5FYgwH+oqu8tZuGSpDHnxKvqWuDUWdsuH1r+LvCvJluaJGkhvmNTkhpmiEtS\nwwxxSWqYIS5JDTPEJalhhrgkNcwQl6SGGeITNLNmhiRTu82smVnqLktaYn7m9wTt2rkLNk3x/jbt\nmt6dSVqWHIlLUsMMcUlqmCEuSQ0zxCWpYYa4JDXMEJekhhniktQwQ1ySGmaIS1LDDHFJapghLkkN\nM8QlqWGGuCQ1zBCXpIaNFeJJzkuyI8k9SS6do00vyc1JvpFk22TLlCSNsuDniSdZAVwGvAJ4BNie\n5Oqq2jHU5hjgQ8CvVtXOJM9crIIlST8xzkh8HXBvVT1QVbuBK4HzZ7W5ELiqqnYCVNV3J1umJGmU\ncUJ8NfDQ0PrD3bZhpwDHJtmWZHuSiyZVoCRpbpP692wrgTOBlwNPBb6c5MtVdd9+LYdny08ETppQ\nBZJ0mOj3+/T7/bHajhPiO4EThtbXdNuGPQx8t6p+BPwoyQ3AC4H9Q3z9WHVJ0hGr1+vR6/X2rW/e\nvHnOtuNMp2wHTk6yNskqYANwzaw2VwMvTXJUkn8CnAXcdYB1S5IO0IIj8arak+QS4HoGob+1qu5K\ncvFgd22pqh1JrgNuA/YAW6rqzkWtXJI03px4VV0LnDpr2+Wz1t8LvHdypUmSFuI7NiWpYYa4JDXM\nEJekhhniktQwQ1ySGmaIS1LDDHFJapghLkkNM8QlqWGGuCQ1zBCXpIYZ4pLUMENckhpmiEtSwwxx\nSWqYIS5JDTPEJalhhrgkNcwQl6SGGeKS1DBDXJIaZohLUsMMcUlq2FghnuS8JDuS3JPk0nnavTjJ\n7iSvmVyJkqS5LBjiSVYAlwHnAs8HLkhy2hzt/hC4btJFSpJGG2ckvg64t6oeqKrdwJXA+SPa/Rbw\nGeCxCdYnSZrHOCG+GnhoaP3hbts+SX4ReHVVfQTI5MqTJM1n5YSO835geK587iDfNrR8InDShCqQ\npMNEv9+n3++P1XacEN8JnDC0vqbbNuxFwJVJAjwTeGWS3VV1zX5HWz9WXZJ0xOr1evR6vX3rmzdv\nnrPtOCG+HTg5yVrgO8AG4ILhBlX17L3LSa4A/nxkgEuSJmrBEK+qPUkuAa5nMIe+taruSnLxYHdt\nmf0ti1CnJGmEsebEq+pa4NRZ2y6fo+1bJ1CXJGkMvmNTkhpmiEtSwwxxSWqYIS5JDTPEJalhhrjG\nNrNmhiRTu82smVnqLkvL3qTedq8jwK6du2DTFO9v067p3ZnUKEfiktQwQ1ySGmaIS1LDDHFJapgh\nLkkNM8QlqWGGuCQ1zBCXpIYZ4pLUMENckhpmiEtSwwxxSWqYIS5JDTPEJalhhrgkNcwQl6SGjRXi\nSc5LsiPJPUkuHbH/wiS3drcbk7xg8qVKkmZbMMSTrAAuA84Fng9ckOS0Wc2+Bbysql4I/D7wsUkX\nKkna3zgj8XXAvVX1QFXtBq4Ezh9uUFU3VdXfdas3AasnW6YkaZRxQnw18NDQ+sPMH9JvA75wKEVJ\nksYz0X+UnGQ98BbgpXM22ja0fCJw0iQrkKT29ft9+v3+WG3HCfGdwAlD62u6bT8lyenAFuC8qnp8\nzqOtH6suSTpi9Xo9er3evvXNmzfP2Xac6ZTtwMlJ1iZZBWwArhlukOQE4Crgoqr65kHULEk6CAuO\nxKtqT5JLgOsZhP7WqrorycWD3bUFeBdwLPDhJAF2V9W6xSxckjTmnHhVXQucOmvb5UPLvw78+mRL\nkyQtxHdsSlLDDHFJapghLkkNM8QlqWGGuCQ1zBCXpIYZ4pLUMENckhpmiEtSwwxx6Qgws2aGJFO7\nzayZWeouHzEm+lG0kpanXTt3waYp3t+mXdO7syOcI3FJapghLkkNM8QlqWGGuNSZ5sU/L/xpUryw\nKXWmefHPC3+aFEfiktQwQ1ySGmaIS1LDDHFJapghLkkNM8QlqWFjhXiS85LsSHJPkkvnaPOBJPcm\nuSXJGZMtU5I0yoIhnmQFcBlwLvB84IIkp81q80rgOVX1XOBi4KMTrfLbEz3a8mP/2nY49+9w7hsc\nFv0bZyS+Dri3qh6oqt3AlcD5s9qcD3wCoKq+AhyT5PiJVXn/xI60PN2/1AUssvuXuoBFdv9SF7CI\n7l/qAhbZ/UtdwKEbJ8RXAw8NrT/cbZuvzc4RbSRJE+aFTUlqWKpq/gbJ2cCmqjqvW/+PQFXVHw21\n+Siwrao+1a3vAM6pql2zjjX/nUmSRqqqjNo+zgdgbQdOTrIW+A6wAbhgVptrgHcAn+pC//uzA3y+\nIiRJB2fBEK+qPUkuAa5nMP2ytaruSnLxYHdtqarPJ3lVkvuAJ4C3LG7ZkiQYYzpFkrR8TfXCZpI9\nSb6e5PYkn0rylHnabkzy29Osb7EkeXWSHyc5ZalrOVRJ3pnkG0lu7R7LdUm27H3vQJIfzPF9ZyW5\nKcnNSe5I8rvTrXxhB3J+HsAx35zkg5Oob5KG+npz9/WEpa5psYw4Z1+81DVN0rT/KcQTVXUmQJI/\nAX4DeP+Ua1gKG4DPMbiWsHmJazlo3fWOVwFnVNWTSY4FVlXV24eazfXS7uPA66rqG0kCnLrI5R6M\ngz4/k6yoqh/PsXs5vtzd19cDkeSoqtqzGAUthrnO2SUua6KW8k8M/wo4GSDJm7rfkjcn+fjshkne\nluSr3f4/2ztCSvL6btR0c5J+t+15Sb7S/ca9JclzptmpEbU/FTiLwYXfDd22JPlwkjuTXJfkfyV5\nTbfvzCT9JNuTfGGib5o6dL8AfLeqngSoqu9V1aNJtiXZGwhJ8r5u5PPFJMd1258F7Oq+r6pqR9d4\nY5JPJPk/Se5O8rZpd2oOw+fnZ7vH4/bh+pL8IMl7k9wMnJ3kRUn+ujvvbuoee4DV3WN5d5I/GnFf\nS2G/PzJIsjbJDUn+prud3W0/p9t+NXBHt+2NQ8+zj3S/mJejuc7Z/Z5nSY7qcuZlAEn+IMm7l7T6\ncVTV1G7AD7qvK4H/yeAt+s8D7gae0e17evd1I/Db3fIzho7xbuAd3fJtwC90yz/fff0AcMHQ/fzs\nNPs4os8XAh/tlv8S+GfAa4HPdduOB74HvKar96+B47p9/5rBheQlq39WX54K3AzsAD4EvKzbvg04\ns1v+MbChW34X8IGh5e8BVwFv3/u4dI/zzQxGR8cBDwIzS9S//c7PWefkU4Dbh87VHwOv7ZZ/Bvjm\n0M/haOAo4M3Afd36zzJ4j+DqZfBYPgl8vfvZXzXUv1Xd8snA9m75HOAHwAnd+mkM/iLtqG79Q8C/\nWeo+jXvOzvc8Y5BHdwCvAL4GrFzqPix0m/Z0ys8l+Xq3fAOwlcFL1k9X1eMAVfX9Ed93evcb8ekM\nHpTruu03Ah9P8mngf3Tbvgy8M8ka4LNVdd/idGVsFwB/3C3/GYNQX9ktU1W7kmzr9p8K/DLwxW5k\nswJ4ZLrlzq2qnuhG3P8CeDlwZZLfmdVsD/DpbvlPGIQ2VfXuboriVxn8DDZ0xwC4uqr+EfjbJH/B\n4KMerlnUzow2fH7+FYPzE+DfJXl1t7wGeC7wVQZBuPe8OxV4pKq+DlBVPwToBqhfGlq/E1jL4F3N\nS+nva//plFXAZRl8gN0eBv3c66tV9WC3/ArgTGB7d54+he5V1nIz6pwF3sP+z7PvdO3v7M7TzwFn\nVTeCX86mHeL7nThjvgq7Avi1GsynvpnByICq+s3uIsW/BL6W5Myq+tMkN3XbPp/k7VXVn2gvxpTk\nGQxOnF/O4I1ORzGYH/3sXN8CfKOqXjKlEg9YDYYrNwA3JLmdwUhzvjnfffuq6tvA5Un+O/B/u5/P\nT7Vh8DNYqjnkUefnOQwew7Oq6h+6X7h7L3j+qPt57Gs+x3H/YWh5D8v3H5T/e+DRqjo9yVHA/xva\n98TQcoCPV9U7p1rdQRpxzr6D+Z9nLwAeZ/Aqedmb9pz4qJP8L4DXdxccGHpiDzsaeDTJzwBv3Hew\n5NlVtb2qNgKPAb+U5KSq+nZVfRC4Gjh94r0Y3+uBT1TVSVX17Kpay+Bz0x4HXtvNjR8P9Lr2dwPP\nGpqLXJnkeUtR+ChJTkly8tCmM9j/I4SOAl7XLb+RwaslkrxqqM0pDEaxe191nZ9kVTd/fg6DN5gt\nhVHn5zHA412AnwacPUf7u4GZJP8cIMnRXRAuV3P19Tvd8psYPJajfAl4XZJnweA5m2X61y1znLN3\nMsfzrLs29QwG0y6XJfn5add8oKY9IthvhNW9fHkP8JdJnmQwf/XWWc1+l8HL18eArwBP67b/lyR7\nX/L976q6LcmlSS4CdjM4Id+zCP0Y1xuA2ReyrgL+KYMPEruDwQeHfQ34u6raneR1wAeTHMPgSfR+\nBifdcnA0P6ntSQZzvW8HPjPU5ofAuiTvYvAS+w3d9ouSvA/4++57L6yq6l6J3Qb0GcyJ/15VPTqF\nvowy6hXAtcBvJLmDQVB/eVT77rF7A4Mn/s8x6OevjHkfS2FUHR8GrkryJgb9fmJEG2rwZr//BFyf\nwUdV/yOD0e2Do9ovsbnO2S3Mep4l2QX8Z+DlVfVIBn8a+t9Y5m9e9M0+SyTJU7v5umMZ/GJ6SVU9\nttR1TVuSjQwuKL5vqWuRWrRc5+aOBJ9L8nQGf9Xwe0digEs6dI7EJalhfp64JDXMEJekhhniktQw\nQ1ySGmaIS1LDDHFJatj/B3B+91gBRHhwAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "idx = np.arange(coef.size)\n", "width = 0.5\n", "p1 = plt.bar(idx, np.abs(coef[0]), width, color='g')\n", "plt.xticks(idx + width/2, X.columns)[1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "И посмотрим на качество модели:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.771" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_test_scaled = scaler.transform(X_test)\n", "accuracy_score(y_test, lr.predict(X_test_scaled))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "А теперь случайный лес:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestClassifier" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n", " max_depth=None, max_features='auto', max_leaf_nodes=None,\n", " min_samples_leaf=1, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, n_estimators=1000, n_jobs=1,\n", " oob_score=False, random_state=42, verbose=0, warm_start=False)" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rf = RandomForestClassifier(n_estimators=1000, random_state=42)\n", "rf.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Теперь посчитаем важности признаков:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([ 0.089, 0.303, 0.044, 0.036, 0.268, 0.26 ])" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "importance = rf.feature_importances_\n", "importance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "В данном случае само значение важности не несет какой-то глубокой информации (кроме как чем больше — тем лучше), оно скорее важно для сравнения признаков между собой." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFYJJREFUeJzt3X+wnFd93/H3R1YUYpvwazxSRsIqQSAKgTpOKzxjAgvu\ngMJ0EBOgyFDM0ME4U5yk5R+nQ4mvQmknSSdNwSFYU08HEqaG1BX2UHDkJtwQhxhfQBgMEjZgG1uW\nBTaGAVOCLH/7xz4Sq6u9us+V9v7Q8fs1s3OfH+fsc87dZz979jy796aqkCS1ZdVyN0CSNHmGuyQ1\nyHCXpAYZ7pLUIMNdkhpkuEtSg3qFe5KtSfYluSPJFWP2vyrJbUn2JPlckpeN7Lt7ZN+tk2y8JGm8\nzPc59ySrgDuAi4D7gRlge1XtGylzZlX9qFt+PrCrqjZ1698EfqWqHl6cLkiSZuszct8C3FlV91TV\nIeBaYNtogSPB3jkbeHBkPT2PI0makD6hux64d2T9vm7bMZK8Osle4BPAb43sKuCmJDNJLj2VxkqS\n+lk9qTuqqo8BH0vyIuDPgM3drgur6kCScxiG/N6qunlSx5UkHa9PuO8Hzh1Z39BtG6uqbk6yOsnT\nquqhqjrQbf9Okl0Mp3mOC/ck/pEbSVqgqsq47X2mZWaATUk2JlkDbAduGC2Q5Jkjy+d3B3woyZlJ\nzu62nwW8HLj9BI1c8O3KK688qXqnw63lvtm/0/9m/5b/diLzjtyr6nCSy4HdDF8MrqmqvUkuG+6u\nncBrklwC/AR4BHh9V30tsKsbla8GPlxVu+c7piTp1PSac6+qG/npHPqRbVePLP8B8Adj6t0FnHeK\nbZQkLdBp/xHFwWCw3E1YNC33Dezf6c7+rWzzfolpqSSpldIWSTodJKFO4YKqJOk0Y7hLUoMMd0lq\nkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ\n7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBvcI9ydYk+5LckeSKMftfleS2JHuSfC7Jy/rWlSRN\nXqrqxAWSVcAdwEXA/cAMsL2q9o2UObOqftQtPx/YVVWb+tQduY+ary2SpJ9KQlVl3L4+I/ctwJ1V\ndU9VHQKuBbaNFjgS7J2zgQf71pUkTV6fcF8P3Duyfl+37RhJXp1kL/AJ4LcWUrd16zasI8mS3dZt\nWLfcXZa0zFZP6o6q6mPAx5L8KvBnwOaF3sfU1NTR5cFgwGAwmFTzltXB/QdhagmPN3Vw6Q4maclM\nT08zPT3dq2yfOfcLgKmq2tqt/w5QVfX7J6jzDYZTMs/qW7flOfckSxruTEGrv0tJP3Wqc+4zwKYk\nG5OsAbYDN8w6wDNHls8HqKqH+tSVJE3evNMyVXU4yeXAboYvBtdU1d4klw13107gNUkuAX4CPMIw\nxOesu0h9kSR15p2WWSpOy0zQlNMy0uPBqU7LSJJOM4a7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJ\napDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcpce5dRvWkWTJbus2\nrFvuLj8uzPtv9iS17eD+g0v6n8IOTh1cuoM9jjlyl6QGGe6S1CDDXZIaZLhLatpSXjBeSReLvaAq\nqWlLecF4JV0s7jVyT7I1yb4kdyS5Ysz+NyS5rbvdnOQFI/vu7rbvSXLrJBsvSRpv3pF7klXAVcBF\nwP3ATJLrq2rfSLFvAi+uqu8n2QrsBC7o9j0GDKrq4ck2XZI0lz4j9y3AnVV1T1UdAq4Fto0WqKpb\nqur73eotwPqR3el5HEnShPQJ3fXAvSPr93FseM/2VuCTI+sF3JRkJsmlC2+iJGmhJnpBNclLgbcA\nLxrZfGFVHUhyDsOQ31tVN4+rPzU1dXR5MBgwGAwm2TxJOq1NT08zPT3dq2yfcN8PnDuyvqHbdozu\nIupOYOvo/HpVHeh+fifJLobTPPOGuyTpWLMHvTt27JizbJ9pmRlgU5KNSdYA24EbRgskORe4DnhT\nVX1jZPuZSc7uls8CXg7c3rsnkqSTMu/IvaoOJ7kc2M3wxeCaqtqb5LLh7toJvAt4KvD+JAEOVdUW\nYC2wK0l1x/pwVe1erM5IkoZ6zblX1Y3A5lnbrh5ZvhQ47mJpVd0FnHeKbZQkLZAfUZSkBhnuktQg\nw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLc\nJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQb3CPcnWJPuS3JHkijH735Dk\ntu52c5IX9K0rSZq8ecM9ySrgKuAVwPOAi5M8Z1axbwIvrqp/AvxHYOcC6kqSJqzPyH0LcGdV3VNV\nh4BrgW2jBarqlqr6frd6C7C+b11J0uT1Cff1wL0j6/fx0/Ae563AJ0+yriRpAlZP8s6SvBR4C/Ci\nk6k/NTV1dHkwGDAYDCbSLklqwfT0NNPT073K9gn3/cC5I+sbum3H6C6i7gS2VtXDC6l7xGi4S5KO\nNXvQu2PHjjnL9pmWmQE2JdmYZA2wHbhhtECSc4HrgDdV1TcWUleSNHnzjtyr6nCSy4HdDF8Mrqmq\nvUkuG+6uncC7gKcC708S4FBVbZmr7qL1RpIE9Jxzr6obgc2ztl09snwpcGnfupKkxeU3VCWpQYa7\nJDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtS\ngwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqUK9wT7I1yb4kdyS5Ysz+\nzUk+k+THSd4xa9/dSW5LsifJrZNquCRpbqvnK5BkFXAVcBFwPzCT5Pqq2jdS7CHgN4FXj7mLx4BB\nVT08gfZKknroM3LfAtxZVfdU1SHgWmDbaIGqerCqPg88OqZ+eh5HkjQhfUJ3PXDvyPp93ba+Crgp\nyUySSxfSOEnSyZl3WmYCLqyqA0nOYRjye6vq5nEFp6amji4PBgMGg8ESNE+STg/T09NMT0/3Ktsn\n3PcD546sb+i29VJVB7qf30myi+E0z7zhLkk61uxB744dO+Ys22daZgbYlGRjkjXAduCGE5TP0YXk\nzCRnd8tnAS8Hbu9xTEnSKZh35F5Vh5NcDuxm+GJwTVXtTXLZcHftTLIW+BzwROCxJL8NPBc4B9iV\npLpjfbiqdi9WZyRJQ73m3KvqRmDzrG1XjywfBJ4+puoPgfNOpYGSpIXzI4qS1CDDXZIaZLhLUoMM\nd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCX\npAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNahXuCfZmmRfkjuSXDFm/+Ykn0ny4yTv\nWEhdSdLkzRvuSVYBVwGvAJ4HXJzkObOKPQT8JvCHJ1FXkjRhfUbuW4A7q+qeqjoEXAtsGy1QVQ9W\n1eeBRxdaV5I0eX3CfT1w78j6fd22Pk6lriTpJK1e7gaMmpqaOro8GAwYDAbL1hZJWmmmp6eZnp7u\nVbZPuO8Hzh1Z39Bt62NBdUfDXZJ0rNmD3h07dsxZts+0zAywKcnGJGuA7cANJyifU6grSZqAeUfu\nVXU4yeXAboYvBtdU1d4klw13184ka4HPAU8EHkvy28Bzq+qH4+ouWm8kSUDPOfequhHYPGvb1SPL\nB4Gn960rSVpcfkNVkhpkuEtSg1bURyGTzF9oAtauX8sD9z2wJMeSpOWwosKdqaU5zMGpg0tzIEla\nJk7LSFKDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalB\nhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ3qFe5JtibZl+SOJFfMUea9Se5M8sUkvzyy\n/e4ktyXZk+TWSTVckjS3ef9BdpJVwFXARcD9wEyS66tq30iZXwOeWVXPSvJC4E+BC7rdjwGDqnp4\n4q2XJI3VZ+S+Bbizqu6pqkPAtcC2WWW2AR8CqKrPAk9Ksrbbl57HkSRNSJ/QXQ/cO7J+X7ftRGX2\nj5Qp4KYkM0kuPdmGSpL6m3daZgIurKoDSc5hGPJ7q+rmsSU/NbL8j4BnLEHrJOk0MT09zfT0dK+y\nfcJ9P3DuyPqGbtvsMk8fV6aqDnQ/v5NkF8NpnvHh/tJebZakx6XBYMBgMDi6vmPHjjnL9pmWmQE2\nJdmYZA2wHbhhVpkbgEsAklwAfK+qDiY5M8nZ3fazgJcDt/fviiTpZMw7cq+qw0kuB3YzfDG4pqr2\nJrlsuLt2VtUnkrwyydeBR4C3dNXXAruSVHesD1fV7sXpipbLug3rOLj/4JIdb+36tTxw3wNLdjzp\ndNRrzr2qbgQ2z9p29az1y8fUuws471QaqJXv4P6DMLWEx5tauhcS6XTlRxQlqUGGuyQ1yHCXpAYZ\n7pLUIMNdmse6DetIsmS3dRvWLXeX1YCl+IaqdFrz00A6HTlyl6QGGe6S1CDDXZIaZLhLUoMMd0lq\nkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1qFe4J9ma\nZF+SO5JcMUeZ9ya5M8kXk5y3kLqSpMmaN9yTrAKuAl4BPA+4OMlzZpX5NeCZVfUs4DLgA33rnrK7\nJnpvK0vLfQP7d7qzfytan5H7FuDOqrqnqg4B1wLbZpXZBnwIoKo+CzwpydqedU/N3RO9t5Xl7uVu\nwCK7e7kbsMjuXu4GLLK7l7sBi+zu5W7AqekT7uuBe0fW7+u29SnTp64kacIW64JqFul+JUk9pKpO\nXCC5AJiqqq3d+u8AVVW/P1LmA8Cnquoj3fo+4CXAM+arO3IfJ26IJOk4VTV2ML26R90ZYFOSjcAB\nYDtw8awyNwBvBz7SvRh8r6oOJnmwR90TNlCStHDzhntVHU5yObCb4TTONVW1N8llw921s6o+keSV\nSb4OPAK85UR1F603kiSgx7SMJOn0syK+oZrkcJIvJPlyko8kecIJyl6Z5B1L2b7FlOTVSR5L8uzl\nbsupSvLOJLcnua17PLck2Xnkuw1JfjBHvRcmuSXJniRfSfK7S9vy+S3kHF3Afb45yfsm0b5JGunr\nnu7nucvdpsUw5nz9Z8vdpknqM+e+FB6pqvMBkvw58BvAHy9vk5bMduDjDK9F7Fjmtpy07lrLK4Hz\nqurRJE8F1lTV20aKzfU28YPAa6vq9iQBNi9yc0/GSZ+jSVZV1WNz7F6Jb52P9nUhkpxRVYcXo0GT\nNtf5uszNmqgVMXKf5W+BTQBJLuleVfck+eDsgknemuTWbv9fHBlNJXldN8Lak2S62/bcJJ/tXqG/\nmOSZS9mpcZKcBbyQ4cXo7d22JHl/kq8m+csk/yfJr3f7zk8ynWQmySe7L4qtFL8APFhVjwJU1Xer\n6oEkn0pyJCiS5I+60dJNSZ7WbT8HONjVq6ra1xW+MsmHknwmydeSvHWpOzWH0XN0V/d4fHm0fUl+\nkOS/JNkDXJDknyb5u+7cu6V77AHWd4/l15Ic9ymyZXLchxuSbEzy6SSf624XdNtf0m2/HvhKt+2N\nI8+1P+1esFeauc7X455jSc7ocubFAEn+c5J3L2vr+6iqZb8BP+h+rgY+xvBPGDwX+BrwlG7fk7uf\nVwLv6JafMnIf7wbe3i1/CfiFbvnnu5/vBS4eOc7ProB+vwH4QLf8N8AvA68BPt5tWwt8F/j1rs1/\nBzyt2/cvGV6gXvbHr2vPWcAeYB/wJ8CLu+2fAs7vlh8DtnfL7wLeO7L8XeA64G1HHpvusd7DcET1\nNOBbwLqVco7OOi+fAHx55Hx9DHhNt/wzwDdGfg9nA2cAbwa+3q3/LMPvRK5fAY/lo8AXut/9dSP9\nW9MtbwJmuuWXAD8Azu3Wn8Pw03NndOt/Avyr5e5Tn/P1RM8xhnn0FeAi4PPA6uXuw3y3lTIt83NJ\nvtAtfxq4huHb3o9W1cMAVfW9MfVe0L2CPpnhg/WX3fabgQ8m+Sjwv7ttfw+8M8kGYFdVfX1xurIg\nFwP/tVv+C4Zhv7pbpoYfJ/1Ut38z8EvATd1IaBVw/9I2d25V9Ug3Qv9V4GXAtUn+/axih4GPdst/\nzjDMqap3d1MdL2f4O9je3QfA9VX1E+ChJH/N8E9a3LConRlv9Bz9W4bnKMC/TfLqbnkD8CzgVoYB\neeTc2wzcX1VfAKiqHwJ0A9q/Gln/KrAR2L+4XZnXj+r4aZk1wFUZ/lHAwwz7ecStVfWtbvki4Hxg\npjtPn0D3rmwlGXe+Au/h+OfYga78V7tz9OPAC6sb8a9kKyXcjzuZer6T+x/Aq2o4V/tmhqMIqurf\ndBdH/gXw+STnV9X/THJLt+0TSd5WVdMT7cUCJHkKw5PqlzL8AtcZDOdfd81VBbi9qi5coiYuWA2H\nOJ8GPp3kywxHpieaUz66r6ruAq5O8t+B73S/n2PKMPwdLNcc9bhz9CUMH8MXVtU/dC/ERy60/rj7\nfRwtPsf9/sPI8mFWznNytn8HPFBVL0hyBvD/RvY9MrIc4INV9c4lbd1JGHO+vp0TP8eeDzzM8B31\nirdS5tzHnfh/Dbyuu9DByJN91NnAA0l+Bnjj0TtLfrGqZqrqSuDbwNOTPKOq7qqq9wHXAy+YeC8W\n5nXAh6rqGVX1i1W1keHfoXsYeE03974WGHTlvwacMzLXuTrJc5ej4eMkeXaSTSObzuP4P710BvDa\nbvmNDN9hkeSVI2WezXDUe+Sd2rYka7r5+Zcw/FLdchh3jj4JeLgL9ucAF8xR/mvAuiS/ApDk7C4g\nV6q5+nqgW76E4WM5zl8Br01yDgyft1mBn7aZ43z9KnM8x7rrXk9hOH1zVZKfX+o2L9RKGSUcNxrr\n3ga9B/ibJI8ynB/717OK/S7Dt8DfBj4LPLHb/odJjrxt/L9V9aUkVyR5E3CI4Un6nkXox0K8Hph9\nAe064B8z/ANrX2H4R9c+D3y/qg4leS3wviRPYvjk+mOGJ+RKcDY/bdujDOeS3wb8r5EyPwS2JHkX\nw7fqr++2vynJHwE/6uq+oaqqe/f2JWCa4Zz771XVA0vQl3HGvWO4EfiNJF9hGOB/P65899i9nmEo\n/BzDfv7znsdYDuPa8X7guiSXMOz3I2PKUMMvOP4HYHeGf/L7JwxHxN8aV34ZzXW+7mTWcyzJQeA/\nAS+rqvsz/Pjqf6P7suZK5ZeYVqAkZ3Vzgk9l+KJ1YVV9e7nbtdSSXMnwQuYfLXdbpNPNShm561gf\nT/Jkhp+y+L3HY7BLOjWO3CWpQSvlgqokaYIMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBv1/nviJ3ePN\npq8AAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "idx = np.arange(importance.size)\n", "width = 0.5\n", "p1 = plt.bar(idx, importance, width, color='g')\n", "plt.xticks(idx + width/2, X.columns)[1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Вполне логично, что возраст, пол и цена билета (которая неявно указывает на класс пассажира) важны в этой задаче." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.782" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "accuracy_score(y_test, rf.predict(X_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Если сравнивать с предыдущим методом отбора признаков, то можно сказать что модели также находят указанные закономерности в данных.\n", "\n", "Обе модели показали сравнимое качество, но в тоже время выбрали разные \"важные признаки\". Так как модели имеют разную природу, следовательно влияние признаков вычисляется по-разному. Это необходимо помнить, но также уметь сравнивать результаты: по обеим моделям явно видно, что возраст, как и пол, достаточно хорошие признаки. В то же время цена билета явно зависела от класса, поэтому это нормально что одна модель обратила внимание на цену, а другая — на класс." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Давайте посмотрим какое качество будет иметь случайный лес, если мы хотим оставить всего три признака:\n", " - Pclass, Fare, Sex (от одномерного метода)\n", " - Pclass, Age, Sex (от линейного метода)\n", " - Age, Fare, Sex (от случайного леса)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.804" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rf = RandomForestClassifier(n_estimators=1000, random_state=42)\n", "rf.fit(X_train[['Pclass', 'Fare', 'Sex']], y_train)\n", "accuracy_score(y_test, rf.predict(X_test[['Pclass', 'Fare', 'Sex']]))" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.771" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rf = RandomForestClassifier(n_estimators=1000, random_state=42)\n", "rf.fit(X_train[['Pclass', 'Age', 'Sex']], y_train)\n", "accuracy_score(y_test, rf.predict(X_test[['Pclass', 'Age', 'Sex']]))" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.771" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rf = RandomForestClassifier(n_estimators=1000, random_state=42)\n", "rf.fit(X_train[['Age', 'Fare', 'Sex']], y_train)\n", "accuracy_score(y_test, rf.predict(X_test[['Age', 'Fare', 'Sex']]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Обратите внимание, что отбор признаков нужно проводить на отложенной выборке." ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.10" } }, "nbformat": 4, "nbformat_minor": 0 }