{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Семинар 7. Sklearn и линейная регрессия" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 0. Что сегодня?\n", "- Проверочная №2\n", " - 10 баллов максимум\n", " - 15 минут \n", " - составляет аудиторную оценку, каждая проверочная с равными весами\n", " ___\n", " 1. Только одну из ненаписанных проверочных можно написать, если нет уважительной причины; любую можно с уважительной причиной (к примеру, поход к врачу, приносите справку)\n", " 2. самая худшая из написанных проверочных не будет учитываться. Важнее всего, что из написанных проверочных выбирается худшая. Ненаписанные проверочные будут учитываться с нулевыми баллами, но обязательно будут учитываться каждая;\n", " 3. в конце модуля по оценкам мы подведем итоги. Если появиться необходимость, устроим контрольную работу, чтобы вы могли поднять оценку.\n", "- Начнем смотреть на код настоящих Data Scientists" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![За работу](doit.jpg)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Sklearn Intro" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- [Sklearn, машинное обучение (часть, обучение с учителем)](http://scikit-learn.org/stable/supervised_learning.html)\n", "- [Sklearn, Линейные модели](http://scikit-learn.org/stable/modules/linear_model.html)\n", "\n", "Что еше?\n", "- metrics" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.datasets import load_boston\n", "from sklearn.linear_model import LinearRegression\n", "from matplotlib import pyplot as plt\n", "\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 85, "metadata": { "collapsed": false }, "outputs": [], "source": [ "boston = load_boston()\n", "X = boston.data\n", "y = boston.target" ] }, { "cell_type": "code", "execution_count": 98, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Boston House Prices dataset\n", "\n", "Notes\n", "------\n", "Data Set Characteristics: \n", "\n", " :Number of Instances: 506 \n", "\n", " :Number of Attributes: 13 numeric/categorical predictive\n", " \n", " :Median Value (attribute 14) is usually the target\n", "\n", " :Attribute Information (in order):\n", " - CRIM per capita crime rate by town\n", " - ZN proportion of residential land zoned for lots over 25,000 sq.ft.\n", " - INDUS proportion of non-retail business acres per town\n", " - CHAS Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n", " - NOX nitric oxides concentration (parts per 10 million)\n", " - RM average number of rooms per dwelling\n", " - AGE proportion of owner-occupied units built prior to 1940\n", " - DIS weighted distances to five Boston employment centres\n", " - RAD index of accessibility to radial highways\n", " - TAX full-value property-tax rate per $10,000\n", " - PTRATIO pupil-teacher ratio by town\n", " - B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n", " - LSTAT % lower status of the population\n", " - MEDV Median value of owner-occupied homes in $1000's\n", "\n", " :Missing Attribute Values: None\n", "\n", " :Creator: Harrison, D. and Rubinfeld, D.L.\n", "\n", "This is a copy of UCI ML housing dataset.\n", "http://archive.ics.uci.edu/ml/datasets/Housing\n", "\n", "\n", "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n", "\n", "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n", "prices and the demand for clean air', J. Environ. Economics & Management,\n", "vol.5, 81-102, 1978. Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n", "...', Wiley, 1980. N.B. Various transformations are used in the table on\n", "pages 244-261 of the latter.\n", "\n", "The Boston house-price data has been used in many machine learning papers that address regression\n", "problems. \n", " \n", "**References**\n", "\n", " - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n", " - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n", " - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)\n", "\n" ] } ], "source": [ "print(boston.DESCR)" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import pandas as pd\n", "boston_df = pd.DataFrame(np.column_stack([boston.data, boston.target]), columns = \n", " np.append(boston.feature_names, \"MEDV_target\"))" ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", " | CRIM | \n", "ZN | \n", "INDUS | \n", "CHAS | \n", "NOX | \n", "RM | \n", "AGE | \n", "DIS | \n", "RAD | \n", "TAX | \n", "PTRATIO | \n", "B | \n", "LSTAT | \n", "MEDV_target | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "0.00632 | \n", "18.0 | \n", "2.31 | \n", "0 | \n", "0.538 | \n", "6.575 | \n", "65.2 | \n", "4.0900 | \n", "1 | \n", "296 | \n", "15.3 | \n", "396.90 | \n", "4.98 | \n", "24.0 | \n", "
1 | \n", "0.02731 | \n", "0.0 | \n", "7.07 | \n", "0 | \n", "0.469 | \n", "6.421 | \n", "78.9 | \n", "4.9671 | \n", "2 | \n", "242 | \n", "17.8 | \n", "396.90 | \n", "9.14 | \n", "21.6 | \n", "
2 | \n", "0.02729 | \n", "0.0 | \n", "7.07 | \n", "0 | \n", "0.469 | \n", "7.185 | \n", "61.1 | \n", "4.9671 | \n", "2 | \n", "242 | \n", "17.8 | \n", "392.83 | \n", "4.03 | \n", "34.7 | \n", "
3 | \n", "0.03237 | \n", "0.0 | \n", "2.18 | \n", "0 | \n", "0.458 | \n", "6.998 | \n", "45.8 | \n", "6.0622 | \n", "3 | \n", "222 | \n", "18.7 | \n", "394.63 | \n", "2.94 | \n", "33.4 | \n", "
4 | \n", "0.06905 | \n", "0.0 | \n", "2.18 | \n", "0 | \n", "0.458 | \n", "7.147 | \n", "54.2 | \n", "6.0622 | \n", "3 | \n", "222 | \n", "18.7 | \n", "396.90 | \n", "5.33 | \n", "36.2 | \n", "
5 | \n", "0.02985 | \n", "0.0 | \n", "2.18 | \n", "0 | \n", "0.458 | \n", "6.430 | \n", "58.7 | \n", "6.0622 | \n", "3 | \n", "222 | \n", "18.7 | \n", "394.12 | \n", "5.21 | \n", "28.7 | \n", "
6 | \n", "0.08829 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "6.012 | \n", "66.6 | \n", "5.5605 | \n", "5 | \n", "311 | \n", "15.2 | \n", "395.60 | \n", "12.43 | \n", "22.9 | \n", "
7 | \n", "0.14455 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "6.172 | \n", "96.1 | \n", "5.9505 | \n", "5 | \n", "311 | \n", "15.2 | \n", "396.90 | \n", "19.15 | \n", "27.1 | \n", "
8 | \n", "0.21124 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "5.631 | \n", "100.0 | \n", "6.0821 | \n", "5 | \n", "311 | \n", "15.2 | \n", "386.63 | \n", "29.93 | \n", "16.5 | \n", "
9 | \n", "0.17004 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "6.004 | \n", "85.9 | \n", "6.5921 | \n", "5 | \n", "311 | \n", "15.2 | \n", "386.71 | \n", "17.10 | \n", "18.9 | \n", "
10 | \n", "0.22489 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "6.377 | \n", "94.3 | \n", "6.3467 | \n", "5 | \n", "311 | \n", "15.2 | \n", "392.52 | \n", "20.45 | \n", "15.0 | \n", "
11 | \n", "0.11747 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "6.009 | \n", "82.9 | \n", "6.2267 | \n", "5 | \n", "311 | \n", "15.2 | \n", "396.90 | \n", "13.27 | \n", "18.9 | \n", "
12 | \n", "0.09378 | \n", "12.5 | \n", "7.87 | \n", "0 | \n", "0.524 | \n", "5.889 | \n", "39.0 | \n", "5.4509 | \n", "5 | \n", "311 | \n", "15.2 | \n", "390.50 | \n", "15.71 | \n", "21.7 | \n", "
13 | \n", "0.62976 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.949 | \n", "61.8 | \n", "4.7075 | \n", "4 | \n", "307 | \n", "21.0 | \n", "396.90 | \n", "8.26 | \n", "20.4 | \n", "
14 | \n", "0.63796 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "6.096 | \n", "84.5 | \n", "4.4619 | \n", "4 | \n", "307 | \n", "21.0 | \n", "380.02 | \n", "10.26 | \n", "18.2 | \n", "
15 | \n", "0.62739 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.834 | \n", "56.5 | \n", "4.4986 | \n", "4 | \n", "307 | \n", "21.0 | \n", "395.62 | \n", "8.47 | \n", "19.9 | \n", "
16 | \n", "1.05393 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.935 | \n", "29.3 | \n", "4.4986 | \n", "4 | \n", "307 | \n", "21.0 | \n", "386.85 | \n", "6.58 | \n", "23.1 | \n", "
17 | \n", "0.78420 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.990 | \n", "81.7 | \n", "4.2579 | \n", "4 | \n", "307 | \n", "21.0 | \n", "386.75 | \n", "14.67 | \n", "17.5 | \n", "
18 | \n", "0.80271 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.456 | \n", "36.6 | \n", "3.7965 | \n", "4 | \n", "307 | \n", "21.0 | \n", "288.99 | \n", "11.69 | \n", "20.2 | \n", "
19 | \n", "0.72580 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.727 | \n", "69.5 | \n", "3.7965 | \n", "4 | \n", "307 | \n", "21.0 | \n", "390.95 | \n", "11.28 | \n", "18.2 | \n", "
20 | \n", "1.25179 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.570 | \n", "98.1 | \n", "3.7979 | \n", "4 | \n", "307 | \n", "21.0 | \n", "376.57 | \n", "21.02 | \n", "13.6 | \n", "
21 | \n", "0.85204 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.965 | \n", "89.2 | \n", "4.0123 | \n", "4 | \n", "307 | \n", "21.0 | \n", "392.53 | \n", "13.83 | \n", "19.6 | \n", "
22 | \n", "1.23247 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "6.142 | \n", "91.7 | \n", "3.9769 | \n", "4 | \n", "307 | \n", "21.0 | \n", "396.90 | \n", "18.72 | \n", "15.2 | \n", "
23 | \n", "0.98843 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.813 | \n", "100.0 | \n", "4.0952 | \n", "4 | \n", "307 | \n", "21.0 | \n", "394.54 | \n", "19.88 | \n", "14.5 | \n", "
24 | \n", "0.75026 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.924 | \n", "94.1 | \n", "4.3996 | \n", "4 | \n", "307 | \n", "21.0 | \n", "394.33 | \n", "16.30 | \n", "15.6 | \n", "
25 | \n", "0.84054 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.599 | \n", "85.7 | \n", "4.4546 | \n", "4 | \n", "307 | \n", "21.0 | \n", "303.42 | \n", "16.51 | \n", "13.9 | \n", "
26 | \n", "0.67191 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "5.813 | \n", "90.3 | \n", "4.6820 | \n", "4 | \n", "307 | \n", "21.0 | \n", "376.88 | \n", "14.81 | \n", "16.6 | \n", "
27 | \n", "0.95577 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "6.047 | \n", "88.8 | \n", "4.4534 | \n", "4 | \n", "307 | \n", "21.0 | \n", "306.38 | \n", "17.28 | \n", "14.8 | \n", "
28 | \n", "0.77299 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "6.495 | \n", "94.4 | \n", "4.4547 | \n", "4 | \n", "307 | \n", "21.0 | \n", "387.94 | \n", "12.80 | \n", "18.4 | \n", "
29 | \n", "1.00245 | \n", "0.0 | \n", "8.14 | \n", "0 | \n", "0.538 | \n", "6.674 | \n", "87.3 | \n", "4.2390 | \n", "4 | \n", "307 | \n", "21.0 | \n", "380.23 | \n", "11.98 | \n", "21.0 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
476 | \n", "4.87141 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.614 | \n", "6.484 | \n", "93.6 | \n", "2.3053 | \n", "24 | \n", "666 | \n", "20.2 | \n", "396.21 | \n", "18.68 | \n", "16.7 | \n", "
477 | \n", "15.02340 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.614 | \n", "5.304 | \n", "97.3 | \n", "2.1007 | \n", "24 | \n", "666 | \n", "20.2 | \n", "349.48 | \n", "24.91 | \n", "12.0 | \n", "
478 | \n", "10.23300 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.614 | \n", "6.185 | \n", "96.7 | \n", "2.1705 | \n", "24 | \n", "666 | \n", "20.2 | \n", "379.70 | \n", "18.03 | \n", "14.6 | \n", "
479 | \n", "14.33370 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.614 | \n", "6.229 | \n", "88.0 | \n", "1.9512 | \n", "24 | \n", "666 | \n", "20.2 | \n", "383.32 | \n", "13.11 | \n", "21.4 | \n", "
480 | \n", "5.82401 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.532 | \n", "6.242 | \n", "64.7 | \n", "3.4242 | \n", "24 | \n", "666 | \n", "20.2 | \n", "396.90 | \n", "10.74 | \n", "23.0 | \n", "
481 | \n", "5.70818 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.532 | \n", "6.750 | \n", "74.9 | \n", "3.3317 | \n", "24 | \n", "666 | \n", "20.2 | \n", "393.07 | \n", "7.74 | \n", "23.7 | \n", "
482 | \n", "5.73116 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.532 | \n", "7.061 | \n", "77.0 | \n", "3.4106 | \n", "24 | \n", "666 | \n", "20.2 | \n", "395.28 | \n", "7.01 | \n", "25.0 | \n", "
483 | \n", "2.81838 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.532 | \n", "5.762 | \n", "40.3 | \n", "4.0983 | \n", "24 | \n", "666 | \n", "20.2 | \n", "392.92 | \n", "10.42 | \n", "21.8 | \n", "
484 | \n", "2.37857 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.583 | \n", "5.871 | \n", "41.9 | \n", "3.7240 | \n", "24 | \n", "666 | \n", "20.2 | \n", "370.73 | \n", "13.34 | \n", "20.6 | \n", "
485 | \n", "3.67367 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.583 | \n", "6.312 | \n", "51.9 | \n", "3.9917 | \n", "24 | \n", "666 | \n", "20.2 | \n", "388.62 | \n", "10.58 | \n", "21.2 | \n", "
486 | \n", "5.69175 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.583 | \n", "6.114 | \n", "79.8 | \n", "3.5459 | \n", "24 | \n", "666 | \n", "20.2 | \n", "392.68 | \n", "14.98 | \n", "19.1 | \n", "
487 | \n", "4.83567 | \n", "0.0 | \n", "18.10 | \n", "0 | \n", "0.583 | \n", "5.905 | \n", "53.2 | \n", "3.1523 | \n", "24 | \n", "666 | \n", "20.2 | \n", "388.22 | \n", "11.45 | \n", "20.6 | \n", "
488 | \n", "0.15086 | \n", "0.0 | \n", "27.74 | \n", "0 | \n", "0.609 | \n", "5.454 | \n", "92.7 | \n", "1.8209 | \n", "4 | \n", "711 | \n", "20.1 | \n", "395.09 | \n", "18.06 | \n", "15.2 | \n", "
489 | \n", "0.18337 | \n", "0.0 | \n", "27.74 | \n", "0 | \n", "0.609 | \n", "5.414 | \n", "98.3 | \n", "1.7554 | \n", "4 | \n", "711 | \n", "20.1 | \n", "344.05 | \n", "23.97 | \n", "7.0 | \n", "
490 | \n", "0.20746 | \n", "0.0 | \n", "27.74 | \n", "0 | \n", "0.609 | \n", "5.093 | \n", "98.0 | \n", "1.8226 | \n", "4 | \n", "711 | \n", "20.1 | \n", "318.43 | \n", "29.68 | \n", "8.1 | \n", "
491 | \n", "0.10574 | \n", "0.0 | \n", "27.74 | \n", "0 | \n", "0.609 | \n", "5.983 | \n", "98.8 | \n", "1.8681 | \n", "4 | \n", "711 | \n", "20.1 | \n", "390.11 | \n", "18.07 | \n", "13.6 | \n", "
492 | \n", "0.11132 | \n", "0.0 | \n", "27.74 | \n", "0 | \n", "0.609 | \n", "5.983 | \n", "83.5 | \n", "2.1099 | \n", "4 | \n", "711 | \n", "20.1 | \n", "396.90 | \n", "13.35 | \n", "20.1 | \n", "
493 | \n", "0.17331 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "5.707 | \n", "54.0 | \n", "2.3817 | \n", "6 | \n", "391 | \n", "19.2 | \n", "396.90 | \n", "12.01 | \n", "21.8 | \n", "
494 | \n", "0.27957 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "5.926 | \n", "42.6 | \n", "2.3817 | \n", "6 | \n", "391 | \n", "19.2 | \n", "396.90 | \n", "13.59 | \n", "24.5 | \n", "
495 | \n", "0.17899 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "5.670 | \n", "28.8 | \n", "2.7986 | \n", "6 | \n", "391 | \n", "19.2 | \n", "393.29 | \n", "17.60 | \n", "23.1 | \n", "
496 | \n", "0.28960 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "5.390 | \n", "72.9 | \n", "2.7986 | \n", "6 | \n", "391 | \n", "19.2 | \n", "396.90 | \n", "21.14 | \n", "19.7 | \n", "
497 | \n", "0.26838 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "5.794 | \n", "70.6 | \n", "2.8927 | \n", "6 | \n", "391 | \n", "19.2 | \n", "396.90 | \n", "14.10 | \n", "18.3 | \n", "
498 | \n", "0.23912 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "6.019 | \n", "65.3 | \n", "2.4091 | \n", "6 | \n", "391 | \n", "19.2 | \n", "396.90 | \n", "12.92 | \n", "21.2 | \n", "
499 | \n", "0.17783 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "5.569 | \n", "73.5 | \n", "2.3999 | \n", "6 | \n", "391 | \n", "19.2 | \n", "395.77 | \n", "15.10 | \n", "17.5 | \n", "
500 | \n", "0.22438 | \n", "0.0 | \n", "9.69 | \n", "0 | \n", "0.585 | \n", "6.027 | \n", "79.7 | \n", "2.4982 | \n", "6 | \n", "391 | \n", "19.2 | \n", "396.90 | \n", "14.33 | \n", "16.8 | \n", "
501 | \n", "0.06263 | \n", "0.0 | \n", "11.93 | \n", "0 | \n", "0.573 | \n", "6.593 | \n", "69.1 | \n", "2.4786 | \n", "1 | \n", "273 | \n", "21.0 | \n", "391.99 | \n", "9.67 | \n", "22.4 | \n", "
502 | \n", "0.04527 | \n", "0.0 | \n", "11.93 | \n", "0 | \n", "0.573 | \n", "6.120 | \n", "76.7 | \n", "2.2875 | \n", "1 | \n", "273 | \n", "21.0 | \n", "396.90 | \n", "9.08 | \n", "20.6 | \n", "
503 | \n", "0.06076 | \n", "0.0 | \n", "11.93 | \n", "0 | \n", "0.573 | \n", "6.976 | \n", "91.0 | \n", "2.1675 | \n", "1 | \n", "273 | \n", "21.0 | \n", "396.90 | \n", "5.64 | \n", "23.9 | \n", "
504 | \n", "0.10959 | \n", "0.0 | \n", "11.93 | \n", "0 | \n", "0.573 | \n", "6.794 | \n", "89.3 | \n", "2.3889 | \n", "1 | \n", "273 | \n", "21.0 | \n", "393.45 | \n", "6.48 | \n", "22.0 | \n", "
505 | \n", "0.04741 | \n", "0.0 | \n", "11.93 | \n", "0 | \n", "0.573 | \n", "6.030 | \n", "80.8 | \n", "2.5050 | \n", "1 | \n", "273 | \n", "21.0 | \n", "396.90 | \n", "7.88 | \n", "11.9 | \n", "
506 rows × 14 columns
\n", "