{"id":2122,"date":"2018-01-01T23:10:39","date_gmt":"2018-01-01T23:10:39","guid":{"rendered":"http:\/\/pcool.dyndns.org:8080\/statsbook\/?page_id=2122"},"modified":"2025-06-30T17:10:14","modified_gmt":"2025-06-30T16:10:14","slug":"linear-discriminant-analysis","status":"publish","type":"page","link":"https:\/\/pcool.dyndns.org\/index.php\/linear-discriminant-analysis\/","title":{"rendered":"Linear Discriminant Analysis"},"content":{"rendered":"\n

Linear Discriminant Analysis(LDA) is a  linear transformation technique of supervised<\/strong> machine learning (there needs to be a classifier). The aim of the technique is to find a linear combination of variables that separates the classifier variables as much as possible.<\/p>\n\n\n\n

To illustrate the method, R\u2019s build in data set \u201ciris\u201d is used. The iris data set contains the sepal and petal length and width of three different types of iris plants: Setosa, Versicolor and Virginica. Have a look at the iris data set:<\/p>\n\n\n\n

head(iris)<\/em><\/span>\nSepal.Length Sepal.Width Petal.Length Petal.Width Species<\/em><\/span>\n1 5.1 3.5 1.4 0.2 setosa<\/em><\/span>\n2 4.9 3.0 1.4 0.2 setosa<\/em><\/span>\n3 4.7 3.2 1.3 0.2 setosa<\/em><\/span>\n4 4.6 3.1 1.5 0.2 setosa<\/em><\/span>\n5 5.0 3.6 1.4 0.2 setosa<\/em><\/span>\n6 5.4 3.9 1.7 0.4 setosa<\/em><\/span>\nstr(iris)<\/em><\/span>\n\u2018data.frame': 150 obs. of 5 variables:<\/em><\/span>\n$ Sepal.Length: num 5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 \u2026<\/em><\/span>\n$ Sepal.Width : num 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 \u2026<\/em><\/span>\n$ Petal.Length: num 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 \u2026<\/em><\/span>\n$ Petal.Width : num 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 \u2026<\/em><\/span>\n$ Species : Factor w\/ 3 levels \u201csetosa\u201d,\u201dversicolor\u201d,..: 1 1 1 1 1 1 1 1 1 1 \u2026<\/em><\/span><\/code><\/pre>\n\n\n\n
Load the MASS package<\/a>:<\/p>\n\n\n\n
library(MASS)<\/em><\/span><\/code><\/pre>\n\n\n\n
And perform linear discriminant analysis:<\/p>\n\n\n\n
lda_iris <- lda(iris$Species ~ iris[,1] + iris[,2] + iris[,3] + iris[,4])<\/em><\/span>\nlda_iris<\/em><\/span>\nCall:\nlda(iris$Species ~ iris[, 1] + iris[, 2] + iris[, 3] + iris[, \n    4])\n\nPrior probabilities of groups:\n    setosa versicolor  virginica \n 0.3333333  0.3333333  0.3333333 \n\nGroup means:\n           iris[, 1] iris[, 2] iris[, 3] iris[, 4]\nsetosa         5.006     3.428     1.462     0.246\nversicolor     5.936     2.770     4.260     1.326\nvirginica      6.588     2.974     5.552     2.026\n\nCoefficients of linear discriminants:\n                 LD1         LD2\niris[, 1]  0.8293776 -0.02410215\niris[, 2]  1.5344731 -2.16452123\niris[, 3] -2.2012117  0.93192121\niris[, 4] -2.8104603 -2.83918785\n\nProportion of trace:\n   LD1    LD2 \n0.9912 0.0088 <\/mark><\/em><\/code><\/pre>\n\n\n\n
Now use the model to predict the species of each plant:<\/p>\n\n\n\n
lda_iris_predict <- predict(<\/span>lda_iris, <\/span>iris[,1:4])<\/span><\/span><\/em>

<\/code><\/pre>\n\n\n\n
Attach this predicted variable (stored in lda_iris_predict$class) to the original iris data frame as a new variable called Predict and have a look at the top (head) of the data frame:<\/p>\n\n\n\n
iris$Predict <- lda_iris_predict$class
head(iris)<\/mark>
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species Predic
1          5.1         3.5          1.4         0.2  setosa  setosa
2          4.9         3.0          1.4         0.2  setosa  setosa
3          4.7         3.2          1.3         0.2  setosa  setosa
4          4.6         3.1          1.5         0.2  setosa  setosa
5          5.0         3.6          1.4         0.2  setosa  setosa
6          5.4         3.9          1.7         0.4  setosa  setosa<\/mark><\/em>

<\/code><\/pre>\n\n\n\n
The values of the discriminant functions can be found by:<\/p>\n\n\n\n
lda_iris_predict$x[,1] # values for the first discriminant function<\/mark><\/em>\n         1          2          3          4          5          6          7          8          9         10         11         12         13         14         15 \n 8.0617998  7.1286877  7.4898280  6.8132006  8.1323093  7.7019467  7.2126176  7.6052935  6.5605516  7.3430599  8.3973865  7.2192969  7.3267960  7.5724707  9.8498430 \n.....<\/em>\n.....\n       136        137        138        139        140        141        142        143        144        145        146        147        148        149        150 \n-6.7967163 -6.5244960 -4.9955028 -3.9398530 -5.2038309 -6.6530868 -5.1055595 -5.5074800 -6.7960192 -6.8473594 -5.6450035 -5.1795646 -4.9677409 -5.8861454 -4.6831543 <\/em><\/mark>\n\nlda_iris_predict$x[,2] # values for the second discriminant function<\/em><\/span>\n           1            2            3            4            5            6            7            8            9           10           11           12           13 \n-0.300420621  0.786660426  0.265384488  0.670631068 -0.514462530 -1.461720967 -0.355836209  0.011633838  1.015163624  0.947319209 -0.647363392  0.109646389  1.072989426 \n .....<\/mark><\/em>\n.....\n         144          145          146          147          148          149          150 \n-1.460686950 -2.428950671 -1.677717335  0.363475041 -0.821140550 -2.345090513 -0.332033811 <\/mark><\/em><\/code><\/pre>\n\n\n\n
How good were the predictions? Just create a confusion table of the classifier (Species) agains the predictor (Predict):<\/p>\n\n\n\n
table(iris$Species, iris$Predict)<\/em><\/span>

<\/code><\/pre>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t
<\/td>Setosa<\/th>Versicolor<\/th>Virginica<\/th>\n<\/tr>\n<\/thead>\n
Setosa<\/td>50<\/td>0<\/td>0<\/td>\n<\/tr>\n
Versicolor<\/td>0<\/td>48<\/td>2<\/td>\n<\/tr>\n
Virginica<\/td>0<\/td>1<\/td>49<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
So, all Setosa species were predicted correctly. Two Versicolor species were incorrectly labelled as Virginica and one Virginica was incorrectly labelled as Versicolor. Consequently, the accuracy is:<\/p>\n\n\n\n
\\(Acc = \\frac{50 + 48 + 49}{150}  = 98\\% \\)