{"id":2124,"date":"2018-01-01T23:11:00","date_gmt":"2018-01-01T23:11:00","guid":{"rendered":"http:\/\/pcool.dyndns.org:8080\/statsbook\/?page_id=2124"},"modified":"2025-06-30T17:25:12","modified_gmt":"2025-06-30T16:25:12","slug":"principle-component-analysis","status":"publish","type":"page","link":"https:\/\/pcool.dyndns.org\/index.php\/principle-component-analysis\/","title":{"rendered":"Principle Component Analysis"},"content":{"rendered":"\n

Principle Component Analysis\u00a0(PCA) is a\u00a0linear transformation technique. It is an unsupervised<\/strong> machine learning method and consequently no classifier is necessary. To illustrate the method, R\u2019s build in data set \u201ciris\u201d is used.<\/p>\n\n\n\n

The build in iris data set contains the sepal and petal length and width of three different types of iris plants: Setosa, Versicolor and Virginica. Have a look at the iris data set:<\/p>\n\n\n\n

head(iris)<\/em><\/span>\n Sepal.Length Sepal.Width Petal.Length Petal.Width Species<\/em><\/span>\n1 5.1 3.5 1.4 0.2 setosa<\/em><\/span>\n2 4.9 3.0 1.4 0.2 setosa<\/em><\/span>\n3 4.7 3.2 1.3 0.2 setosa<\/em><\/span>\n4 4.6 3.1 1.5 0.2 setosa<\/em><\/span>\n5 5.0 3.6 1.4 0.2 setosa<\/em><\/span>\n6 5.4 3.9 1.7 0.4 setosa<\/em><\/span>\nstr(iris)<\/em><\/span>\n'data.frame': 150 obs. of 5 variables:<\/em><\/span>\n $ Sepal.Length: num 5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...<\/em><\/span>\n $ Sepal.Width : num 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...<\/em><\/span>\n $ Petal.Length: num 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...<\/em><\/span>\n $ Petal.Width : num 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...<\/em><\/span>\n $ Species : Factor w\/ 3 levels \"setosa\",\"versicolor\",..: 1 1 1 1 1 1 1 1 1 1 ...<\/em><\/span><\/code><\/pre>\n\n\n\n
A plot matrix of the first four numerical variables can be illustrative and is easily obtained:<\/p>\n\n\n\n
plot(iris[,1:4])<\/span><\/em><\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
This illustrates that some variables are associated with each other. The aim of principle component analysis is to describe the data with linear combinations of the variables. Would it be possible to describe the data with less than 4 variables?? It is important to standardise the data first (mean = 0 and standard deviation = 1) as otherwise, the magnitude of a variable could have an influence. To standardise the four numerical variables (the categorical classifier is not needed as this is an unsupervised learning technique):<\/p>\n\n\n\n
iris_standardised <- scale(iris[,1:4])\nhead(iris_standardised)<\/mark>\n     Sepal.Length Sepal.Width Petal.Length Petal.Width\n[1,]   -0.8976739  1.01560199    -1.335752   -1.311052\n[2,]   -1.1392005 -0.13153881    -1.335752   -1.311052\n[3,]   -1.3807271  0.32731751    -1.392399   -1.311052\n[4,]   -1.5014904  0.09788935    -1.279104   -1.311052\n[5,]   -1.0184372  1.24503015    -1.335752   -1.311052\n[6,]   -0.5353840  1.93331463    -1.165809   -1.048667<\/mark><\/em><\/code><\/pre>\n\n\n\n
Next perform the principle component analysis on these standardised data and display a summary:<\/p>\n\n\n\n
pca_iris <- prcomp(iris_standardised)
summary(pca_iris)<\/mark>
Importance of components:
                          PC1    PC2     PC3     PC4
Standard deviation     1.7084 0.9560 0.38309 0.14393
Proportion of Variance 0.7296 0.2285 0.03669 0.00518
Cumulative Proportion  0.7296 0.9581 0.99482 1.00000<\/mark><\/em>

<\/code><\/pre>\n\n\n\n
The summary shows that the first principle component explains 73% of the variance, the second 96% and the third 99%. All variance is explained by all four components. To check this, sum the variances of each component (the standard deviation is stored in pca_iris$sdev):<\/p>\n\n\n\n
pca_iris$sdev<\/span><\/em>\n[1] 1.7083611 0.9560494 0.3830886 0.1439265<\/em><\/span>\n# all variance should add up to the number of variables<\/em><\/span>\nsum(pca_iris$sdev^2) <\/span><\/em>\n[1] 4<\/em><\/span><\/code><\/pre>\n\n\n\n
So, the sum of the variances is four, which is the number of variables in the data set. How many principle components should be used to describe the data set? One component explains only 73% of the variance. Two components seems reasonable, but should be quantified. First draw a sreeplot:<\/p>\n\n\n\n
screeplot(pca_iris, type = 'lines')<\/em><\/span><\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The elbow of the plot is at 3 principle components, so it is reasonable to maintain 2 principle components. Furthermore, according to Kaiser’s criterium, the variance of the last component on the standardised data should be less than 1; so again it is reasonable to use 2 principle components. Finally, to explain at least 80% of the variance, again 2 components are required.<\/p>\n\n\n\n
The loadings of the first principle component can be obtained by:<\/p>\n\n\n\n
pca_iris$rotation[,1]<\/span><\/em>\nSepal.Length  Sepal.Width Petal.Length  Petal.Width \n   0.5210659   -0.2693474    0.5804131    0.5648565 <\/mark><\/em><\/code><\/pre>\n\n\n\n
and the second principle component by:<\/p>\n\n\n\n
pca_iris$rotation[,2]<\/em><\/span>\nSepal.Length  Sepal.Width Petal.Length  Petal.Width \n -0.37741762  -0.92329566  -0.02449161  -0.06694199  <\/span><\/mark><\/em><\/code><\/pre>\n\n\n\n
The sum of all loadings should be one:<\/p>\n\n\n\n
sum(pca_iris$rotation[,1]^2) # should be one<\/span><\/em>\n[1] 1<\/em><\/span>\nsum(pca_iris$rotation[,2]^2) # should be one<\/span><\/em>\n[1] 1<\/em><\/span><\/code><\/pre>\n\n\n\n
Looking at the loadings, the first principle component can be regarded as a contrast between sepal width and the other three variables (sepal length, petal width and petal length). The second principle component takes sepal width and length into account, but not petal length or width.<\/p>\n\n\n\n
The first principle component can be plotted versus the second principle component to illustrate how the principle components separate the data into clusters. Remember, no classifier has been used, but the clusters are compared with the classifier as an illustration how principle component analysis can aid to gain insight:<\/p>\n\n\n\n
iris_pca_df <- data.frame(first = pca_iris$x[,1], <\/em><\/span>\nsecond = pca_iris$x[,2], species = iris$Species)<\/em><\/span>\n\nlibrary(ggplot2)<\/em><\/span>\nggplot(iris_pca_df, aes(x = first, y = second, colour = species)) +<\/em><\/span> \n  geom_point() +<\/em><\/span>\n  ggtitle(\"Principle Component Analysis\") +<\/em><\/span>\n  scale_x_continuous(\"First Principle Component\") +<\/em><\/span><\/em><\/span>\n  scale_y_continuous(\"Second Principle Component\") +<\/em><\/span>\n  theme_bw()<\/em><\/span><\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
As can be seen the first principle component is good in separating Setosa species from the other two and to a lesser extend in separating Versicolar and Virginica species. The second principle component helps to separate Versicolar and Virginica species.<\/p>\n","protected":false},"excerpt":{"rendered":"
Principle Component Analysis\u00a0(PCA) is a\u00a0linear transformation technique. It is an unsupervised machine learning method and consequently no classifier is necessary. To illustrate the method, R\u2019s build in data set \u201ciris\u201d is used. The build in iris data set contains the sepal and petal length and width of three different types of iris plants: Setosa, Versicolor and Virginica. […]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"inline_featured_image":false,"footnotes":""},"class_list":["post-2124","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages\/2124","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/comments?post=2124"}],"version-history":[{"count":1,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages\/2124\/revisions"}],"predecessor-version":[{"id":4622,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages\/2124\/revisions\/4622"}],"wp:attachment":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/media?parent=2124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}