Dimension reduction techniques

The idea of principal component analysis (PCA) is to reduce the dimensionality of a data set consisting of a large number of related variables while retaining as much variance in the data as possible. PCA finds a set of new variables whose original variables are only their linear combinations. The new variables are called principal components (PC). These principal components are orthogonal: in a 3D case, the principal components are perpendicular to each other. X cannot be represented by Y or Y cannot be represented by Z.

The following figure shows the intuition of PCA: it “rotates” the axes to better align with your data. The first principal component will capture most of the variance in the data, followed by the second, third, and so on. Therefore, the new data will have fewer dimensions.

Let's use the iris dataset to illustrate PCA:

# Use the iris dataset to illustrate PCA:
import pandas as pd
url = “https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"
# load dataset into Pandas DataFrame
df = pd.read_csv(url, names=['sepal length','sepal width','petal length','petal width','target'])
df.head()

Note that this IRIS dataset comes with the target variable. In PCA you only transform the X variables without the target Y variable.

All variables must be on the same scale before applying PCA, otherwise a feature with large values will dominate the result.

Below I am using StandardScaler in scikit-learn to standardize dataset features on unit scale (mean=0 and variance=1).

from sklearn.preprocessing import StandardScaler
variables = ['sepal length', 'sepal width', 'petal length', 'petal width']
x = df.loc[:, variables].values
y = df.loc[:,['target']].values
x = StandardScaler().fit_transform(x)
x = pd.DataFrame(x)

There are four characteristics in the original data. PCA will therefore supply the same number of main components.

from sklearn.decomposition import PCA
pca = pca()
x_pca = pca.fit_transform(x)
x_pca = pd.DataFrame(x_pca)
x_pca.head()

What are the variances explained by each of the principal components? Use pca.explained_variance_ratio_ to return a vector of the variance:

explained_variance = pca.explained_variance_ratio_
explained_variancearray([0.72770452, 0.23030523, 0.03683832, 0.00515193])

It shows that the first principal component represents a variance of 72.22 %, the second, third and fourth represent a variance of 23.9 %, 3.68 % and 0.51 % respectively. We can say that 72.22 + 23.9 = 96.21% of information is captured by the first and second principal components.

We often want to keep only the meaningful features and remove the insignificant ones. A rule of thumb is to keep the major principal components that capture large variance and ignore the small ones.

We can plot the results using the first two components. Let's add the target variable y to the new x_pca data:

x_pca['target']=y
x_pca.columns = ['PC1','PC2','PC3','PC4','target']
x_pca.head()

The result shows that the data is separable in the new space.

import matplotlib.pyplot as plt
fig = plt. figure()
ax = fig.add_subplot(1,1,1) 
ax.set_xlabel('Main Component 1') 
ax.set_ylabel('Main Component 2') 
ax.set_title('2 component PCA') 
targets = ['Iris-setosa', 'Iris-versicolor', 'Iris-virginica']
colors = ['r', 'g', 'b']
for target, color in zip(targets, colors):
 indicesToKeep = x_pca['target'] == target
 ax.scatter(x_pca.loc[indicesToKeep, 'PC1']
 , x_pca.loc[indicesToKeep, 'PC2']
 , c = color
 , s = 50)
ax.legend(targets)
ax.grid()

How do we use PCA to detect outliers? Let me give you the intuition. After the transformation, the "normal" data points will align along the eigenvectors (new axes) with small eigenvalues. Outliers are far from eigenvectors with large eigenvalues. Therefore, the distances between each data point and the eigenvectors become a measure of the outlier. A large distance indicates an anomaly.