PCA, based on linear algebra, identifies the linear combinations of the original predictor set that have the largest effect on the response variable. These linear combinations, called principal components, can be ranked according to their effect on the variance in response. You can select the top few principal components of the data set, and use this smaller set of predictors instead of the original larger predictor set.
Principal components are determined by calculating the eigenvectors of an MxM matrix, where M is the number of predictors in the original data set. Because they are eigenvectors, the principal components are mutually orthogonal and linearly independent.