# Linear Discriminant Analysis¶

LDA learns a linear classifier via finding a projection matrix that maximally discriminates the provided classes. The learned linear classification rule is optimal under the assumption that both classes a gaussian distributed with equal co-variance. To find a linear separation $${\bf w}$$ in training, the in-between class variance is maximized and the within class variance is minimized. The projection matrix is computed by maximizing the following objective:

$J({\bf w})=\frac{{\bf w^T} S_B {\bf w}}{{\bf w^T} S_W {\bf w}}$

where $${\bf S_B}$$ is between class scatter matrix and $${\bf S_W}$$ is within class scatter matrix. The above derivation of LDA requires the invertibility of the within class matrix. This condition however, is violated when there are fewer data-points than dimensions. In this case SVD is used to compute projection matrix using an orthonormal basis $${\bf Q}$$

${\bf W} := {\bf Q} {\bf{W^\prime}}$

See Chapter 16 in [Bar12] for a detailed introduction.

## Example¶

We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CBinaryLabels from files with training and test data.

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = BinaryLabels(f_labels_train)
labels_test = BinaryLabels(f_labels_test)

features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = BinaryLabels(f_labels_train);
labels_test = BinaryLabels(f_labels_test);

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);

features_train = Shogun::RealFeatures.new f_feats_train
features_test = Shogun::RealFeatures.new f_feats_test
labels_train = Shogun::BinaryLabels.new f_labels_train
labels_test = Shogun::BinaryLabels.new f_labels_test

features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- BinaryLabels(f_labels_train)
labels_test <- BinaryLabels(f_labels_test)

features_train = shogun.RealFeatures(f_feats_train)
features_test = shogun.RealFeatures(f_feats_test)
labels_train = shogun.BinaryLabels(f_labels_train)
labels_test = shogun.BinaryLabels(f_labels_test)

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);

auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CBinaryLabels>(f_labels_train);
auto labels_test = some<CBinaryLabels>(f_labels_test);


We create an instance of the CLDA classifier and set features and labels. By default, Shogun automatically chooses the decomposition method based on $${N<=D}$$ or $${N>D}$$.

lda = LDA()
lda.set_features(features_train)
lda.set_labels(labels_train)

lda = LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);

LDA lda = new LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);

lda = Shogun::LDA.new
lda.set_features features_train
lda.set_labels labels_train

lda <- LDA()
lda$set_features(features_train) lda$set_labels(labels_train)

lda = shogun.LDA()
lda:set_features(features_train)
lda:set_labels(labels_train)

LDA lda = new LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);

auto lda = some<CLDA>();
lda->set_features(features_train);
lda->set_labels(labels_train);


Then we train and apply it to test data, which here gives CBinaryLabels.

lda.train()
labels_predict = lda.apply_binary(features_test)

lda.train();
labels_predict = lda.apply_binary(features_test);

lda.train();
BinaryLabels labels_predict = lda.apply_binary(features_test);

lda.train
labels_predict = lda.apply_binary features_test

lda$train() labels_predict <- lda$apply_binary(features_test)

lda:train()
labels_predict = lda:apply_binary(features_test)

lda.train();
BinaryLabels labels_predict = lda.apply_binary(features_test);

lda->train();
auto labels_predict = lda->apply_binary(features_test);


We can extract weights $${\bf w}$$.

w = lda.get_w()

w = lda.get_w();

DoubleMatrix w = lda.get_w();

w = lda.get_w

w <- lda$get_w()  w = lda:get_w()  double[] w = lda.get_w();  auto w = lda->get_w();  We can evaluate test performance via e.g. CAccuracyMeasure. eval = AccuracyMeasure() accuracy = eval.evaluate(labels_predict, labels_test)  eval = AccuracyMeasure(); accuracy = eval.evaluate(labels_predict, labels_test);  AccuracyMeasure eval = new AccuracyMeasure(); double accuracy = eval.evaluate(labels_predict, labels_test);  eval = Shogun::AccuracyMeasure.new accuracy = eval.evaluate labels_predict, labels_test  eval <- AccuracyMeasure() accuracy <- eval$evaluate(labels_predict, labels_test)

eval = shogun.AccuracyMeasure()
accuracy = eval:evaluate(labels_predict, labels_test)

AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);

auto eval = some<CAccuracyMeasure>();
auto accuracy = eval->evaluate(labels_predict, labels_test);


## References¶

Wikipedia: Linear_discriminant_analysis

 [Bar12] D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.