Derivation of Symmetric PCA Learning Rules from a Novel Objective Function
Neural learning rules for principal component / subspace analysis (PCA / PSA)
can be derived by maximizing an objective function (summed variance of the
projection on the subspace axes) under an orthonormality constraint. For a
subspace with a single axis, the optimization produces the principal
eigenvector of the data covariance matrix. Hierarchical learning rules with
deflation procedures can then be used to extract multiple eigenvectors.
However, for a subspace with multiple axes, the optimization leads to PSA
learning rules which only converge to axes spanning the principal subspace but
not to the principal eigenvectors. A modified objective function with distinct
weight factors had to be introduced produce PCA learning rules. Optimization of
the objective function for multiple axes leads to symmetric learning rules
which do not require deflation procedures. For the PCA case, the estimated
principal eigenvectors are ordered (w.r.t. the corresponding eigenvalues)
depending on the order of the weight factors.
Here we introduce an alternative objective function where it is not necessary
to introduce fixed weight factors; instead, the alternative objective function
uses squared summands. Optimization leads to symmetric PCA learning rules which
converge to the principal eigenvectors, but without imposing an order. In place
of the diagonal matrices with fixed weight factors, variable diagonal matrices
appear in the learning rules. We analyze this alternative approach by
determining the fixed points of the constrained optimization. The behavior of
the constrained objective function at the fixed points is analyzed which
confirms both the PCA behavior and the fact that no order is imposed. Different
ways to derive learning rules from the optimization of the objective function
are presented. The role of the terms in the learning rules obtained from these
derivations is explored.