Fast ICA Algorithm

The deflation-based fast ICA algorithm was first presented in the paper:
[fast_ica_hyvarinen99] A. Hyvarinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. Neural Networks, vol. 10, no. 3, May 1999.
The symmetric fast ICA algorithm in this paper is not the one that is actually in the downloadable code. Both the deflation-based and the symmetric fast ICA algorithm as currently implemented are reviewed in the paper:
[fastica_analysis_06] P. Tichavsky, Z. Koldovsky, E. Oja, “Performance analysis of the FastICA algorithm and Cramer-Rao bounds for linear independent component analysis,” IEEE Trans. Signal Processing, vol. 54, no. 4, April 2006.

Fast ICA code is downloadable from http://www.cis.hut.fi/projects/ica/fastica

Infomax Algorithm

The extended Infomax algorithm, which is what is implemented in the downloadable code, is presented in:
[infomax_sejnowski] T.-W. Lee, M. Girolami, T. J. Sejnowski, “Independent component analysis using an extended Infomax algorithm for mixed sub-Gaussian and super-Gaussian sources,” Neural Computation, vol. 11, no. 2, 1999.

The code is downloadable as part of EEGLab from http://sccn.ucsd.edu/eeglab/

Other algorithms of interest

JADE Algorithm
Cardoso’s cumulant-based JADE algorithm operates on the entire batch of data at once. See this page for a general guide to his work:
http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html
and use this link for matlab code for the JADE algorithm for real-valued data (which is what we are interested in for the EEG application):
http://perso.telecom-paristech.fr/~cardoso/Algo/Jade/jadeR.m

Robust ICA Algorithm

Zarzoso and Comon recently proposed a “robust ICA” algorithm which cites prior work saying that imperfect sphering (because of errors in estimation of the covariance matrix from finite data) can impact the performance of fast ICA. This paper is interesting because it is recent (2010), and hence has a perspective on the field:
[comon_2010] V. Zarzoso, P. Comon, “Robust independent component analysis by iterative maximization of the kurtosis contrast with algebraic optimal step size,” IEEE Trans. Neural Networks, vol. 21, no. 2, February 2010.

The code is downloadble from:
http://www.i3s.unice.fr/~zarzoso/robustica.html

RADICAL Algorithm

This algorithm tries to minimizes entropy directly, estimating it using order statistics. The resource page (including downloadable code is at:
http://www.cs.umass.edu/~elm/ICA/
The algorithm is described in:
[radical_03] E. G. Learned-Miller, J. W. Fisher, “ICA using spacings estimate of entropy,” J. Machine Learning Research, vol. 4, pp. 1271-1295, 2003.

Delving into ICA theory is not required for the homeworks and projects in this course.  However, I have put together a brief guide for those of you who might be interested.  There are many many other papers that I can also point you to if this list is not enough to satisfy your curiosity.

Perhaps the best starting point for understanding the theory behind ICA is Comon’s 1994 paper:
[comon94] P. Comon, “Independent component analysis: a new concept?,” Signal Processing, pp. 287-314, 1994.

The key ideas behind the fast ICA algorithm are presented in the following two papers:
[fast_ica_hyvarinen99] A. Hyvarinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. Neural Networks, vol. 10, no. 3, May 1999.
[hyvarinen_oja_98] A. Hyvarinen, E. Oja, “Independent component analysis by general nonlinear Hebbian-like learning rules,” Signal Processing, pp. 301-313, 1998.
A more recent performance analysis is provided in:
[fastica_analysis_06]P. Tichavsky, Z. Koldovsky, E. Oja, “Performance analysis of the FastICA algorithm and Cramér–Rao bounds for linear independent component analysis,” IEEE Trans. Signal Processing, vol. 54, no. 4, April 2006.

As we discuss in class, the extended Infomax algorithm uses the concept of relative gradient, which is described in the paper by Cardoso and Laheld (this concept is essentially the same as Amari’s concept of natural gradient for our context, but requires less mathematical sophistication to understand):
[cardoso_laheld_96] J.-F. Cardoso, B. H. Laheld, “Equivariant adaptive source separation,” IEEE Trans. Signal Processing, vol. 44, no. 12, December 1996.

The following reference says that sphering with an imperfect estimate of the covariance matrix may lead to performance problems in fast ICA:
[fast_ica_finite_sample_07] S. Bermejo, “Finite sample effects of the fast ICA algorithm,” Neurocomputing, pp. 392-399, March 2007.
This observation appears to be one of the motivations behind the robust ICA algorithm proposed by Zarzoso and Comon.

Demonstrations of spurious solutions to ICA with infomax or the non-cumulant based fast ICA nonlinearities are provided in several papers:
[ica_spurious_2010] F. Ge, J. Ma, Spurious solution of the maximum likelihood approach to ICA,” IEEE Signal Processing Letters, vol. 17, no. 7, July 2010. (this implies, for example, that the Infomax algorithm may be vulnerable to spurious minima).
[ica_spurious_vrins05] F. Vrins, M. Verleysen, “Information theoretic versus cumulant-based contrasts for multimodal source separation,” IEEE Signal Processing Letters, vol. 12, no. 3, March 2005. (this implies, for example, that cumulant-based contrast functions such as the fourth moment in fast ICA or JADE may be less vulnerable to local minima than some of the other nonlinearities used in the fast ICA and Infomax algorithms).
A longer follow-up paper is:
[ica_spurious_vrins07]F. Vrins, D.-T. Pham, M. Verleysen “Mixing and Non-Mixing Local Minima of the Entropy Contrast for Blind Source Separation,” IEEE Trans. Information Theory, vol. 53, no. 3, March 2007.