Archive ScienceWatch

 ScienceWatch
Aapo Hyvarinen Aapo Hyvarinen
Featured Paper Interview

According to Essential Science IndicatorsSM from Thomson Reuters, the paper "Fast and robust fixed-point algorithms for independent component analysis," (Hyvarinen A, IEEE Trans. Neural Networks 10[3]: 626-34, May 1999) is ranked at #7 among Highly Cited Engineering papers published over the past decade, with 636 citations to its credit up to April 30, 2009.

Author Professor Aapo Hyvarinen is based at the University of Helsinki, where he is Professor of Computational Data Analysis jointly in the Department of Mathematics and Statistics and the Department of Computer Science.


In this interview, he talks with ScienceWatch.com about his paper and its impact on the field of independent component analysis.

What factors prompted you to undertake this study, and how was it conducted?

In August 1995, I started my Ph.D. at the Helsinki University of Technology, Finland, in a research group focusing on the method called independent component analysis (ICA). ICA is a statistical method for analyzing multidimensional data, which can find underlying components in the data based on independence of the components. Such linear decompositions had been used before; factor analysis and principal component analysis are two well-known methods. However, it was also well-known that those methods couldn't find the actual underlying components which generated the data.

Figure 1 Click for a larger view and description
Click figure to enlarge and read description.
Each small square is one feature of small colour image... Read full description

I was really fascinated by ICA. In fact, when I first saw the description of the model, I found it difficult to believe the model could be estimated (i.e., the problem could be solved). I thought that if it really were possible, I certainly would have heard about it during my undergraduate studies! But the trick is that ICA is based on the rather unconventional framework on non-normality (non-Gaussianity), which is why it became well-known only in the 1990s.

During the first year of my Ph.D., my advisor, Prof. Erkki Oja, suggested that I consider fixed-point algorithms to solve the demanding computational problems encountered in ICA. I was able to find one that worked quite well, and we soon published it (Hyvarinen A and Oja E, "A Fast Fixed-Point Algorithm for Independent Component Analysis," Neural Computation 9[7]:1483-92, 1997).

However, when I was writing that paper I already thought that this could not be the final answer because the method was statistically not very good. The ICA problem has two parts, statistical (how to get maximum information out of your data) and computational (how to do the required computations as efficiently as possible). The initial fixed-point algorithm in our 1997 paper provided a good solution for the computational part, but it was rather bad from the statistical viewpoint. In particular, it suffered from being extremely non-robust, i.e. very sensitive to outliers. This means that a single bad measurement point—for example, due to a faulty sensor—can ruin the whole analysis.

Thus, I struggled for a few months more before I found a fixed-point algorithm which was not only fast but also robust. The algorithm was initially published in the proceedings of the International Conference in Acoustics, Speech, and Signal Processing in 1997. Then, I spent some more time analyzing the algorithm and polishing the theory, and finally at the end of 1997, I had finished the manuscript. However, as is typical in many engineering and mathematical sciences, it took more than 18 months before the article was actually published.

How was the paper received by the community?

" I was really fascinated by ICA. In fact, when I first saw the description of the model, I found it difficult to believe the model could be estimated (i.e., the problem could be solved)."

There were some initial problems due to the fact that the utility of ICA was not widely appreciated in the 1990s. In my initial conference presentation I mentioned above, a senior scientist came to my poster and basically explained how I had got it all wrong! But when the paper was published in 1999, people were starting to appreciate ICA more and more, and many new application areas were being found, so my paper was published at the right time.

Essential for the success of the algorithm was the development, by Dr. Jarmo Hurri, of an excellent software package that enabled people to use it with minimal effort.

What are the applications for these algorithms?

The reason for the popularity of the algorithm is that ICA can be applied in almost any discipline of science, as well as in technology and even some of the humanities. Whenever you measure several variables at many different time points or for many individuals, your data can potentially be analyzed by ICA. Often, of course, the analysis does not give anything useful, but because of the great generality of the model, cases where it actually is useful are numerous.

Probably the most successful application field is biomedical engineering, where the assumptions of the model (linear mixing, non-Gaussian source signals) are quite often approximately fulfilled.

Where have you taken this work since the publication of this paper?

ICA has been the basis of most of my research ever since. Regarding this particular algorithm, I have published some extensions, but in general, the extensions are not nearly as important as the original algorithm. This has actually been rather annoying: I have worked really hard to develop an even better algorithm for ICA, but I have never found anything that beats this algorithm, which I developed in my first year of Ph.D. studies!

Fortunately, there is much more to this direction of research than developing better algorithms. I have written a book simply called Independent Component Analysis with Prof. Juha Karhunen and Prof. Erkki Oja, I have developed models which generalize the ICA model, and I have applied these models in computational neuroscience (see, e.g., our new book).

Aapo Hyvarinen, Ph.D.
Department of Mathematics and Statistics
and
Department of Computer Science
University of Helsinki
Helsinki, Finland


Figure 1:

Click for a larger view

Figure 1:

Image features learned by ICA. Each small square is one feature of small colour image patches. The image data consisted of wild-life photographs. From P.O. Hoyer and A. Hyvärinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. Network: Computation in Neural Systems, 11(3):191-210, 2000. Click for a larger view.

Aapo Hyvarinen's current most-cited paper in Essential Science Indicators, with 636 cites:
Hyvarinen A, "Fast and robust fixed-point algorithms for independent component analysis," IEEE Trans. Neural Networks 10(3): 626-34, May 1999. Source: Essential Science Indicators from Thomson Reuters.

KEYWORDS: FIXED-POINT ALGORITHMS, INDEPENDENT COMPONENT ANALYSIS, MULTIDIMENSIONAL RANDOM VECTOR, INFORMATION-THEORETIC APPROACH, PROJECTION PURSUIT, BLIND SEPARATION, NEURAL NETWORKS, NATURAL IMAGES, EXTRACTION, ARTIFACTS, FILTERS.

Download this article



2009 : September 2009 - Author Commentaries : Aapo Hyvarinen