Essential Science IndicatorsSMfrom
Reuters, the paper "Fast and robust fixed-point
algorithms for independent component analysis,"
(Hyvarinen A, IEEE Trans. Neural Networks
10: 626-34, May 1999) is ranked at #7 among
Highly Cited Engineering papers published over the past
decade, with 636 citations to its credit up to April
Author Professor Aapo Hyvarinen is based at the
University of Helsinki, where he is Professor of
Computational Data Analysis jointly in the Department
of Mathematics and Statistics and the Department of
In this interview, he
talks with ScienceWatch.com about his paper and
its impact on the field of independent component
What factors prompted you to undertake this study,
and how was it conducted?
In August 1995, I started my Ph.D. at the Helsinki University of
Technology, Finland, in a research group focusing on the method called
independent component analysis (ICA). ICA is a statistical method for
analyzing multidimensional data, which can find underlying components in
the data based on independence of the components. Such linear
decompositions had been used before; factor analysis and principal
component analysis are two well-known methods. However, it was also
well-known that those methods couldn't find the actual underlying
components which generated the data.
small square is one
feature of small colour
I was really fascinated by ICA. In fact, when I first saw the description
of the model, I found it difficult to believe the model could be estimated
(i.e., the problem could be solved). I thought that if it really were
possible, I certainly would have heard about it during my undergraduate
studies! But the trick is that ICA is based on the rather unconventional
framework on non-normality (non-Gaussianity), which is why it became
well-known only in the 1990s.
During the first year of my Ph.D., my advisor, Prof. Erkki Oja, suggested
that I consider fixed-point algorithms to solve the demanding computational
problems encountered in ICA. I was able to find one that worked quite well,
and we soon published it (Hyvarinen A and Oja E, "A Fast Fixed-Point
Algorithm for Independent Component Analysis," Neural Computation
However, when I was writing that paper I already thought that this could
not be the final answer because the method was statistically not very good.
The ICA problem has two parts, statistical (how to get maximum information
out of your data) and computational (how to do the required computations as
efficiently as possible). The initial fixed-point algorithm in our 1997
paper provided a good solution for the computational part, but it was
rather bad from the statistical viewpoint. In particular, it suffered from
being extremely non-robust, i.e. very sensitive to outliers. This means
that a single bad measurement point—for example, due to a faulty
sensor—can ruin the whole analysis.
Thus, I struggled for a few months more before I found a fixed-point
algorithm which was not only fast but also robust. The algorithm was
initially published in the proceedings of the International Conference
in Acoustics, Speech, and Signal Processing in 1997. Then, I spent
some more time analyzing the algorithm and polishing the theory, and
finally at the end of 1997, I had finished the manuscript. However, as is
typical in many engineering and mathematical sciences, it took more than 18
months before the article was actually published.
How was the paper received by the community?
" I was really fascinated by ICA. In
fact, when I first saw the description of the
model, I found it difficult to believe the
model could be estimated (i.e., the problem
could be solved)."
There were some initial problems due to the fact that the utility of ICA
was not widely appreciated in the 1990s. In my initial conference
presentation I mentioned above, a senior scientist came to my poster and
basically explained how I had got it all wrong! But when the paper was
published in 1999, people were starting to appreciate ICA more and more,
and many new application areas were being found, so my paper was published
at the right time.
Essential for the success of the algorithm was the development, by Dr.
Jarmo Hurri, of an excellent software package that enabled people to use it
with minimal effort.
What are the applications for these algorithms?
The reason for the popularity of the algorithm is that ICA can be applied
in almost any discipline of science, as well as in technology and even some
of the humanities. Whenever you measure several variables at many different
time points or for many individuals, your data can potentially be analyzed
by ICA. Often, of course, the analysis does not give anything useful, but
because of the great generality of the model, cases where it actually is
useful are numerous.
Probably the most successful application field is biomedical engineering,
where the assumptions of the model (linear mixing, non-Gaussian source
signals) are quite often approximately fulfilled.
Where have you taken this work since the publication of
ICA has been the basis of most of my research ever since. Regarding this
particular algorithm, I have published some extensions, but in general, the
extensions are not nearly as important as the original algorithm. This has
actually been rather annoying: I have worked really hard to develop an even
better algorithm for ICA, but I have never found anything that beats this
algorithm, which I developed in my first year of Ph.D. studies!
Fortunately, there is much more to this direction of research than
developing better algorithms. I have written a book simply called
Independent Component Analysis with Prof. Juha Karhunen and Prof.
Erkki Oja, I have developed models which generalize the ICA model, and I
have applied these models in computational neuroscience (see, e.g., our
Aapo Hyvarinen, Ph.D.
Department of Mathematics and Statistics
Department of Computer Science
University of Helsinki
Image features learned by ICA. Each small square is one feature of small
colour image patches. The image data consisted of wild-life photographs.
From P.O. Hoyer and A. Hyvärinen. Independent Component Analysis
Applied to Feature Extraction from Colour and Stereo Images. Network:
Computation in Neural Systems, 11(3):191-210, 2000.
Click for a larger view.