Iteratively reweighted least squares
|Part of a series on Statistics|
IRLS is used to find the maximum likelihood estimates of a generalized linear model, and in robust regression to find an M-estimator, as a way of mitigating the influence of outliers in an otherwise normally-distributed data set. For example, by minimizing the least absolute errors rather than the least square errors.
L1 minimization for sparse recovery
IRLS can be used for ℓ1 minimization and smoothed ℓp minimization, p < 1, in compressed sensing problems. It has been proved that the algorithm has a linear rate of convergence for ℓ1 norm and superlinear for ℓt with t < 1, under the restricted isometry property, which is generally a sufficient condition for sparse solutions. However, in most practical situations, the restricted isometry property is not satisfied.
Lp norm linear regression
the IRLS algorithm at step t + 1 involves solving the weighted linear least squares problem:
where W(t) is the diagonal matrix of weights, usually with all elements set initially to:
and updated after each iteration to:
In the case p = 1, this corresponds to least absolute deviation regression (in this case, the problem would be better approached by use of linear programming methods, so the result would be exact) and the formula is:
To avoid dividing by zero, regularization must be done, so in practice the formula is:
where is some small value, like 0.0001. Note the use of in the weighting function is equivalent to the Huber loss function in robust estimation.
- C. Sidney Burrus, Iterative Reweighted Least Squares
- Chartrand, R.; Yin, W. (March 31 – April 4, 2008). "Iteratively reweighted algorithms for compressive sensing". IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2008. pp. 3869–3872.
- Daubechies, I.; Devore, R.; Fornasier, M.; Güntürk, C. S. N. (2010). "Iteratively reweighted least squares minimization for sparse recovery". Communications on Pure and Applied Mathematics. 63: 1–38. arXiv:0807.0575. doi:10.1002/cpa.20303.
- Gentle, James (2007). "6.8.1 Solutions that Minimize Other Norms of the Residuals". Matrix algebra. Springer Texts in Statistics. New York: Springer. doi:10.1007/978-0-387-70873-7. ISBN 978-0-387-70872-0.
- William A. Pfeil, Statistical Teaching Aids, Bachelor of Science thesis, Worcester Polytechnic Institute, 2006
- Fox, J.; Weisberg, S. (2013),Robust Regression, Course Notes, University of Minnesota
- Numerical Methods for Least Squares Problems by Åke Björck (Chapter 4: Generalized Least Squares Problems.)
- Practical Least-Squares for Computer Graphics. SIGGRAPH Course 11