Bayesian information criterion
In statistics, the Bayesian information criterion (BIC) or Schwarz information criterion (also SIC, SBC, SBIC) is a criterion for model selection among a finite set of models; the model with the lowest BIC is preferred. It is based, in part, on the likelihood function and it is closely related to the Akaike information criterion (AIC).
When fitting models, it is possible to increase the likelihood by adding parameters, but doing so may result in overfitting. Both BIC and AIC attempt to resolve this problem by introducing a penalty term for the number of parameters in the model; the penalty term is larger in BIC than in AIC.
The BIC was developed by Gideon E. Schwarz and published in a 1978 paper, where he gave a Bayesian argument for adopting it.
- = the maximized value of the likelihood function of the model , i.e. , where are the parameter values that maximize the likelihood function;
- = the observed data;
- = the number of data points in , the number of observations, or equivalently, the sample size;
- = the number of parameters estimated by the model. For example, in multiple linear regression, the estimated parameters are the intercept, the slope parameters, and the constant variance of the errors; thus, .
Konishi and Kitagawa (2008, p. 217) derive the BIC to approximate the distribution of the data, integrating out the parameters using Laplace's method, starting with the following:
where is the prior for under model .
where is the average observed information per observation, and prime () denotes transpose of the vector . To the extent that is negligible and is relatively linear near , we can integrate out to get the following:
where BIC is defined as above, and either (a) is the Bayesian posterior mode or (b) uses the MLE and the prior has nonzero slope at the MLE. Then the posterior
- It is independent of the prior.
- It can measure the efficiency of the parameterized model in terms of predicting the data.
- It penalizes the complexity of the model where complexity refers to the number of parameters in the model.
- It is approximately equal to the minimum description length criterion but with negative sign.
- It can be used to choose the number of clusters according to the intrinsic complexity present in a particular dataset.
- It is closely related to other penalized likelihood criteria such as Deviance information criterion and the Akaike information criterion.
- the above approximation is only valid for sample size much larger than the number of parameters in the model.
- the BIC cannot handle complex collections of models as in the variable selection (or feature selection) problem in high-dimension.
Gaussian special case
Under the assumption that the model errors or disturbances are independent and identically distributed according to a normal distribution and that the boundary condition that the derivative of the log likelihood with respect to the true variance is zero, this becomes (up to an additive constant, which depends only on n and not on the model):
where is the error variance. The error variance in this case is defined as
In terms of the residual sum of squares (RSS) the BIC is
When testing multiple linear models against a saturated model, the BIC can be rewritten in terms of the deviance as:
where is the number of model parameters in the test.
When picking from several models, the one with the lowest BIC is preferred. The BIC is an increasing function of the error variance and an increasing function of k. That is, unexplained variation in the dependent variable and the number of explanatory variables increase the value of BIC. Hence, lower BIC implies either fewer explanatory variables, better fit, or both. The strength of the evidence against the model with the higher BIC value can be summarized as follows:
|ΔBIC||Evidence against higher BIC|
|0 to 2||Not worth more than a bare mention|
|2 to 6||Positive|
|6 to 10||Strong|
The BIC generally penalizes free parameters more strongly than the Akaike information criterion, though it depends on the size of n and relative magnitude of n and k.
It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable are identical for all estimates being compared. The models being compared need not be nested, unlike the case when models are being compared using an F-test or a likelihood ratio test.
BIC for high-dimensional model
For high dimensional model with the number of potential variables , and the true model size is bounded by a constant, modified BICs has been proposed in Chen and Chen (2008) and Gao and Song (2010). For high dimensional model with the number of variables , and the true model size is unbounded, a high dimensional BIC has been proposed in Gao and Carroll (2017). The high dimensional BIC is of the form:
where can be any number greater than zero.
Gao and Carroll (2017) proposed a pseudo-likelihood BIC for which the pseudo log-likelihood is used instead of the true log-likelihood. The high dimensional pseudo-likelihood BIC is of the form:
where is an estimated degrees of freedom, and the constant is an unknown constant.
To achieve the theoretical model selection consistency for divergent , the two high dimensional BICs above require the multiplicative factor . However, in practical use, the high dimensional BIC can take a simpler form:
where various choices of the multiplicative factor can be used. In empirical studies, or can be used and it is shown to have good empirical performance.
- Schwarz, Gideon E. (1978), "Estimating the dimension of a model", Annals of Statistics, 6 (2): 461–464, doi:10.1214/aos/1176344136, MR 0468014.
- Wit, Ernst; Edwin van den Heuvel; Jan-Willem Romeyn (2012). "'All models are wrong...': an introduction to model uncertainty" (PDF). Statistica Neerlandica. 66 (3): 217–236. doi:10.1111/j.1467-9574.2012.00530.x.
- NOTE: The AIC, AICc and BIC defined by Claeskens and Hjort (2008) is the negative of that defined in this article and in most other standard references.
- Giraud, C. (2015). Introduction to high-dimensional statistics. Chapman & Hall/CRC. ISBN 9781482237948.
- Priestley, M.B. (1981). Spectral Analysis and Time Series. Academic Press. ISBN 978-0-12-564922-3. (p. 375).
- Kass, Robert E.; Raftery, Adrian E. (1995), "Bayes Factors", Journal of the American Statistical Association, 90 (430): 773–795, doi:10.2307/2291091, ISSN 0162-1459, JSTOR 2291091.
- Bhat, H. S.; Kumar, N (2010). "On the derivation of the Bayesian Information Criterion" (PDF). Archived from the original (PDF) on 28 March 2012. Cite journal requires
- Claeskens, G.; Hjort, N. L. (2008), Model Selection and Model Averaging, Cambridge University Press. NOTE: The AIC and AICc defined by Claeskens and Hjort are the negative of that defined by most other authors.
- Findley, D. F. (1991). "Counterexamples to parsimony and BIC". Annals of the Institute of Statistical Mathematics. 43 (3): 505–514. doi:10.1007/BF00053369.
- Kass, R. E.; Wasserman, L. (1995). "A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion". Journal of the American Statistical Association. 90 (431): 928–934. doi:10.2307/2291327. JSTOR 2291327.
- Konishi, Sadanori; Kitagawa, Genshiro (2008). Information criteria and statistical modeling. Springer. ISBN 978-0-387-71886-6.
- Liddle, A. R. (2007). "Information criteria for astrophysical model selection". Monthly Notices of the Royal Astronomical Society. 377 (1): L74–L78. arXiv:astro-ph/0701113. Bibcode:2007MNRAS.377L..74L. doi:10.1111/j.1745-3933.2007.00306.x.
- McQuarrie, A. D. R.; Tsai, C.-L. (1998). Regression and Time Series Model Selection. World Scientific.
- Chen, J.; Chen, Z. (2008). "Extended Bayesian information criteria for model selection with large model spaces". Biometrika. 95 (3): 759–771. CiteSeerX 10.1.1.505.2456. doi:10.1093/biomet/asn034.
- Gao, X.; Song, P. (2010). "Composite likelihood Bayesian information criteria for model selection in high-dimensional data". Journal of the American Statistical Association. 105 (492): 1531–1540. doi:10.1198/jasa.2010.tm09414.
- Gao, X.; Carroll, R. J. (2017). "Data integration with high dimensionality". Biometrika. 104 (2): 251–272. doi:10.1093/biomet/asx023. PMC 5532816. PMID 28757650.