Orthogonal Data Augmentation for Bayesian Model Averaging

Authors: 
Joyee Ghosh, Merlise A Clyde
Duke University, University of Iowa

May 10 2011

New  title  "Rao-Blackwellization for Bayesian Variable Selection and Model Averaging in Linear and Binary Regression: A Novel Data Augmentation Approach"

Choosing the subset of covariates to use in regression or generalized linear models is a ubiquitous problem. The Bayesian paradigm addresses the problem of model uncertainty by considering models corresponding to all possible subsets of the covariates, where the posterior distribution over models is used to select models or combine them via Bayesian model averaging (BMA). Although conceptually straightforward, BMA is often difficult to implement in practice, since either the number of covariates is too large for enumeration of all subsets, calculations cannot be done analytically, or both. For orthogonal designs with the appropriate choice of prior, the posterior probability of any model can be calculated without having to enumerate the entire model space and scales linearly with the number of predictors, p. In this article we extend this idea to a much broader class of non-orthogonal design matrices. We propose a novel method which augments the observed non-orthogonal design by at most p new rows to obtain a design matrix with orthogonal columns and generate the ``missing'' response variables in a data augmentation algorithm. We show that our data augmentation approach keeps the original posterior distribution of interest unaltered, and develop methods to construct Rao-Blackwellized estimates of several quantities of interest, including posterior model probabilities of any model, which may not be available from an ordinary Gibbs sampler. Our method can be used for BMA in linear regression and binary regression with non-orthogonal design matrices in conjunction with independent ``spike and slab'' priors with a continuous prior component that is a Cauchy or other heavy tailed distribution that may be represented as a scale mixture of normals. We provide simulated and real examples to illustrate the methodology. Supplemental materials for the manuscript are available online.

 

Please cite as: Ghosh, J. and Clyde, M.A. (2011) Rao-Blackwellization for Bayesian Variable Selection and Model Averaging in Linear and Binary Regression: A Novel Data Augmentation Approach. Journal of the American Statistical Association  106(495): 1041-1052  DOI: 10.1198/jasa.2011.tm10518

Keywords: 

Gibb Sampling, MCMC, Missing Data, Model Uncertainty, Orthogonal Design, BMA, Posterior probability

Manuscript: 

PDF icon 2010-15.pdf

BibTeX Citation: 

@article{doi:10.1198/jasa.2011.tm10518,
author = {Ghosh, Joyee and Clyde, Merlise A.},
title = {Rao–Blackwellization for Bayesian Variable Selection and Model Averaging in Linear and Binary Regression: A Novel Data Augmentation Approach},
journal = {Journal of the American Statistical Association},
volume = {106},
number = {495},
pages = {1041-1052},
year = {2011},
doi = {10.1198/jasa.2011.tm10518},
URL = {http://pubs.amstat.org/doi/abs/10.1198/jasa.2011.tm10518},
eprint = {http://pubs.amstat.org/doi/pdf/10.1198/jasa.2011.tm10518}
}