Location
Logan, UT
Start Date
5-6-2018 12:00 AM
Description
I explore three related issues concerning pooling of error variances: when is it appropriate (or not) to pool, how best to evaluate equality of variances, and whether there is a cost to never pooling. I focus on pooling decisions in a combined analysis of a multi-site experiment. A-priori, sites should have different error variances. My primary question is whether an analysis that ignores unequal variances is wrong.
I find that ignoring heteroscedasticity between sites maintains, or provides slightly conservative, tests of average treatment effects and treatment-by-site interactions. Models with site-specific variances do provide more powerful tests when variances are different. Never pooling, i.e., using site-specific variances when variances are equal, also reduces power. In contrast to the relatively benign effects of pooling across sites, incorrectly pooling across treatments is much more serious.
AIC-based evaluations of variances are very sensitive to non-normality, with a strong tendency to indicate unequal variances when that is incorrect and the data are non-normal. While Levene’s test is somewhat liberal when errors are skewed or heavy-tailed, it is much more robust than AIC.
I conclude that ignoring site-specific error variances is not wrong, but modeling that heterogeneity will increase power. If there is any possibility that errors are non-normal, I suggest that variance models be evaluated using Levene’s test instead of AIC.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Included in
Pooling of Variances: The Skeleton in the Mixed Model Closet?
Logan, UT
I explore three related issues concerning pooling of error variances: when is it appropriate (or not) to pool, how best to evaluate equality of variances, and whether there is a cost to never pooling. I focus on pooling decisions in a combined analysis of a multi-site experiment. A-priori, sites should have different error variances. My primary question is whether an analysis that ignores unequal variances is wrong.
I find that ignoring heteroscedasticity between sites maintains, or provides slightly conservative, tests of average treatment effects and treatment-by-site interactions. Models with site-specific variances do provide more powerful tests when variances are different. Never pooling, i.e., using site-specific variances when variances are equal, also reduces power. In contrast to the relatively benign effects of pooling across sites, incorrectly pooling across treatments is much more serious.
AIC-based evaluations of variances are very sensitive to non-normality, with a strong tendency to indicate unequal variances when that is incorrect and the data are non-normal. While Levene’s test is somewhat liberal when errors are skewed or heavy-tailed, it is much more robust than AIC.
I conclude that ignoring site-specific error variances is not wrong, but modeling that heterogeneity will increase power. If there is any possibility that errors are non-normal, I suggest that variance models be evaluated using Levene’s test instead of AIC.