Multiple significance testing

Panos Ipeirotis recently posted some interesting thoughts about statistical significance tests for comparing systems when incrementally developing techniques to solve a problem.  While I don’t have the answer to his question, I did notice that he made note of the Bonferroni method for correcting for mutiple testing.  The need for multiple test correction arises when you need to make more than one comparison; the more comparisons you make, the more likely an uncorrected test will be rejected by chance.

One limitation of Bonferroni method is its conservativeness.  If you would normally reject the null hypothesis when p < \alpha, when using Bonferroni correction you would only reject when p < \alpha/m, where m is the numer of tests.  This can make it very difficult to reject any of the tests when m is large.  This arises because the Bonferroni method controls the probability of falsely rejecting any null hypothesis; it controls the family-wise error rate.

There’s a newer technique that addresses the conservative nature of the Bonferroni method.  Instead of controlling the probability of falsely rejecting any null hypothesis, the idea is to control the false discovery rate.  The false discovery rate is the expected proportion of false rejections of the null hypothesis.  By controlling the false discovery rate, we acknowledge that we are willing to accept that for each rejection of the null hypothesis, we expected that the probability it was rejected in error is \alpha or less.

The Benjamini-Hochberg method is a simple approach for controlling the false discovery rate.  Given P_1, P_2, \dots P_m p-values resulting from a statistical significance test:

  1. Let P_{(1)} \leq P_{(2)} \leq \dots \leq P_{(m)} be the p-values sorted in increasing order.
  2. Define l_i = \frac{i \alpha}{C_m m} and k = \max\{i : P_{(i)} < l_i\} where C_m = 1 if the p-values are independent and C_m = \sum_{i=1}^{m} 1/i otherwise.
  3. Reject all null hypotheses where P_i \leq P_{(k)}.

In my CIKM paper from 2006 with Mounia Lalmas, we performed extensive system pairwise comparisons to understand some aspects of evaluation measures in XML element retrieval.  We did look into controlling family-wise error rate through the Bonferroni method, but because we were doing pair-wise tests on roughly 40 different result lists per task, none of the system differences were identified as statistically significant.   Controlling the false discovery rate allowed us to identify differences, despite the large number of comparisons and relatively small sample sizes.

Post a comment | Trackback URI