FeedHub’s goal is to provide our users with a personalized feed that delivers only the most relevant posts.  This feed is unique to each of our users, and I use “relevant” here to refer to any post the user would like to see in their personalized feed.  By observing a user’s interactions with their feed items (such as clicking on the thumbs up or thumbs down icons), FeedHub tries to learn a model of which feed item attributes are predictive of the user’s interests.  These attributes may be inferred from the content of the feed item, such as the Wikipedia category based labels we assign to items.  Alternatively, they could be based on other attributes of the feed item, such as the number of comments people have made to that post.

The base assumption is that this comment count is predictive of whether or not the item is interesting to the user.  The more discussion around the item, the more likely it is to be interesting to a user.

However, it is important that we normalize the input to the model used in FeedHub such that the comment counts are useful predictors across all input feeds.  Here are some factors that may be important:

1. Some feed sources have a lot of discussion around every item (e.g. Slashdot).  Other feeds may have many fewer comments per post (e.g. John Langford’s Machine Learning (Theory) blog).  If you’ve subscribed to both of these feeds in FeedHub, that implies you are probably interested in receiving content from both of these feeds.  Naively using the comment count directly as a feature in a learning algorithm is unlikely to work well; it’s hard to believe that the post with the most comments from John Langford’s blog is worse than the post with the fewest comments on Slashdot.  Some per-feed normalization is likely to be important.
2. For a feed which typically receives a small number of comments per item, the difference between 1 comment and 10 comments is likely to reflect a larger difference in “interest” than the difference between 51 and 60 comments.  Careful normalization within a feed may be necessary.

These two concerns led me to hypothesize that a good way to normalize the number of comments an item has received would be to model the probability distribution of comments per item for each feed.  Given an observed number of comments for an item, we can estimate , the probability that another randomly selected item from that same feed will have fewer comments.  I’m not an expert in the combination of evidence, but this approach does mitigate the above concerns.  Using this probability estimate also has a nice intuitive interpretation: a normalized value of 0.9 means that we expect an item to have more comments than 90% of the items from that feed.

The desire for the Bayesian estimation for the distribution described in my previous post arises from concerns that we may not be able to estimate the distribution well from a small number of items in an input feed where posts are relatively infrequent.  Using a Bayesian estimate of allows us to reduce the bumpiness compared to maximum likelihood estimates.

For example, consider coin flips.  If we observe two heads in two flips, the maximum likelihood estimator would suggest that the coin is unfair.  In practice, most people have a prior belief that the coin is unlikely to be unfair.  It would take many more coin flips all coming up heads for us to overcome that prior belief.  A Bayesian estimate formally models a similar estimation process.  When we have few observations of the data, our estimates will be close to those specified by our prior beliefs.  As we get more data, the estimates from the observations become more reliable and we place less emphasis on our prior beliefs.

where and

However, in many cases we have small sample sizes and I felt the natural urge to specify prior distributions over the negative binomial’s parameters. The beta distribution is conjugate for when is known, but for our data is not fixed.

This left me a little stuck.  What I really want is a closed form conjugate prior when both parameters of the negative binomial distribution that is efficient to compute.  What I found was Morgan and Hickman, which requires a large sample to be accurate (kind of defeats the point).  I also found Bradlow, Hardie, and Fader, which has a “closed-form” solution which requires a 300-term expansion to accurately estimate the posterior.   While noticeably faster than Markov chain Monte Carlo methods, it is still more heavyweight than I want.

I felt defeated until I realized that if I accept inference using maximum a posteriori estimates of and as a good enough alternative to full-blown Bayesian inference, things become much simpler.  I need only periodically recompute and and perform inference directly on the negative binomial distribution using those estimates.  I also get an additional benefit from being willing to use maximum a posteriori estimates; I can use expectation-maximization to estimate and .

This task is quite easy when estimating the parameters of a negative binomial.  Given observations estimates and priors over and we estimate new maximum a posteriori estimates ,   Wash, rinse, repeat until convergence.  This iterative procedure means that when estimating , we can assume a constant (and vice versa).

To estimate , write

where

If then

which indicates .  To estimate , we use the posterior mode of :

That was the easy part.  Now for the harder part.

For my purposes, the beta prime distribution is a good fit for .  The beta prime distribution has a similar shape to the more familiar gamma distribution.  If then .  For the beta prime distribution,

Our posterior is distributed

where

Following Bradlow, Hardie, and Fader, we note that the ratio of the two Gamma functions can computed exactly:

where , , and is the number of observations equal to .

While can be computed exactly, it is not easy to integrate analytically.  Although I am unwilling to perform computationally expensive operations regularly, I am very comfortable with using numeric techniques to update the MAP estimates.  So I can rely on black-box estimation functions in R when testing or the Apache Commons Math library for use in our system.  The resulting recipe to estimate for :

In practice, what I actually maximize is the log of the above quantity, which reduces the chance of overflow or underflow during computation.  While is not easily integrable, it is differentiable (as is its derivative).  One could define first and second derivatives and find the maximum value using Newton-Raphson, but the derivatives require recurrence relations to state succinctly (and I couldn’t be bothered).

All that remains is the initial choice of and .  I chose as starting points the method of moments estimator for these parameters (Savani and Zhigljavsky):

where and .  When the method of moments estimates are not well-defined for the sample data, I use the means of the prior distributions as starting points.

Here’s a snapshot of a few feeds.  The red line indicates the method of moments estimator and the blue line shows the maximum a posteriori estimates. Yes, I know that the negative binomial is a discrete distribution and plotting it as a line is misleading, but I wanted to look at a large number of feeds at a time and using lines to plot the density is easier to see.

Update:

Michelle asked a couple of questions about the use of the beta prime distribution as a prior for the parameter of the negative binomial, so I figured I’d update this post with a little more detail about this choice.

When choosing a distribution to model the prior for , I started by looking at the histogram of values estimated using the method of moments estimator described above.  I first considered using the gamma distribution, but it didn’t turn out to be a great fit.  The red line on the histogram below shows the method of moments estimator for the gamma function proposed by Wiens et al (Pak. J. Statist. 2003 Vol.19(1) pp129-141) with .  The blue line shows the much better fit given by the beta prime distribution.

To fit the beta prime parameters, I fit a beta distribution to .  Here’s a plot showing the histogram for and the method of moments estimator for the beta distribution:

I hope this makes my choice of the beta prime distribution and its estimation a little more clear.

To measure a practical difference between systems, we must:

1. have an evaluation measure that correlates with user satisfaction,
2. understand the difference needed under the evaluation measure for a user to notice that one system is preferable to another, and
3. have confidence (statistically) that the difference between the two systems is larger than that minimum noticeable difference.

The third challenge is the most straightforward as we can rely on statistics.  The first two, however, are far more challenging.  It is often the case that traditional system evaluation measures such as mean-average precision do not correlate well with human satisfaction.  Other measures focusing on early precision may correlate better with human experience, but there is much higher variability when focusing only on the top ranked documents.  This means we need a much larger number of queries to measure statistical significance.

There has been a lot of recent research in the Information Retrieval community on evaluation.  Much of this work has focused on statistical techniques and efficiently building test collections.  I think these works are great to have, but I wish there were more working to answer the first two questions.

One limitation of Bonferroni method is its conservativeness.  If you would normally reject the null hypothesis when , when using Bonferroni correction you would only reject when , where is the numer of tests.  This can make it very difficult to reject any of the tests when is large.  This arises because the Bonferroni method controls the probability of falsely rejecting any null hypothesis; it controls the family-wise error rate.

There’s a newer technique that addresses the conservative nature of the Bonferroni method.  Instead of controlling the probability of falsely rejecting any null hypothesis, the idea is to control the false discovery rate.  The false discovery rate is the expected proportion of false rejections of the null hypothesis.  By controlling the false discovery rate, we acknowledge that we are willing to accept that for each rejection of the null hypothesis, we expected that the probability it was rejected in error is or less.

The Benjamini-Hochberg method is a simple approach for controlling the false discovery rate.  Given p-values resulting from a statistical significance test:

1. Let be the p-values sorted in increasing order.
2. Define and where if the p-values are independent and otherwise.
3. Reject all null hypotheses where .

In my CIKM paper from 2006 with Mounia Lalmas, we performed extensive system pairwise comparisons to understand some aspects of evaluation measures in XML element retrieval.  We did look into controlling family-wise error rate through the Bonferroni method, but because we were doing pair-wise tests on roughly 40 different result lists per task, none of the system differences were identified as statistically significant.   Controlling the false discovery rate allowed us to identify differences, despite the large number of comparisons and relatively small sample sizes.