You will probably want to measure and report global measures for each testing group  in real time.  You may also have some pre-defined demographic groups of interest you may wish to track.  An obvious one for our related context example is to report logged in users separately from the users not logged in.

However, it is very important that you be able to revisit this data later and explore it in new ways.  For users who are not logged in, it is important that you store the session information and the user’s actions.   For users who are logged in, you will want this information in addition to your system’s internal user identifier.  With this information, you can later do analysis on your logs to separate user groups by more subtle demographics for logged in users, such as occupation, interests, and historical levels of activity.

The value of this post hoc analysis cannot be understated.  It is not enough to simply choose the better of two method; an understanding of when one method outperforms another can help motivate the next method to test.  A thorough understanding of the strengths and weaknesses of an approach is essential for the motivation of the next round of experiments.  Without it, you rely solely on your own intuition and ability to guess new methods.  I don’t know about you, but I’d place more bets on data than luck.

When examining your data, it can be helpful to look at the same numbers in different ways.  For example, consider hourly traffic to Wikipedia for the week of March 01 to March 07 2010.  I downloaded the data for these plots from http://dammit.lt/wikistats/.  The following plots show traffic to the English and German language Wikipedia sites, normalized by dividing by the sum of all page views to the language’s site.  These plots will give us a sense of when users visit the two different sites, but will not be useful for comparing relative traffic from one language to another.  While these data are not the results of a split test, I think they are helpful for illustrating the point that different representations of data can lead to different insights.

This first plot shows a simple line plot with time on the horizontal axis and the percent of weekly traffic on the vertical axis.  This is a very traditional representation for this type of data.  Some things immediately jump out.  The usage of the German language Wikipedia varies much more widely with time than the English Wikipedia, although we do see some time-of-day effects on English too.  We also see lower traffic levels on both Wikipedia sites on the weekend (March 06 and March 07).  In this plot it is quite easy to read the values of peaks and valleys, but it is harder to isolate trends across days.

Another way to present this data is to plot the data with day on the horizontal axis and hour of day on the vertical axis.  The width between two lines for a day varies proportionally to the percent of traffic observed in that day’s hour.  This presentation of the data allows us to see effects that weren’t obvious in the other plot.  For example, we can see that usage on the German language website increased between 7 and 10 am on Saturday and Sunday, while on a typical weekday, this behavior was observed early in the day, between 6 and 9.  Traffic to the German website peaked later in the day on the weekends than weekdays.

To conclude, measuring performance is perhaps the most important aspect of split testing.  In addition to measuring key performance measures such as click-through rates and conversion rates, it is wise to track the performance of new components to help in the estimation of cost.   It is important to store enough data to later explore the results of a test in new ways, digging deeper to gain insights, such as which demographics were well served by the test and for which users the tested approach did poorly.  During this investigation, it can be helpful to look at your data in multiple ways, because different  presentations of the data my facilitate different insights.  These insights can be used to help you form a better mental model of your users and techniques and motivate the next set of hypotheses you wish to explore.

Software developers are well educated about the value of modular design for reuse and testing of correctness.  This can also extend to testing value of components.  By defining a clean interface for a component, it becomes possible to easily substitute other implementations or algorithms.

When we consider our related content widget example from the last post, we may have a new snippet generation algorithm we wish to compare to our existing algorithm.  By deploying a service implementing a standard snippet generation interface, a test framework could direct a portion of all requests to the new algorithm, recording this new component’s impact on click-through-rate.

Complementary to modularity is the need to handle data flow between components.  Without the ability to handle data flow between components in a generic web architecture, to developer is left reimplementing these interactions routinely.  With data flow management, the web architecture can intercept the output of components, transform results, monitor component throughput, and manage split tests.

In some ways the importance of creating modular components and the need to handle data flow between these components a generic web architecture for testing is similar to an enterprise service bus.  An enterprise service bus is middleware which handles messaging between components of complex information architectures.  The bus handles data flow between components, frequently legacy or written in a variety of programming languages.  The routing, message handling, monitoring, and management support provided by many enterprise service buses may be attractive for the design and development a web architecture for testing.  On the other hand, the multi-system support and extra bells and whistles which may not be necessary for your web architecture could create extra overhead and computing resources not desirable in your context.  Most enterprise service buses use XML as the communication language, which as a hefty representation of data, the processing and transformation messages could greatly add to the latency of your website.  I personally have not evaluated any enterprise serial bus for use with web systems, so it is possible that my fears are misplaced.

To conclude, modularity is an important component of any software system, and its use in a web architecture can facilitate web testing.  Coupled with modularity is the need for the test-oriented web architecture to handle data flow in a lightweight way.  Enterprise serial buses provide some of this functionality, but may have more overhead than is desirable for use in a low-latency web system, where every millisecond counts.  Any existing middleware solution for handling data flow would need to be existed to provide additional support for testing.

In order for a culture of experimentation to exist, it must be easy to perform these experiments and measure the results.  There are some great frameworks for offline experimentation.  MapReduce solutions are great for experimentation and examination of large data sets.  I’ve seen the importance of a good software architecture for experimentation in my own experience with the Lemur Toolkit.  However, I haven’t seen much discussion of frameworks for performing online experiments in complex web systems.

This post considers as an example split testing for a related content widget.  Such a related content widget could be a deployed on a news publisher’s article pages.  The widget would show related articles in a sidebar or below the content of the article.  We may wish to test variations of the related content widget to see if they increase some measure of success, such as the click-through rate.

Conceptually, a related content widget is quite simple.  There are two major components, one that returns a list of related items and one that renders the items on the web page.  While this system is quite simple conceptually, a typical implementation of such a system may be much more complex.  The renderer in the client may make use of a mix of XML, CSS, and HTML.  Parts of the rendering algorithm may also live server-side.  For example, if the user visited the web page from a search engine result page, the keywords could be used to create context sensitive snippets for the related content items.  A snippet generation algorithm would more likely be run on the server than on the client.

The experiments we may wish to perform on the related-content could depend on any of the components used in the creation of the widget.  For example, here is just a small sample of things we may wish to test for the impact on the click-through rate:

• the color of the recommended item titles (CSS changes),
• the number of items presented (parameters passed to the related content service), and
• keyword highlighting, use of keywords in ranking content selection, and custom snippet generation when the user arrives from search engine landing page (the ranking algorithm used by the related content service, the snippet generation algorithm, and possibly CSS changes).

To me, this means that a test framework will interface with many of a web system’s components.  Here are some additional thoughts about the attributes I’d like a test management framework have.  It should

1. handle data flow between components,
2. track metrics such as click-through-rates and response time of components,
3. handle multiple parallel tests along with component dependencies,
4. be able to handle various user types (such as split by session, long-term cookie, or logged-in user),
5. be easy to register new tests,
6. handle deployment of code and files to servers,
7. verify to some degree the integrity of new code before returning results of the code to user requests,
8. be cloud-aware, and
9. be able to do all of this without site downtime.

I should acknowledge that there are some tools out there for managing split tests. However, I believe (possibly in error) that they only scratch at the surface of the requirements I’ve listed above.  Also, it may be unfair for me to talk of this as test framework, because it’s really more of a web framework which has adequate support for testing.

I know I’m asking for quite a lot, but I believe these attributes are important for the creation of a culture of experimentation.  Over the next few weeks I hope to go into more detail about why these attributes are important and share some initial thoughts on how a framework may support these goals.  Since much of these ideas are still formative, I’d love to hear your own thoughts as well.

1. Daniel argues that in order for there to be “search,” there must be an information need.  I use Twitter frequently to gauge sentiment and quantity of discussions around products.  Twitter search can greatly aid my decision making process of whether or not to use a product (such as an open source tool).
2. It is arguable that the recency of a tweet is a very large component of relevance.  Twitter is a way of discussing and interacting with others about what is happening right now.  A reverse chronological ordering of tweets makes sense.
3. At 140 characters, traditional text ranking algorithms might not perform well.  There’s very little context within a tweet to determine how well such a short tweet matches a short query.  Ordering by well tuned, state-of-the-art text retrieval approaches such as Okapi-BM25 may give bad orderings on tweets.
4. At 140 characters, the entire tweet can be comfortably displayed in the result list.  There is no need for snippet generation, and query term highlighting facilitates quick scanning of the results.  Other aspects of relevance other than timeliness may be easier to gauge than in other search tasks.
5. Given the emphasis on timeliness, the search engine must be indexing tweets in real time and making new tweets available very quickly.  While this is no doubt possible with in memory inverted indexes and many people have built this functionality, textbooks rarely address issues of searching and indexing documents in real time.  The real-time demands on Twitter search are greater than other web search engines.

Now that I’m done defending Twitter search, I agree there are things they could do better.  Daniel alludes in comments that one dimension of relevance that is currently not reflected in search results is the influence of the Twitter user on relevance.  There are search tasks where who is participating in the discussion is just as important as when the statement was made.

Also, I mentioned that I use Twitter to measure activity and sentiment.  Twitter search does little to summarize or aggregate results.  External tools such as twist.flaptor.com have done a better job of using tweets to measure and show volume of discussion.  Measuring sentiment from 140 characters is difficult, but it may be possible to measure general reaction as an aggregate.

Close reading is a technique used in literary criticism which focuses on the careful analysis of the language of a text.  These analyses are typically done on passages or short texts and critique low level features of the text, such as word choice, syntax, and the structure and flow of the ideas expressed in the text.  Performing a close reading often begins with a careful annotation of the text, making notes on observations of structure, tone, rhetorical devices, or anything about the text the reader notices.  During this process, the reader will also look up any unfamiliar words, make note of word choice, such as the use of slang, and so on.

Interestingly, we aren’t the only people to consider doing close reading online.  The Golden Notebook Project launched about a month ago.  Seven authors and critics are collaborating to do a close reading of Doris Lessing’s The Golden Notebook.  Their goals our different than ours; this is an interesting case study on one novel.

Our goals are more focused on making it easy for instructors to create close readings in educational environments.  In order to ease adoption, we’d like to seed the tool with publicly available texts, such as those found in Project Gutenberg.  We also want to allow instructors to easily create discussion groups for students, where all students’ annotations could be shared for collaborative reading or private for individual assignments.  Finally, we’d like this tool to allow the instructor to import annotations they’ve previously created for the text into a group, so that instructors can easily reuse their own annotations from semester to semester.  I’m personally very excited about this side project, because I think it is a great example of how online tools can be easier to use than traditional tools.

Currently, Summize’s showcase product is a Twitter search application.  Their search application is nice because you can get real-time updates of matches as the tweets are happening.  Twitter’s recognized the value of Summize’s conversational search, and linked to Summize for live coverage of WWDC.  I think it’s great when small companies collaborate in these ways.  It’s also a smart move for Twitter today, given the troubles they’ve been having recently.  Diverting some of their traffic to Summize might help them handle today’s load.

Looking at the results of Twitter’s suggested search, I feel a little overwhelmed.  Just a few seconds after clicking on the link, Summize’s search results informed me that there have been an additional 26 posts matching the search since the page was loaded.  Letting the page go for a minute without refreshing showed hundreds of new tweets.  That’s not terribly surprising, given the excitement around WWDC.  But here’s the rub.  Most of these tweets are useless to me or contain redundant information.  I want only the unique information, not the chatter.  For high volume topics of discussion, I really need the conversational search to filter or summarize the results for me.  Abdur and Eric, please bring some of the great summarization and organizational aspects of your product search to your conversational search.