Imagine that you are responsible for monitoring all the major news feeds in every single country your company does business in order to advise the CEO on trends and events that effect your company.  Obviously, you can’t read all of them on a daily basis and having a large staff to help would be costly.  This is exactly the kind of scenario that online event detection is meant to solve.  You need a program that can identify and organize news feeds as they come in, allowing you to see the key events (or even minor events) as they are reported, not days later.

This paper discusses the approach of a group of researchers at Carnegie Mellon University in the field of Topic Detection and Tracking.  After providing some background  information on the topic, the authors dive into the details of their approach.  The task at hand was to analyze a corpus of documents in a temporal fashion to identify events and track them.  In other words, even though they had the whole corpus at the time, they had to pretend like they were receiving the news in chronological order just like you and I do on a daily basis.  They could not look “into the future”, if you will, in doing their analysis.

Yang’s group attempts to solve this problem by using a clustering approach.  In fact, they are modifying Cutting’s Scatter/Gather approach that we discussed here and also a single-pass, incremental approach.  They approach the problem in two ways.  First, they use the scatter/gather approach on a “retrospective” collection containing articles that occurred in the “simulated” past.  This is done to build up statistics about past events, figuring that new events will contain similar structures and statistics (TF/IDF, etc.) albeit with variations due to new names and events.  Section 3.1 discusses the representation of the clusters and 3.2 discusses their modifications of Cutting’s Scatter/Gather approach into what they call Group Average Clustering (GAC).

To solve the real-time, online problem, an incremental, single-pass approach was used.  To do this kind of thing, one needs to somehow estimate corpus statistics like IDF (inverse document frequency) in order to come up with reasonable estimations in order to come up with the proper weights for the term vectors used in the clustering algorithm.  The CMU group solves this problem by originally estimating the IDF using the stats from the retrospective process and then updating it as new information becomes available in the real-time approach.

Much of the rest of the paper is about picking parameters and evaluation.  Section 4.3 has some interesting discussion of “Behavior Analysis” that is worth looking into.  The gist of it being that the GAC approach seemed to be good at identifying large news bursts, while the incremental approach is better at tracking at long-lasting events.  In our scenario, you will most likely be interested in both kinds of events.  The key, of course, is having the ability to zoom in/out on the various news feeds and to setup alerts, etc. that help you manage the clusters.

Diving into section 2, the basic premise of the approach is that documents are “scattered” into small groups of documents and summaries are shown to the user.  Based on the groups the user looks at, they are then gathered back in and then this subcollection is done again.  Section 2.1 has a nice, easily understood example that is well worth the read.  Section 2.2 lays out the requirements needed to implement Scatter/Gather, which are:

1. A good clustering algorithm that can cluster a lot of documents in a a very reasonable time.
2. Summarization capabilities for describing the clusters.

Section 3, title Document Clustering, lays the groundwork for the rest of the paper by outlining much of the current terminology and strategies for document clustering.  It will be interesting, in coming weeks, to compare how far we have come in clustering since 1992.

To do clustering, one needs a notion of document similarity.  A common one is the cosine similarity function that is used in the Vector Space Model of IR as well.  Essentially, two documents can be represented as vectors in some vector space, and the cosine of the angle between the two vectors can be used as a measure of similarity.  See Section 4 for more information on this, as well as other definitions useful to the problem at hand.  I’m waving my hands here for the moment, as I want to see how the notion of the document profiles introduced here play out in later sections.

Section 5 takes us into the needs of a partitional clustering algorithm, which are: 1) find a set of centers, 2) assign documents to a center and 3) refine the partition.  Once complete, there is a set of disjoint document groups that cover all the documents in the corpus.  The two clustering algorithms introduced in this paper, Buckshot and Fractionation, are useful for finding the centers, Buckshot being the faster of the two, while Fractionation is more accurate.  Section 5.1 looks at how Buckshot and Fractionation find the initial centers.  Sections 5.2 then covers how documents are assigned to the center, the simplest approach being “Assign-to-Nearest” which assigns each document to the center that maximizes the similarity between the document and the center.  Section 5.3 then looks at the refinement process, which can be broken out into 3 parts, an iteration  of “Assign-to-Nearest”, the Split algorithm which divides clusters that are not well defined and the join algorithm which merges clusters that are similar.

Section 6 goes into a good amount of detail on how the work of Section 5 can be incorporated into the Scatter/Gather algorithm and how it works out in practice.  The rest of the paper makes conclusions about the work and why it is useful.  The Appendix A covers an in-depth session of the approach.

Intuitively, the approach and application makes sense, even though my knowledge of clustering leaves some me lacking a bit of understanding on the low level details.  I wonder if there are any current systems that make use of similar browsing approaches in real systems.

To start, the paper explains the background behind clustering, highlighting two important features that clustering engines need: 1.) Good clusters! 2) Human readable labels.  Section 3 elaborates by noting that most approaches try to assign labels after the cluster, while their approach is different.  They only assign documents to a good label.

To achieve this, they extract frequent phrases from the input, to use as good labels.  Then use Singular Value Decomposition (SVD) on the term-document matrix.  At last, they match up labels with the clusters.  Section 3.1 has pseudocode for the algorithm.  In preparation for clustering, they do stopword removal and stemming.  Interestingly, they propose using the stopwords as language indicators so they know what stemmer to use.  This is clever in that it should be quite fast in comparison to using n-grams.  I wonder how accurate this approach is compared with n-grams.

Section 3.2 delves into how frequent phrases are extracted.  This is important because they form the candidate labels to assign to a cluster. Section 3.3 then discusses how cluster labels are induced from the phrases.  It is a three step process: 1) build the term-document matrix, 2) discover abstract concepts, 3) match phrases and prune labels.  The matrix is built using standard TF-IDF calculations, with titles receiving a constant scaling factor to weight them higher.  SVD is then used to find the orthogonal basis of the matrix.  I am not totally familiar with this technique (dang,  it has been a long time since linear algebra class) but, I guess the gist of it is that the basis represents the abstract concepts of the input documents.  Finally, to match phrases and prune labels, they do a cosine calculation between the abstract concepts and the terms, which allows for the selection of the best label based on the cosine score.  Labels are then pruned by finding overlapping descriptions.

Next, documents are added to clusters using the classic Vector Space Model (VSM), assigning documents to a label if it exceeds the “Snippet Assignment Threshold”, a control parameter used in the algorithm.  They do not specify how the threshold is determined.

Section 4 is an example of the process in action and 5 and 6 cover evaluation and future work.  The evaluation section mostly refers to another paper for more in-depth results.  Having used Carrot, I am quite impressed with the results, but I didn’t do a formal evaluation.  The clusters are good and the library is quite fast.