The LexRank paper by Erkan and Radev is another PageRank/Graph Theory based approach to working with text, this time applied to the task of summarization.

Key parts of sections 1 and 2 discuss the general problem of corpus-based summarization.  Unlike TextRank by Mihalcea, Erkan and Radev are interested in summarizing groups of documents (although it can be applied to individual documents as well.)  They propose a “centroid-based” model whereby they try to determine the most important sentences in a group of documents.  In this approach, the sentences that have more words near to the center of the cluster of documents are deemed to be more important, thus giving higher weight.  The authors in section 3 then layout their methods for determining the similarity between two sentences using a TF-IDF formula (think vector space search).  From these similarity scores, a matrix can be constructed tallying all of the similarities between all the sentences.

After doing some fancy linear algebra, they come to define LexRank, which is really just Google’s PageRank applied to this particular problem.  Sections 3.2 do provides all the gory details on the math and how the problem can be described in terms of stochastic matrices and markov chains, which all seems to boil down to the lovely formula that Brin and Page gave us.   This section also lays out in more detail the iterative algorithm for calculating PageRank/TextRank/LexRank/YouFillInTheBlankRank.

Taking this matrix, we put it into a graph and weight the edges according to our similarity and then run the iterative algorithm over it.

I know this sounds like skimping on my part, but the rest of the paper is just about the experiments they ran and I don’t feel like rehashing those.  Guess what?  It did pretty well.  One thing that is interesting is the availability of the MEAD summarization system, available at http://www.summarization.com/.

Like most of the graph based approaches we have seen so far, it does well with noisy data, which is one of the big selling points for me.

Next week, on to something new: question answering!  See you then!

LexRank, Radev, Erkan, PageRank, Brin, Page, Google, linear algebra, graph theory

Technorati Tags: LexRank, Radev, Erkan, PageRank, Brin, Page, Google, linear algebra, graph theory

As is typical of most academic papers, section 1 introduces us to the problem that is being discussed, in this case how to determine the authoritative sites on the web.  Specifically, in the prehistoric web days, search engines would bring you back lots of hits, but seldom did they bring you back the most important hits, at least not high in the rankings (we are relatively spoiled these days in this regard.)  As Brin and Page alluded to, doing a query on “microsoft” on these old engines may not even have microsoft.com as the first result, even though most users would tell you that it should be.  Klienberg asserts that the big thing missing from search engines in these days is the use of link analysis to determine the authoritative sites on the web so that they may be ranked higher.  Unlike closed corpus search engines that suffer from the scarcity problem (not enough good content to search), the web suffers from an abundance problem, namely too much content.  Given this, the goal, Kleinberg says, is to identify the authoritative pages that are relevant to the user’s query.

Much like Brin and Page and Mihalcea (see eariler posts), Kleinberg suggests a graph based approach based on the links in HTML to determine the authoritative pages.  Additionally, he also defines “hub” pages, i.e. those pages that have links to multiple relevant authoritative pages.  Hubs never point to hubs and authoritative pages never point to authoritative pages, which yields a dense directed bipartite subgraph (don’t worry, I had to look it up as well.)  Hubs and authorities reinforce each other, “a good hub is a page that points to many good authorities; a good authority is a page that is pointed to by many good hubs.”  Intuitively this make sense as we all know pages that we trust and we know where to go to find them.  I am not sure if this is correct, but I think to some extent in our post-google world, the hubs are now the search engines themselves.  That is, being ranked highly in the engine, tends to reinforce a site as being the authority.

Unlike the previous papers we looked at, this one actually has a fair amount of math, especially in section 2, so dig out your Linear Algebra book if you really want an in-depth understanding.  The algorithm for calculating the hubs and authorities is very similar to both the PageRank and TextRank algorithims we discussed in prior posts and can be calculated using a similar, simple, iterative algorithm (I implemented it in a few hours). The graph in use for the algorithm consists of a root set of nodes identified by a search engine (Alta Vista in this case) and is then expanded to include pages that are pointed to by a node in the set, or point to a node in the set (possibly bounded by some predetermined amount.)  Edges are the links themselves. Given the graph it comes down to calculating the authority weights and the hub weights using the iterative algorithm.   Authority weights are the sum of the hub weights and the hub weights are the sum of the authority weights.   See pages 9, 10, 11 and 12 for the exact details of proving why this works.  You may want to refresh your knowledge of Eigenvectors, etc.   Seriously, though, don’t you love when “discovered solutions” to problems have such a pure mathematical relationship?  That is, isn’t it great when the math matches up with your intuition?

Section 3 breaks expands on the notion of hubs and authorities to discover communities of links, and, also, “anti” communities.  For instance, using a variant of the main algorithm, one can determine clusters of sites that are related.  For instance, one can easily determine those sites in favor of abortion and those opposed.  Mathematically, this correlates to working with the non-principal eigenvectors, whereas the primary algorithm involves working with the principal eigenvector.

Section 4 continues Section 3 and shows how it is fairly easy to use the link structure to find similar pages given an initial query.  Unlike relevance feedback, Kleinberg’s approach relies solely on the link structure.

Tomorrow or Friday, I will look at the remaining sections of the paper.

graph theory, Kleinberg, PageRank, TextRank, Mihalcea, Brin and Page, eigenvectors, link analysis, bibliometrics, linear algebra, information retrieval, search engines

Technorati Tags: graph theory, Kleinberg, PageRank, TextRank, Mihalcea, Brin and Page, eigenvectors, link analysis, bibliometrics, linear algebra, information retrieval, search engines

I’m not going to post the code just yet, but I may in the future, but here’s the Gist of what I did:

1. Create your graph and relations using JGraphT. I created a Vertex and Edge objects. The Vertex has two Generic properties (Key, Value) and maintains a score value as a double. The Edge has a weight as a double and what I call an EdgeLoad, namely a Generic property that can hold some object that is associated with an edge. While this is overkill for my initial implementation of keyword extraction, I think having the ability to store info on the vertex and edge will come in handy later as I explore different approaches.
2. Create a GraphRanker interface that has a rank method that is overloaded to take in either an Undirected or Directed Graph
3. Implement the GraphRanker interface to rank the Graph. This involved:
   1. while error rate is > than the threshold and we haven’t reached the maximum number of iterations
   2. for each vertex in the Graph
   3. calculate the rank of the vertex
      1. The rank is calculated according to the formula in the paper. Just remember that it doesn’t need to be recursive, you can just use any dummy starting value when calculating the first S(V_j)
   4. Set the error rate (estimated) to be the new rank – the old rank
   5. Repeat from step 1 until convergence is reached or you max out your iterations
   6. Sort the vertices by score and return

Some tips to know you got it: The average rank of an unweighted graph should be approach 1. Start very simple and do the keyword extraction version first. Put only two keywords in your graph. Since they co-occur within 2 positions, they should have equal rank upon convergence since they both contribute evenly to each other.

Give it a try and let me know if you have any questions. This really is quite easy to implement. If you graph gets really large, you may want to look into using Hadoop to distribute.

Also, go out and see what you can apply it to, don’t just limit yourself to text. Let me know what works and doesn’t work for you.

TextRank, Graph Theory, JGraphT, Hadoop, Implementation, Java, PageRank, Google, Mihalcea, Brin and Page

Technorati Tags: TextRank, Graph Theory, JGraphT, Hadoop, Implementation, Java, PageRank, Google, Mihalcea, Brin and Page

With the problem defined, Mihalcea describes the preprocessing steps for building the graph.  Namely, words are tokenized and part of speech is assigned.  These words are then added as nodes to the graph (note, they optionally can filter out certain types of words, such as all adverbs or all nouns) and edges are added between two nodes if the two words co-occur within some variable number of words, usually somewhere between 2 and 10.  Once the graph is setup, the convergence algorithm is run until some approximated error threshold is met and the results are ranked and returned.

Performance wise, the algorithm does quite well, beating the best reported supervised result of the time.  It should be noted that this approach is completely unsupervised, which is a real benefit.  In my mind, if you can get better results without supervision, you have a winner.  The rest of section 3 is a discussion of the TextRank results versus other comparable systems.  See the paper for details.

Next post we will discuss the approach applied to Sentence extraction for automatic summarization, and we will end the week with the remainder of the paper on Friday.

TextRank, PageRank, Google, Mihalcea, Keyword Extraction, Natural Language Processing, NLP, machine learning, algorithms, computer science

Technorati Tags: TextRank, PageRank, Google, Mihalcea, Keyword Extraction, Natural Language Processing, NLP, machine learning, algorithms, computer science

TextRank, NLP, natural language processing, unsupervised learning, PageRank, HITS, computer science, algorithms

Technorati Tags: TextRank, NLP, natural language processing, unsupervised learning, PageRank, HITS, computer science, algorithms

1. PageRank – The formula and approach for ranking web pages.
2. System Anatomy – The discussion of how Brin and Page setup their initial system.

I’ll skip the details on performance, etc. since it should be pretty obvious that it works. It should be noted, though, that the importance of this new approach did wonders for how the web was searched. If you remember prior to Google, most search results were pretty low quality except Yahoo! which relied on a directory structure.
PageRank
Here is the discussion of PageRank from http://infolab.stanford.edu/~backrub/google.html

We assume page A has pages T1…Tn which point to it (i.e., are citations). The parameter d is a damping factor which can be set between 0 and 1. We usually set d to 0.85. There are more details about d in the next section. Also C(A) is defined as the number of links going out of page A. The PageRank of a page A is given as follows:PR(A) = (1-d) + d (PR(T1)/C(T1) + … + PR(Tn)/C(Tn))

Note that the PageRanks form a probability distribution over web pages, so the sum of all web pages’ PageRanks will be one.

PageRank is really just the notion that the more pages that point to a page, the more higher that page should rank. Additionally, this is a cumulative effect, that is if A points at B and B at C, A’s PR is also a factor in calculating C’s PR. The paper doesn’t go into details about the exact algorithm for calculating PR, simply stating that it is a simple iterative algorithm and of course this is now part of Google’s proprietary system, so the world may never know much to the chagrin of many an SEO consultant. In the upcoming papers, we will see references to PR and how it can be used for doing other NLP tasks, so it is good to know the basic idea.

System Anatomy

From an engineering standpoint, the System Anatomy section of the paper is quite interesting especially since it goes into some detail about how to setup a basic Google-like search engine. Of note are the discussion, albeit brief, of the Distributed File System (called BigFiles, section 4.2.1. For an open source version see Hadoop, which has both a DFS and an implementation of Google’s Map/Reduce algorithm.) Moving on, there is some discussion of how the lexicon is stored as well as the documents. These sections strike me as standard indexing techniques.

Following the lexicon subsection, though, is the discussion information on how they store the “Hit Lists”, that is, the list of occurrences of terms in documents, along with payload information about the terms such as font size, capitalization, etc. In comparison, Lucene currently only supports the term occurrence info and not payload information, although there is a submitted patch that allows for indexing payloads. These Hit Lists are then stored in both the forward index and the inverted index.

Section 4.4 concerns indexing the web and dealing with the plethora of errors that occur in malformed HTML as well as how to create the various indexes. Currently, I think there are a number of good solutions that can help solve this problem, given the right amount of hardware (see Nutch for an Open Source version.)

Section 4.5 deals with the issues of searching, namely how to handle single word queries and multi-word queries.  Multi-word queries can be a bit tricky since you want to weight hits with terms that occur closer together in a document higher than those that are a further apart.

The final section (section 5) covers results and performance, which I’m not going to go into because I think it is obvious to anyone who has an online pulse in the last 10 years that the Google approach works.

Next week, we will start looking into some graph based papers that have some basis in the PageRank calculation, but use it in different contexts.

Google, PageRank, Information Retrieval, Graph Theory, Brin, Page

Technorati Tags: Google, PageRank, Information Retrieval, Graph Theory, Brin, Page