The LexRank paper by Erkan and Radev is another PageRank/Graph Theory based approach to working with text, this time applied to the task of summarization.

Key parts of sections 1 and 2 discuss the general problem of corpus-based summarization.  Unlike TextRank by Mihalcea, Erkan and Radev are interested in summarizing groups of documents (although it can be applied to individual documents as well.)  They propose a “centroid-based” model whereby they try to determine the most important sentences in a group of documents.  In this approach, the sentences that have more words near to the center of the cluster of documents are deemed to be more important, thus giving higher weight.  The authors in section 3 then layout their methods for determining the similarity between two sentences using a TF-IDF formula (think vector space search).  From these similarity scores, a matrix can be constructed tallying all of the similarities between all the sentences.

After doing some fancy linear algebra, they come to define LexRank, which is really just Google’s PageRank applied to this particular problem.  Sections 3.2 do provides all the gory details on the math and how the problem can be described in terms of stochastic matrices and markov chains, which all seems to boil down to the lovely formula that Brin and Page gave us.   This section also lays out in more detail the iterative algorithm for calculating PageRank/TextRank/LexRank/YouFillInTheBlankRank.

Taking this matrix, we put it into a graph and weight the edges according to our similarity and then run the iterative algorithm over it.

I know this sounds like skimping on my part, but the rest of the paper is just about the experiments they ran and I don’t feel like rehashing those.  Guess what?  It did pretty well.  One thing that is interesting is the availability of the MEAD summarization system, available at http://www.summarization.com/.

Like most of the graph based approaches we have seen so far, it does well with noisy data, which is one of the big selling points for me.

Next week, on to something new: question answering!  See you then!

LexRank, Radev, Erkan, PageRank, Brin, Page, Google, linear algebra, graph theory

Technorati Tags: LexRank, Radev, Erkan, PageRank, Brin, Page, Google, linear algebra, graph theory

LexRank: Graph-based Lexical Centrality as Salience in Text Summarization

Enjoy!

Discussion of Sections 5 through 8

The remaining sections of this paper are discussions of two applications of the algorithms plus the body of related work and conclusions that can be drawn from the work. Section 5 gives the details on what corpora were used (Enron, plus an internal email set not publicly available) before proceeding to the task of name disambiguation.

Name disambiguation in email is the task of correlating the mention of a name in an email with the actual person. While this is fairly straightforward in most cases for people reading their own email, it becomes difficult when reading other’s email since one may not know all the people the other person does. Multiply this by a collection of emails filled with nicknames and initials and it is not hard to see why this is difficult. The task is useful for establishing social networks as well as other applications. One could easily imagine an automated system that retrieved relevant information about a person mentioned in an email (bio, address, phone, past conversations, etc.) and made it available to the reader for instant access.

The remaining parts of section 5 go into the details of applying the algorithm and the results that are achieved.   The interesting thing in my mind is that the graph often connects different types of nodes and associates different probabilities with the transitions from one node to the other.  Applying the graph walking strategy then leads to the desired results.  Suffice it to say, the new approach performs better than the baseline approach!

Identifying threads of emails is the second application the authors use to demonstrate their capabilities.  Threading is the problem of identifying one or more messages that are related to some chosen email.  Many email systems do a basic job at this by comparing subject lines, esp. those that use the “RE:” prefix.  However, we all know people often treat the subject line differently.  Furthermore, people tend to quote the previous messages in the thread differently.  Some use “>”, while others use “|”, while still others use nothing at all.  Add in inline replies, which is especially common on mailing lists, and you see why the problem becomes difficult.  Section 5.4 lays out the graph walk approach and compares it to TF-IDF IR approach, which, of course it does better, especially when using a machine learning re-ranking approach.

The rest of the paper is on related work and conclusions.  I am glad the authors address the performance in terms of scalability in the conclusions section, as I had my doubts about how well the approach could perform on large amounts of data.  In fact, I find a lot of papers in the NLP realm fail to account for performance, so it is refreshing to see it addressed.

graph theory, email, natural language processing, NLP, information retrieval, IR, threading, name disambiguation

Technorati Tags: graph theory, email, natural language processing, NLP, information retrieval, IR, threading, name disambiguation

email, graph theory, disambiguation

Technorati Tags: email, graph theory, disambiguation

Section 5 examines the quality of the authority measure by comparing the results achieved by running the Kleinberg algorithm against some of the searchable hierarchies that exist on the web (Yahoo, others that are long gone) and shows that this approach does pretty well at the task.

Section 6 then covers one of the problems with the approach, which the authors label as diffusion.  Diffusion happens when the algorithm produces results that are not on the original topic.  In most cases, the results are more generalized given a specific query.  Kleinberg’s example is the query “medical conferences” which yields results that are mainly about medicine.  A proposed solution is to reintroduce the query back into the problem after the algorithm has run by using the terms of the query to rerank the results in the result set.  Section 6 gives a fair amount of detail on how this reranking can take place.

Well, that wraps up another week.  I hope people are finding this useful.  I know I am (even if I cheat a little bit like I did tonight and gloss over the details a little more.)  If anyone is interested in discussing a paper on a particular topic, please feel free to leave a comment suggesting one.  For now, I am going to do one or two more weeks on graph based approaches to NLP and then I’m going to start looking at some Question Answering papers.

NLP, Graph Theory, Kleinberg, diffusion, information retrieval, IR

Technorati Tags: NLP, Graph Theory, Kleinberg, diffusion, information retrieval, IR

Computing Eigenvectors and Eigenvalues

linear algebra, eigenvalues, eigenvectors

Technorati Tags: linear algebra, eigenvalues, eigenvectors

As is typical of most academic papers, section 1 introduces us to the problem that is being discussed, in this case how to determine the authoritative sites on the web.  Specifically, in the prehistoric web days, search engines would bring you back lots of hits, but seldom did they bring you back the most important hits, at least not high in the rankings (we are relatively spoiled these days in this regard.)  As Brin and Page alluded to, doing a query on “microsoft” on these old engines may not even have microsoft.com as the first result, even though most users would tell you that it should be.  Klienberg asserts that the big thing missing from search engines in these days is the use of link analysis to determine the authoritative sites on the web so that they may be ranked higher.  Unlike closed corpus search engines that suffer from the scarcity problem (not enough good content to search), the web suffers from an abundance problem, namely too much content.  Given this, the goal, Kleinberg says, is to identify the authoritative pages that are relevant to the user’s query.

Much like Brin and Page and Mihalcea (see eariler posts), Kleinberg suggests a graph based approach based on the links in HTML to determine the authoritative pages.  Additionally, he also defines “hub” pages, i.e. those pages that have links to multiple relevant authoritative pages.  Hubs never point to hubs and authoritative pages never point to authoritative pages, which yields a dense directed bipartite subgraph (don’t worry, I had to look it up as well.)  Hubs and authorities reinforce each other, “a good hub is a page that points to many good authorities; a good authority is a page that is pointed to by many good hubs.”  Intuitively this make sense as we all know pages that we trust and we know where to go to find them.  I am not sure if this is correct, but I think to some extent in our post-google world, the hubs are now the search engines themselves.  That is, being ranked highly in the engine, tends to reinforce a site as being the authority.

Unlike the previous papers we looked at, this one actually has a fair amount of math, especially in section 2, so dig out your Linear Algebra book if you really want an in-depth understanding.  The algorithm for calculating the hubs and authorities is very similar to both the PageRank and TextRank algorithims we discussed in prior posts and can be calculated using a similar, simple, iterative algorithm (I implemented it in a few hours). The graph in use for the algorithm consists of a root set of nodes identified by a search engine (Alta Vista in this case) and is then expanded to include pages that are pointed to by a node in the set, or point to a node in the set (possibly bounded by some predetermined amount.)  Edges are the links themselves. Given the graph it comes down to calculating the authority weights and the hub weights using the iterative algorithm.   Authority weights are the sum of the hub weights and the hub weights are the sum of the authority weights.   See pages 9, 10, 11 and 12 for the exact details of proving why this works.  You may want to refresh your knowledge of Eigenvectors, etc.   Seriously, though, don’t you love when “discovered solutions” to problems have such a pure mathematical relationship?  That is, isn’t it great when the math matches up with your intuition?

Section 3 breaks expands on the notion of hubs and authorities to discover communities of links, and, also, “anti” communities.  For instance, using a variant of the main algorithm, one can determine clusters of sites that are related.  For instance, one can easily determine those sites in favor of abortion and those opposed.  Mathematically, this correlates to working with the non-principal eigenvectors, whereas the primary algorithm involves working with the principal eigenvector.

Section 4 continues Section 3 and shows how it is fairly easy to use the link structure to find similar pages given an initial query.  Unlike relevance feedback, Kleinberg’s approach relies solely on the link structure.

Tomorrow or Friday, I will look at the remaining sections of the paper.

graph theory, Kleinberg, PageRank, TextRank, Mihalcea, Brin and Page, eigenvectors, link analysis, bibliometrics, linear algebra, information retrieval, search engines

Technorati Tags: graph theory, Kleinberg, PageRank, TextRank, Mihalcea, Brin and Page, eigenvectors, link analysis, bibliometrics, linear algebra, information retrieval, search engines

PageRank, HITS, Joe Kliengberg, Paper of the Week, Cornell, graph theory, ranking

Technorati Tags: PageRank, HITS, Joe Kliengberg, Paper of the Week, Cornell, graph theory, ranking

I’m not going to post the code just yet, but I may in the future, but here’s the Gist of what I did:

1. Create your graph and relations using JGraphT. I created a Vertex and Edge objects. The Vertex has two Generic properties (Key, Value) and maintains a score value as a double. The Edge has a weight as a double and what I call an EdgeLoad, namely a Generic property that can hold some object that is associated with an edge. While this is overkill for my initial implementation of keyword extraction, I think having the ability to store info on the vertex and edge will come in handy later as I explore different approaches.
2. Create a GraphRanker interface that has a rank method that is overloaded to take in either an Undirected or Directed Graph
3. Implement the GraphRanker interface to rank the Graph. This involved:
   1. while error rate is > than the threshold and we haven’t reached the maximum number of iterations
   2. for each vertex in the Graph
   3. calculate the rank of the vertex
      1. The rank is calculated according to the formula in the paper. Just remember that it doesn’t need to be recursive, you can just use any dummy starting value when calculating the first S(V_j)
   4. Set the error rate (estimated) to be the new rank – the old rank
   5. Repeat from step 1 until convergence is reached or you max out your iterations
   6. Sort the vertices by score and return

Some tips to know you got it: The average rank of an unweighted graph should be approach 1. Start very simple and do the keyword extraction version first. Put only two keywords in your graph. Since they co-occur within 2 positions, they should have equal rank upon convergence since they both contribute evenly to each other.

Give it a try and let me know if you have any questions. This really is quite easy to implement. If you graph gets really large, you may want to look into using Hadoop to distribute.

Also, go out and see what you can apply it to, don’t just limit yourself to text. Let me know what works and doesn’t work for you.

TextRank, Graph Theory, JGraphT, Hadoop, Implementation, Java, PageRank, Google, Mihalcea, Brin and Page

Technorati Tags: TextRank, Graph Theory, JGraphT, Hadoop, Implementation, Java, PageRank, Google, Mihalcea, Brin and Page

Section 5 has a discussion of why TextRank works from a conceptual point of view.  The idea is that nodes end up recommending other nodes based on the strength of the connections between them, much how people build up structural understanding of concepts.

The biggest strength of TextRank is that it is almost completely portable and requires no knowledge of the domain other than how you want to model your nodes and relations.  No training data is needed.

TextRank, keyword extraction, sentence extraction, document summarization, text summarization, PageRank, Mihalcea, Tarau, Paper of the Week

Technorati Tags: TextRank, keyword extraction, sentence extraction, document summarization, text summarization, PageRank, Mihalcea, Tarau, Paper of the Week