Section 1 outlines the problem of training, namely obtaining a good
sample of text to be labeled for the trainer.  After disposing of
several other methods of garnering samples (random, relevance
feedback based), Lewis and Gale introduce an iterative approach for
manually labeling examples.

Section 2 then discusses the benefits of “learning by query” in
theory, namely the possibility of reducing the error rate very
quickly in comparison to the number of queries required.

Figure 1 (described in section 3) outlines their basic approach,
which relies on having a human judge some subset of examples that the
currently used classifier is least certain about.  This process is
iterated until the human feels satisfied with the results.  One
caveat of this approach is that the classifier must not only predict
the class, it must give a measurement of certainty for that class.

Continuing on into section 4, we are introduced to how to build a
classifier and use uncertainty sampling to train it.  Most of the
section details the probability theory behind it, finishing up with
how to do the sampling.  One thing I always wish for in these papers
are concrete examples (maybe as an appendix or a reference) that work
through the math on an actual toy problem.  Section 5 does just this,
laying out an experiment and discussing the details, minus the math,
which probably suits most people just fine.

Section 7 has an excellent discussion of the results, the pay dirt
being that using this new method significantly reduces the number of
examples required for training, at the cost of having a human in the
loop.

Text classification is the process of assigning a document to one or more categories (we looked at classification/categorization earlier when exploring Support Vector Machines, SVMs).  My understanding of the difference between categorization and classification is that categorization has a set number of categories, whereas classification  does not.  At any rate, this paper is  comparing two different classifiers that  use  a naive Bayes approach.  The naive Bayes approach assumes that all attributes of the examples we are studying are independent of each other.  Even though this is rarely true in the real world for text (after all, if I chose all the words of this post independently we probably would have gibberish) it turns out that it still works pretty well in practice.  But I digress… The two approaches that are being compared are the Bernoulli model and the multinomial model.  The Bernoulli model uses the document as an event and builds a vector of binary attributes based on whether a term occurs or not in the document.  It DOES NOT take into account the number of times the word occurs.  In the multinomial approach, words are the event and term frequency does matter.  The next couple of sections after the introduction layout the common ground between the two approaches as well as the differences.  The differences come down to how the probabilities are calculated.

There is some interesting discussion of feature selection (a way of reducing the size of the vocabulary, which speeds things up, without, hopefully, losing too much information) using mutual information that is worth digging into a bit more if you have the time.

The next sections are where the rubber meets the road and the authors do a side by side comparison of the two approaches using 5 different collections.  You can see the results on pages 5 and 6.  Finally the discussion of the results occurs on page 6 and 7, with the bottom line seeming to be that the multinomial model seems to “be almost uniformly better than the multi-variate Bernoulli model.”

For those interested, Weka has tools for building Naive Bayes classifiers.

After covering some related work about Latent Semantic Indexing (I suppose I should look into that some day), the authors delve into the methodology of identifying phrases and their histories. There are 3 steps to the process: 1) identify frequent phrases, 2) generating histories for the phrases and 3) identifying the phrases for a given trend.

Phrases in this paper go beyond the simple sequence of terms, introducing the notion of a “k-phrase”.   A k-phrase is essentially a nesting of phrases and they can span sentences, etc. when appropriate.  For the histories, each word gets a transaction id and associated timestamps.  Then, given these bits of informations, the authors use a shape query language to mine the phrases and histories.  The shape query language allows the user to specify they are interested when items are “spiking” or “trending downward”, etc.  There is a reference for the shape language in the paper.

Finally, the paper ends with a discussion of how IBM used the approach in a patent mining system to  identify trends in patents from the US Patent office.

Next week, I think I am going to start looking into things like event detection, etc.  However, I am also considering looking into some non-NLP areas related to data mining, so if you have a preference, let me know.  I am open to explore many new ideas in the field.

Otherwise, I will be taking a brief hiatus from reviewing papers until I return from ApacheCon Europe in the early part of May where I am giving a talk and a training on Lucene.  I have several key deadlines over the next two weeks that must take higher priority, including the publication of a couple of articles that I have been working on.  I will post details on the publication on my Lucene blog when they are available.

The system is divvied up into several components, as displayed in Figure 1 of the document.  They are the question processing unit, document processing, factoid answer processing, list answer processing and definition answer processing.  All documents are processed in the same way and passages are retrieved based on the keywords in question.  Depending on the type of question, some passages are removed if they do not have the right answer type.  Passages having a higher number of expected answer types are favored for the list based approach.

For factoid questions, LCC used their CICERO LITE system to provide extractions and answers were calculated based on the extractions and/or the expected answer types.    The extraction process had to identify a variety of semantic classes, such as quantity, date, people, city, etc.  The paper then discusses the types of questions that were answered by these approaches, as well as some special case scenarios related to manner of death (kind of morbid, but death is often a fascination of this kind of research, in my experience).  From here, there is a discussion of the role of theorem proving in the algorithm (see page 5), but the details are left to another paper (guess what will be the paper next week?)  I must admit, I don’t fully understand page 5 and 6 just yet, mostly, I think, because I’m not familiar with the syntax they are using, so maybe reading the next paper will make it more clear.

Page 6 continues with discussion of finding answers for definition questions, which relies on a pattern matching approach to find answers based on 38 internally developed patterns, some of which are in Table 5 in the paper.

Page 7 finishes of with a discussion of their list based approach, which uses a threshold cutoff approach that determines the similarity between the first and last answers in the list, and all those in between, cutting off the answers when they reach a threshold.

The rest of the paper is performance evaluation and references.

Discussion of Sections 5 through 8

The remaining sections of this paper are discussions of two applications of the algorithms plus the body of related work and conclusions that can be drawn from the work. Section 5 gives the details on what corpora were used (Enron, plus an internal email set not publicly available) before proceeding to the task of name disambiguation.

Name disambiguation in email is the task of correlating the mention of a name in an email with the actual person. While this is fairly straightforward in most cases for people reading their own email, it becomes difficult when reading other’s email since one may not know all the people the other person does. Multiply this by a collection of emails filled with nicknames and initials and it is not hard to see why this is difficult. The task is useful for establishing social networks as well as other applications. One could easily imagine an automated system that retrieved relevant information about a person mentioned in an email (bio, address, phone, past conversations, etc.) and made it available to the reader for instant access.

The remaining parts of section 5 go into the details of applying the algorithm and the results that are achieved.   The interesting thing in my mind is that the graph often connects different types of nodes and associates different probabilities with the transitions from one node to the other.  Applying the graph walking strategy then leads to the desired results.  Suffice it to say, the new approach performs better than the baseline approach!

Identifying threads of emails is the second application the authors use to demonstrate their capabilities.  Threading is the problem of identifying one or more messages that are related to some chosen email.  Many email systems do a basic job at this by comparing subject lines, esp. those that use the “RE:” prefix.  However, we all know people often treat the subject line differently.  Furthermore, people tend to quote the previous messages in the thread differently.  Some use “>”, while others use “|”, while still others use nothing at all.  Add in inline replies, which is especially common on mailing lists, and you see why the problem becomes difficult.  Section 5.4 lays out the graph walk approach and compares it to TF-IDF IR approach, which, of course it does better, especially when using a machine learning re-ranking approach.

The rest of the paper is on related work and conclusions.  I am glad the authors address the performance in terms of scalability in the conclusions section, as I had my doubts about how well the approach could perform on large amounts of data.  In fact, I find a lot of papers in the NLP realm fail to account for performance, so it is refreshing to see it addressed.

graph theory, email, natural language processing, NLP, information retrieval, IR, threading, name disambiguation

Technorati Tags: graph theory, email, natural language processing, NLP, information retrieval, IR, threading, name disambiguation