Section 1 outlines the problem of training, namely obtaining a good
sample of text to be labeled for the trainer.  After disposing of
several other methods of garnering samples (random, relevance
feedback based), Lewis and Gale introduce an iterative approach for
manually labeling examples.

Section 2 then discusses the benefits of “learning by query” in
theory, namely the possibility of reducing the error rate very
quickly in comparison to the number of queries required.

Figure 1 (described in section 3) outlines their basic approach,
which relies on having a human judge some subset of examples that the
currently used classifier is least certain about.  This process is
iterated until the human feels satisfied with the results.  One
caveat of this approach is that the classifier must not only predict
the class, it must give a measurement of certainty for that class.

Continuing on into section 4, we are introduced to how to build a
classifier and use uncertainty sampling to train it.  Most of the
section details the probability theory behind it, finishing up with
how to do the sampling.  One thing I always wish for in these papers
are concrete examples (maybe as an appendix or a reference) that work
through the math on an actual toy problem.  Section 5 does just this,
laying out an experiment and discussing the details, minus the math,
which probably suits most people just fine.

Section 7 has an excellent discussion of the results, the pay dirt
being that using this new method significantly reduces the number of
examples required for training, at the cost of having a human in the
loop.

Text classification is the process of assigning a document to one or more categories (we looked at classification/categorization earlier when exploring Support Vector Machines, SVMs).  My understanding of the difference between categorization and classification is that categorization has a set number of categories, whereas classification  does not.  At any rate, this paper is  comparing two different classifiers that  use  a naive Bayes approach.  The naive Bayes approach assumes that all attributes of the examples we are studying are independent of each other.  Even though this is rarely true in the real world for text (after all, if I chose all the words of this post independently we probably would have gibberish) it turns out that it still works pretty well in practice.  But I digress… The two approaches that are being compared are the Bernoulli model and the multinomial model.  The Bernoulli model uses the document as an event and builds a vector of binary attributes based on whether a term occurs or not in the document.  It DOES NOT take into account the number of times the word occurs.  In the multinomial approach, words are the event and term frequency does matter.  The next couple of sections after the introduction layout the common ground between the two approaches as well as the differences.  The differences come down to how the probabilities are calculated.

There is some interesting discussion of feature selection (a way of reducing the size of the vocabulary, which speeds things up, without, hopefully, losing too much information) using mutual information that is worth digging into a bit more if you have the time.

The next sections are where the rubber meets the road and the authors do a side by side comparison of the two approaches using 5 different collections.  You can see the results on pages 5 and 6.  Finally the discussion of the results occurs on page 6 and 7, with the bottom line seeming to be that the multinomial model seems to “be almost uniformly better than the multi-variate Bernoulli model.”

For those interested, Weka has tools for building Naive Bayes classifiers.

Text Categorization is the problem of assigning one or more predefined categories to a piece of text. The Support Vector Machines approach is a supervised machine learning approach that attempts to learn a linear classifier (actually, it can be polynomial or other type using plugin functionality to the algorithm.) It is actually trying to find a “linear separating hyperplane (see guide for more info on the math.) For an implementation in Java (and other languages) check out http://www.csie.ntu.edu.tw/~cjlin/libsvm/. This site has many useful resources explaining the algorithm, how to do feature selection, etc. (see the guide.)  In two dimensional space, think of finding a line (or polynomial) that separates your good examples from your bad ones.
After introducing the topic of text categorization, the author discusses a bit on feature selection. A feature vector in the paper is a vector of distinct words and their document frequency, minus stop words. The author also throws out any words that occur less than 3 times. I understand the stop word removal piece, but I don’t get the less 3 times reasoning. My guess is it is because those terms don’t significantly contribute to the solution. Finally, features are scaled by their inverse doc. frequency. This all makes sense to me coming from IR (Info. Retrieval land.) Removing stopwords and doing length normalization are standard techniques for improving search results.
The cool thing Joachims points out about SVMs is that they “can be independent of the dimensionality of the feature space”. In other words, SVMs work pretty well with a lot of features or very few features.

In section 4, Joachims makes a very strong case for why SVMs are well suited for Text Categorization.  Summed up they are:

1. Text can have many features (10k+ words in a collection)
2. Most features are important
3. Doc. vectors are sparse.  That is, most words in the collection do not occur in a particular document
4. Most Text Cat. problems are linearly separable.  This is the key idea behind why they work.

The rest of the paper is a discussion of experiments and why SVMs are much better than the other popular approaches in use at the time.  Most notably, if you remember our discussion of Yang 97 from last week, SVMs beat up on kNN quite handily.
computer science, algorithms, support vector machines, text categorization, machine learning, supervised, libSVM

Technorati Tags: computer science, algorithms, support vector machines, text categorization, machine learning, supervised, libSVM

I found sections 5 through 7 to be pretty straightforward.  The author tries to run a variety of classifiers on a number of different corpora.  kNN seems to hold up the best and seems to be the only approach that really scales.  Of course, keep in mind this was 1997, this is probably no longer the case.  Perhaps we can see if there have been any updates to this paper in the future (or perhaps someone can leave a link doing just that.)  Table 3 of the paper contains the guts of what you need to know about the experiments.

All in all, I think this paper is a nice soft introduction to the field of text categorization. The lessons I take from the paper are the following:

1. Pay special attention to what is in the corpus that is being tested.  The inclusion/exclusion of unlabeled documents can make a big difference.  Try to use a standard corpus, if there is such a thing.
2. Know what measures are being used when reading the paper and why people use those measures.
3. Sometimes, simpler is better.  I know the fields need to advance, but sometimes a good solid algorithm like kNN is all you need.

I think for next week, I’m going to continue on with the theme of text categorization and try to dig into a specific algorithm to see how it works.  After that, I have been wanting to look into some Graph Theory algorithms in use with IR and NLP.  As always, leave comments with suggestions on anything you think I just have to read lest I miss the boat completely.

recall = # categories found correct / total # of categories correct = the number we got right divided by the total number of right answers in the system
precision = # categories found correct / total categories found = the number we got right out of the number that we retrieved

For example, you could have perfect recall by assigning every possible category to every document, but our precision would not be very good.
Finally, section 4 ends with a discussion on what average (micro or macro)  to use.  The micro average was chosen because, according to the paper, it favors the more common categories.   This, I think, makes sense given that one is probably more interested in how a system does on the common categories, unless of course you are really interested in the rare categories .   Perhaps someone with a better understanding can contribute some thoughts on this.