Unlike Microsoft that gets its money from shifting boxes Google relies on advertising to pay its way. There is a tremendous incentive to improve the quality of searching. The first reason is obvious. The better Google is perceived to perform as a search engine, the more people will use Google for their searches and the greater the traffic for advertisers. The second reason is a little bit more sinister. Google gets paid according to the number of clicks made on an advertisement. Google as well as telling you the results of your search needs also to put some ads your way. The share price of Google is closely linked to the perceived quality of search.

The quest for AI
As one might expect Google is deeply into AI. AI one might argue is essentially what the core business of Google depends on. Suppose we can take a web page, find out exactly what it is about, extract all the relevant facts and put them into a database, then on the prompting of a query from a user marshal all the facts that are relevant to that enquiry. This is what an AI system looking at web pages would essentially do.

http://news.com.com/2100-11395_3-6160372.html
Google is talking about the size of the human genome and the size of AI. I think the arguments are a little bit misleading. I would prefer to look at what we would expect from AI. Suppose I were to show you a box and I told you that that box was “intelligent“. What would you expect. Well Alan Turing devised what is now known as the Turing Test. He said that if the response of a computer to a conversation was indistinguishable from that of a human, it had passed the Turing Test.

On the subject of the Turing Test, Alan Turing envisaged a test which would distinguish between men and women and also would be psychic. Turing believed in ESP. Looking at Alice I am aghast, whenever I say something she always changes the subject. Hardly surprising in view of the Spanish! (La estacion de resorte – El barco attravesta una cerradura)

In other words I would expect to be able to ask questions and get an intelligible response. I could engage in a conversation if I wanted greater depth. If the box claimed to speak Spanish I would expect translations which showed an understanding of context. In fact it could not produce an intelligible response without context. We would also want the answer to statistical questions, like how do people like BMW cars? What is the correlation between this and that? Can we deduce anything about cancer from the people who get it their lifestyles etc?

We would also like to see some evidence of reasoning ability. Google is not committed specifically to reasoning. In a sense reasoning comes after the ability to retrieve efficiently. This has been discussed by myself and other people in “Creating Artificial Intelligence”
http://groups.google.co.uk/group/creatingAI?hl=en
I have also written the following blogs.

http://ipai.blogspot.com/
http://ipai1.blogspot.com/
http://ipai2.blogspot.com/

One thing to remember and that is that the ability to find facts is closely related to the ability to automatically construct wrappers. This is one of the main features of Web 3.0.

Let us now return to Google and what they are doing to produce a Web based AI

Searching – The fundamentals
Search engines are basically databases. The information which is contained in the database has changed throughout the years. What the user needs to know about a Web page is :-
1) What is it about?
2) How is it rated, is it written by a crank or does it contain good and useful stuff?
http://infolab.stanford.edu/~backrub/google.html
Describes the main techniques used in search engines.
Google became the primary search engine on the basis of what might be termed a citation index. Scientists have used this principle almost from the year dot. At the bottom of an academic paper are references and these references are “citations”. The “Science Citation Index” is an index of papers which cite a given paper. Now a paper which is frequently cited is generally regarded as being a good paper. Google does exactly the same things with hyperlinks. There is also the number of times other people access a website.

http://209.85.163.132/papers/sawzall-sciprog.pdf
The Web is of course very large and Google has to find a way of dividing up the tasks. This paper is the key to the way in which Google does this. The database is far too large to place on a single machine, and is therefore stored on a number of servers. Sawzall is quite ingenious. A query is passed round from server to server, but while the query is in transit other queries are being worked on. Hence although a query takes a few seconds to process on the network, the fact that other queries can be processed at the same time means that a high throughput is maintained. One quite important fact is that it is possible to discuss aggregations. That is to say once websites are found with their keywords a further search based on programs written in C++ can be performed.

http://infolab.stanford.edu/pub/papers/google.pdf
Describes what Google was doing in 2000

Google wants to know your surfing history. This will enable it to both target web pages and ads. Suppose I am a civil engineer and I enter “Bridge” as one of my search terms. A civil engineer is interested in “puente that is the sort of bridge that crosses a river. If I am a card player I will be interested in the game of Bridge. A website containing “4 hearts”, is about a card game.

Google also wants to target it advertisments. It wants to know what you think of a particular organization.
http://ryanmcd.googlepages.com/sentimentACL07.pdf
Does just that. It used a training set There is of course one other thing. Advertisers like some sort of feedback on how they and their product is perceived. This paper attempts to achieve this and manages to achieve scores approximating to 80%.

It is not my aim to make moral judgements about Google. Google in fact, unlike Microsoft, has not broken the law. Indeed the Google code is mostly open source. How it is all put together is highly proprietary, but there are references to source code in all the papers. If you are bundling inaccessible code with a inferior operating system (Windows as a sheer operating system is inferior to Linux.) a fine of x million Euros a day is appropriate. Google technology is immensely powerful and society will have to come to terms with it in some way.

Google and Semantic Analysis
http://www2007.org/papers/paper342.pdf
This is a most remarkable paper. Let us dissect some of the terminology. It talks about “Vectors”. What are these “vectors”? They are all derived from Latent Semantic Analysis, or some other allied method. It talks about partially indexing the vectors (not storing the full vector). It takes queries and search results. It actually looks at what people have put into their query as keywords and the web pages they actually click on. An algorithm is developed for giving people exactly what they want. The paper makes great play on optimization for an inveted file search. Now an inverted file is a database file where the entries are indexed. Quite clearly if you are doing web based searches

http://labs.google.com/papers/orkut-kdd2005.html

This paper is 2007 so its results are not yet in “Google”. The methodology is amazingly powerful and could be applied in a variety of circumstances. Slightly chillingly the “Orkut” set which correlates friendship and personality and other similarities is used. The paper can effectively find you matches and build you up a friendship network. Equally it can judge you by the friends that you have!

Potentially you could take El Cid and its English translation and match words up. Rather you are not just matching words you are matching vectors. An inverted file then gives the correct Spanish translation for an English vector and vice versa. This program will take any set of vector pairs and do a match.

Translation
At present translation with Google Translate is rather poor.
El barco attravesta una cerradura - The boat goes through a lock
La estacion de resorte - The season of spring.

http://www.stefanriezler.com/PAPERS/NAACL06.pdf
Google have in 2006 recruited Stefan Riezler. It is interesting in that it indicates a direction in which Google is moving. Here is his CV
http://www.stefanriezler.com/CV07.pdf
It is probably a pretty good summary of the way in which Google intend to go. One thing should be pointed out straight away and that is that is that the Google NLP initiatives are based on strict parsing as their starting point. This contrasts with some versions of Latent Semantic Analysis where unparsed words are entered. Google looks at subjects, verbs, adjectives, adverbs, objects and possessives. Google is also interested in question and answer responses.

http://www.cs.nyu.edu/~mohri/postscript/hbka.pdf
This paper is a review article about a very much related area of speech recognition. I think I should say straight away that the recognition of individual phonemes by computer is as good if not better than that performed manually. The reason why human speech recognition is better than that of a machine is that humans recognize words in context. This in fact makes speech very similar to translation. I can illustrate this with words that have different meanings and spellings but the same phoneme structure. Whether (si), weather (tiempo) hear (oir), here (aqui). One thing that is a little bit disappointing is that the speech and NLP groups in Google appear to be working independently.

Speech is in fact a far harder problem than translation or the discernment of meaning from text. This is because in translating from text you have fewer choices. The methos used is Markov chains and the association of neighbouring words including grammar. Interestingly in neither Riezler’s work or this are words chosen on the basis of long range meaning. Let us say we have a medical paper and we could bias the search to medical terms. They do not seem to do this.

To produce the right words in speech you need an iterative annealing process. This means you may wish to change the phoneme, or word, assignment once other words have been found.

http://www.stefanriezler.com/PAPERS/ACL07.pdf
Suppose I am not looking for a website. Suppose I want to know a fact. “What is the velocity of light?”, “What is somebody’s address?”, “What is the turnover of company X?”. To answer a question the question needs to be parsed so that its meaning can be ascertained. We are here quite close to the Turing test.
http://www.cs.cmu.edu/~acarlson/semisupervised/million-fact-aaai06.pdf
http://www.cs.bell-labs.com/cm/cs/who/pfps/temp/web/www2007.org/papers/paper560.pdf

This is the first stage of Google’s program. A database of, initially, a million facts will be gathered. These facts are going into a database which will be used to answer questions. This will of course be extended as time goes on.

Head to Head with Microsoft
Google has a spreadsheet and a word processor. It also features desktop publishing.
https://www.google.com/accounts/ServiceLogin?service=writely&passive=true&continue=http%3A%2F%2Fdocs.google.com%2F%3Fhl%3Den_GB&hl=en_GB&ltmpl=homepage&nui=1&utm_source=en_GB-more&utm_medium=more&utm_campaign=en_GB
There are advantages and disadvantages in using the Web for basic word processing and spreadsheets. The advantages are that the software is :-

1) Up to date.
2) Will run of both Linux and windows systems.
3) Is free.
4) There are facilities for work sharing.

http://labs.google.com/papers/gfs.html

5) Your work is backed up automatically.

The disadvantages are that you need to be connected to the Web to access your work. There are question marks over security, although to be fair Google is investing a considerable effort in this field.

http://labs.google.com/papers.html
This gives a list of Google papers. Note those on security. I have not mentioned them individually since my main thrust is AI.
I feel that we should look at spellcheckers and how word processing and AI can be integrated. Often when we spell words wrong the spelling is valid but means something different. People will often spell words that sound the same wrongly. This puts spellcheckers in the same position as translators, and on a Web spell-check the latest translator can be used. If I use a translator as a spellchecker I am one stage up on anything Microsoft has produced. If you are writing in Spanish “si” and “tiempo” are never confused. In English large number of people confuse “whether” and “weather”. Present day spellcheckers pass both.

There is one other point. If I want to write something learned, I want references. If I write on web software Google can suggest them to me. If I have Microsoft software on my own computer, it cannot do this.

Conclusion
I started off this investigation rather skeptical. I came away from Google translate distinctly unimpressed. “La estacion de resorte” – I did not know stations were elastic! I came away deeply impressed with the work which Google are doing and its scope. My criticism that the research on Natural Language should involve more interchange of information is perhaps rather carping, considering the difficulties involved in running a program on this scale.
On the question of personal information I can see where Google is coming from. Lets put it this way. If you meet a friend in the street you will have remembered some of the “personal” information that they have told you if you were to start a conversation. We can thus show that any Turing machine must store personal information, you need personal information stored if you are ever going to “talk to Google”. It is also vital for proper retrieval of information. The information you get must be relevant to you.
http://michaelaltendorf.wordpress.com/2007/06/13/top-100-alternative-search-engines-from-readwrite-web/
The whole point of search engine technology is to get relevant references and facts. This reference misses this point completely. If you need a 3D display your basic engine is lousy.

Google is now entering the world of facts rather than just websites. This could have some very interesting consequences in the future.
There is one fact that society in the future will have to come to terms with. To become president of the United States you need television exposure. Television, telephones and the Internet are now becoming one. Who will choose the programs you watch? Why Google of course. This is a tremendous responsibility.

Otherwise, I will be taking a brief hiatus from reviewing papers until I return from ApacheCon Europe in the early part of May where I am giving a talk and a training on Lucene.  I have several key deadlines over the next two weeks that must take higher priority, including the publication of a couple of articles that I have been working on.  I will post details on the publication on my Lucene blog when they are available.

Microsoft’s AskMSR system, as detailed in this paper, takes a different tact for trying to solve the QA problem. Unlike most systems that use sophisticated NLP techniques, AskMSR makes the bet that somewhere on the web the exact answer is written in just the right way to answer the question. Their approach is touted to be much simpler and much more efficient. Also, they suggest this approach is complementary to the other approaches, even though in this paper they are trying to see just how far they can get.

AskMSR relies on the Internet as a massively redundant data source. Section 2 of the paper lays out the System Architecture, which can be br0ken down into 4 sections:

1. Query Reformulation
   1. Takes the input query and rewrites into a number of different possible answers. As a last resort, they also produce some less precise versions made up of non-stopwords. No POS tagger or parser is used.
2. n-gram mining
   1. Given the rewrites, they submit them to a search engine, get the page summaries and use n-grams to calculate candidate answers and score them.
3. Filtering
   1. Given the n-grams, AskMSR then filters and reweights the answers based on answer-type, as specified by a few in-house, handwritten filters. This is most likely one area where more advanced NLP techniques could be used to determine.
4. n-gram tiling
   1. The tiling algorithm merges similar answers and creates longer answers from overlapping answers.

Section 3 then details the experiments that were run, how they performed and how the different components contributed. Ironically, they use Google for their search engine, but my guess is there current system does not. Section 3.2 lays out how important each of the individual components are to the process. While query rewrites weighting are pretty important (11.2% drop in performance when all are weighted equally), n-gram filtering is the single most important part, contributing nearly 18% over the baseline. Tiling rounds things off by contributing approximately 14% to the process.

Section 4 has a discussion of how the various components contribute to errors. Interestingly, many of the errors are due to their projection of answers onto the TREC-9 corpus and wouldn’t really be an issue in a live system. Another interesting error source is the inability to query the search engine for numbers without including the number in the query:

“For example, a good rewrite for the query How many islands does Fiji have wold be <<Fiji has <NUM> islands >> but we are unable to give this type of query to the search engine”

It seems to me, though, that this could be addressed by some type of bounded approach.  For instance, why not just iterate through trying some numbers, within reason.  I wonder, too, how good their potential answers would be if they just put in any number.  I tried the Google query: “Fiji number islands” and the top three results have the answer and the third one, I think would be handled by the n-gram approach.

Section 5 is a good discussion on knowing when the AskMSR system does not know.  They use a decision tree that uses a number of factors, like query length, number of stopwords, etc. The thought process behind this is that why bother trying to answer questions that you know have a very low likelihood of being answered in the first place.  For example, they know that they don’t do very well on how questions, so they could simply say they don’t know the answer or just return keyword search results for the question (in a live system).  Of course, it all depends on how tolerant your users are of incorrect answers.

Well, that’s it for this week.  Haven’t decided on next week yet.  May do another QA paper, may look into some clustering algorithms.  Anyone have anything in particular they want to look at?

In this paper, LCC demonstrates that using a theorem proving strategy to validate answers found through traditional QA means can improve performance by over 30% on TREC style questions.  To support this claim, LCC takes us through, at a high level, the steps involved in creating a theorem prover for use in QA.

Section 1 introduces us to the problem at hand and why they think COGEX is useful for QA and what types of challenges were overcome to implement it.  To summarize, COGEX creates logical representations of the questions, candidates and other world and language knowledge to re-rank candidate answers and remove incorrect answers.  It should be noted that it doesn’t identify the candidates to begin with, just “proves” they are correct.  The main technical challenges in the approach occur in two main areas: creating the logically representation of the necessary world knowledge and other inputs and the high failure rates and long processing time required to apply the prover (I wonder if LCC has overcome these for their online demo, or if they don’t use the theorem prover in the demo.)

Section 2 outlines how the theorem prover is integrated into the QA system.  They have a module called the Axiom Builder which takes in various free text inputs and WordNet glosses and builds logical representations of these.  On a side note, I wonder about the storage and retrieval algorithms used for these.  I would imagine one needs a way of quickly looking up the appropriate pieces, especially when it comes to the Wordnet glosses.  All of this input is fed into the justification module, which tries to prove out the theorem, relying on some relaxation techniques if the proof fails.

Getting into the guts of the program, sections 3 and 4 discuss in detail how to create logical representations of the text and the world knowledge incorporated into the system.  For the text, they use what they call a “logic form” which is a middle ground between a syntactic parse and a deep semantic representation.  In other words, they capture the various pieces of the structure of the text that they think are important, in this case: “(1) syntactic subjects, (2) syntactic objects, (3) prepositional attachments, (4) complex nominals, and (5) adjectival/adverbial adjuncts” (page 2).  To build the logic form, they map the words to the predicates, using the base form plus part of speech info.  Nouns are supplemented with a tag that allows it to be referred to from other predicates.  From here, they use grammar rules to construct the actual representation.  Since there are so many grammar rules, they observe that the top ten most frequently used rules cover more than 90% of the cases that they need in WordNet.  If I understand this correctly, they are saying common grammar rules such as Subject Verb Object, etc. are sufficient for most of their cases.

The next part  of the paper details how to use WordNet glosses to build a bank of world knowledge for use in the theorem prover by developing lexical chains linking synsets across hierarchies.  This is useful for increasing passage retrieval and extractions, as well.  The chains codify topically related concepts and are used in the proving algorithm.  Several examples are given at the end of section 4 for those interested in seeing more details.

Before getting to how the theorem prover works in section 6, section 5 discusses how NLP Axioms are built for the various language nuggets they are interested in, such as complex nominals, possessives, etc.  One thing to note is these axioms have strengths associated with them that play a role in scoring the answers in the prover.  Weaker axioms result in weaker confidence in a given proof.

Sections 6 and 7 discuss how the axioms are used and show an in-depth example of the process in action.  Reading the syntax is a bit mind numbing, but essentially it relies on a couple key pieces: hyperresolution, paramodulation and proof by contradiction.   See the subsection of section 6 on Inference Rules for definitions of hyperresolution and paramodulation.  In my naive understanding, they are ways of reducing and substituting to make the proof more manageable.  Proof by contradiction is simply a way of proving something by assuming it is incorrect and then logically deducing that if it were incorrect than some known fact would be wrong.  In the example of section 7, they assume there is NOT a company/organization that developed the Mosaic Internet browser which is contradicted by the fact that their passage states that NCSA developed Mosaic.  One interesting thing to note here, is this type of proof would not work well for subjective things or on passages that are lying or incorrect.

Section 8 discusses results.  Namely, they claim a 30% improvement in correct answers for TREC 2002.  Quite significant results with the trade off lying in the time and effort it takes to both codify the prover and actually run it on real systems.  This is often the trade off in NLP systems in the current state of things.  It seems you can have either fast or good, but rarely can you have both when it comes to deep, complex problems such as QA.

The system is divvied up into several components, as displayed in Figure 1 of the document.  They are the question processing unit, document processing, factoid answer processing, list answer processing and definition answer processing.  All documents are processed in the same way and passages are retrieved based on the keywords in question.  Depending on the type of question, some passages are removed if they do not have the right answer type.  Passages having a higher number of expected answer types are favored for the list based approach.

For factoid questions, LCC used their CICERO LITE system to provide extractions and answers were calculated based on the extractions and/or the expected answer types.    The extraction process had to identify a variety of semantic classes, such as quantity, date, people, city, etc.  The paper then discusses the types of questions that were answered by these approaches, as well as some special case scenarios related to manner of death (kind of morbid, but death is often a fascination of this kind of research, in my experience).  From here, there is a discussion of the role of theorem proving in the algorithm (see page 5), but the details are left to another paper (guess what will be the paper next week?)  I must admit, I don’t fully understand page 5 and 6 just yet, mostly, I think, because I’m not familiar with the syntax they are using, so maybe reading the next paper will make it more clear.

Page 6 continues with discussion of finding answers for definition questions, which relies on a pattern matching approach to find answers based on 38 internally developed patterns, some of which are in Table 5 in the paper.

Page 7 finishes of with a discussion of their list based approach, which uses a threshold cutoff approach that determines the similarity between the first and last answers in the list, and all those in between, cutting off the answers when they reach a threshold.

The rest of the paper is performance evaluation and references.

1. The Passages Task
2. Main Task – Divided into three sub tasks
   1. Factoids
   2. Lists
   3. Definitions

The passages task is used to test whether a QA system can find factoids in fairly short spans of text (250 characters). As compared to the factoids sub-task under the main task, this approach allows answers to be a little more “loose”. Answers are judged correct if they contained the right answer, the document that they came from supports the answer and that the answer was “responsive”. Responsive, in my understanding, means answers have the correct units or they refer to the actual item of interest and not a replica or imitation. The evaluation judged the accuracy of the answers as the fraction of correct answers as determined by human judges.

The main task was broken up into three sub tasks. Participants had to do all three tasks. The factoid task is pretty much the same as the passage task, but more strict in that exact answers had to be returned, not just passages. The list task requires systems to return one or more list of items that make up an answer. For example, if the question was “what are the 50 states that make up the United States?” a correct answer would have to enumerate all 50 states. Finally, the definition task required the system to define things, such as “Who is George Bush?” or “what is information retrieval?” Much of section 2.3 details how the definition task was evaluated. If anything, it makes you think about how you go about, as a person, finding a correct answer to a question. Many people, I think, take for granted that google or something gives you a correct answer to your questions.

Section 4 is an extensive discussion of how the evaluation was completed.

In my opinion, having worked on a QA system, I would say there are a couple of other areas that are of interest in QA.  First, though, it should be noted that QA is a very hard problem, not only do most systems rely on an IR system for retrieval, which is less than perfect, but then you need to find the exact answer in the given passage.  At any rate, I also think there are several other areas that warrant research and are more interesting in some ways.  First, are questions of that require longer answers that may span multiple passages or that answer questions targeted towards a more demanding audience.  For instance, how could you answer “why, in scientific terms, is the sky blue?” in 250 characters or less?  Second, of interest are questions that require essay style answers or answers that fit into positive and negative sections, such as “what are the pros and cons of tariffs on China?”  Granted, these are much harder, but I think they are much more realistic in some ways.  To me, definition questions are fairly quickly answered by searches on Google or Wikipedia.  To some extent this is also true for list questions as well.

Now that we have a nice intro to the QA task, we will start to look into how people implement QA systems.

This paper should provide us an introduction to the QA problem and provide some background on evaluation.  From here, we will start looking into the different approaches taken for question answering.

natural language processing, QA, question answering

Technorati Tags: natural language processing, QA, question answering