Figure 1. Screen Shot of AnswerBus News Engine
AnswerBus News Engine uses news stories published on CNN Web sites as its knowledge base and intends to answer questions on just-happened facts.
Question answering, QA specific indexing
AnswerBus Question Answering System is a Web-based open-domain system. It successfully uses NLP/IR techniques and reaches high correct answer rate. Although it is not designed for TREC, it still correctly answers over 70% of TREC-8 questions with Web resources ([2,3]). The question remains if a special indexing system will work better for the QA tasks.
In the experiment, we locally indexed over 700,000 news stories published on CNN Web sites since 1996. We developed a search engine for the new QA system. The goal of this experiment is to use most techniques used in AnswerBus QA system together with some new techniques, such as QA specific indexing, described in [2,3] but not fully implemented in original AnswerBus system, and build a QA system to answer time sensitive questions in the real world.
Comparing to other QA systems, AnswerBus News Engine has some new features not seen in other QA systems including its previous versions.
The current size of indexed data has been over 700K Web pages from CNN Web site and some of its sub sites. We believe that it has been the largest size of knowledge base for QA tasks at current time. And the designed size can be much bigger than the size we have already reached. It is possible for the future system to index the whole Web and answer questions.
Partially because of the local indexing, AnswerBus News Engine is now able to extract the possible answers for a user question from CNN news stories in 2-4 seconds. This makes the system fast enough to process more documents to mine the answers.
System load has been largely decreased than its previous systems and the system can answer more questions at the same time than its previous versions with same resource.
A QA systems usually uses some search tools to retrieve documents. Many systems use commercial search engines while others use local search engines for local data, for example, local Web contents or TREC QA corpus. For this experiment, we partially deployed the techniques used in Seven Tones Search Engine ([7,6]) for the search task, since it has a high indexing speed and it is possible to update the indexed database part by part. Some new functions including sentence level indexing, temporal indexing have been also implemented in the system.
As the results of the new techniques, AnswerBus News Engine is now able to answer some time sensitive questions about the some factual issues just happened half an hour ago.
The system has a similar Web interface as its original version. As in Figure 1, the system lists up to ten possible answers to a specific user question. Each of these answers has a dynamic link back to a specific CNN Web page containing the answer sentence. The navigation bar at the end provides an easy way to try user question with other online systems.
Some times a QA system cannot find any answer from the working knowledge base for a question. This doesn't mean there is no answer for the question. In this case, AnswerBus News Engine redirects the question to the embedded search engine so users will get a bunch of documents instead of answers. Very likely, if there is an answer to the question, the user can dig it out from the documents given by the search engine.
It gets more difficult to evaluate the system because we don't have any baseline or comparable systems. And also because of the dynamic content, it is difficult to design a question set to do the evaluation.
However, the techniques used in this system and in its previous local archive version ([8]) are almost same. The evaluation data of the local archive version should be able to level the performance of the system.
We refer to the milestones described in [1] and provided questions, which covered all 16 Arthur Graesser's questions categories and 3 other question categories that ranged from easy to very difficult. Table 1 shows the encouraging test result. The accuracy of is 72% in top 1 and 80% in top 5 (Table 1).
| Question Type | Number | Top1 | Top5 | Wrong |
| 1. Verification | 3 | 1 | 1 | 1 |
| 2. Comparison | 2 | 0 | 1 | 1 |
| 3. Disjunctive | 2 | 2 | 0 | 0 |
| 4. Concept Completion | 6 | 5 | 0 | 1 |
| 5. Definition | 6 | 5 | 0 | 1 |
| 6. Example | 3 | 3 | 0 | 0 |
| 7. Interpretation | 3 | 2 | 1 | 0 |
| 8. Feature Specification | 5 | 5 | 0 | 0 |
| 9. Quantification | 6 | 4 | 0 | 2 |
| 10. Causal antecedent | 3 | 2 | 0 | 1 |
| 11. Cause Consequence | 0 | 0 | 0 | 0 |
| 12. Goal orientation | 1 | 1 | 0 | 0 |
| 13. Enablement | 0 | 0 | 0 | 0 |
| 14. Instrumental/Procedural | 1 | 1 | 0 | 0 |
| 15. Expectational | 1 | 1 | 0 | 0 |
| 16. Judgemental | 3 | 1 | 0 | 0 |
| 17. Assertion | 3 | 1 | 1 | 1 |
| 18. Request/Directive | 0 | 0 | 0 | 0 |
| 19. Nils question | 2 | 0 | 0 | 2 |
We also compare our search engine results with the search result from the LookSmart Search Engine used by CNN Web site, and the result from the Google site search. We conclude that our system outperforms these systems in terms of recall and precision.
Question-sentence matching formula used in original AnswerBus system was proved effective in Web-based QA system. However, in the new QA system, it is not working as good as in original AnswerBus QA system. Probably it is because 1) The text in CNN Web site is very formal and the style is almost unique. 2) Few redundant information can be found in CNN Web site.
Based on our experiment of our new QA system, we found that QA specific indexing and searching are quite feasible. Most techniques used in original AnswerBus System are scalable to large size knowledge base. A question answering system uses these techniques can reach a high speed.
Some more new tasks have been already in our future plan. One of them is to add more Web news resource to the system and make the system itself into a news portal. It could be something like a combination of a question answering system and Google News or other online news systems.