Experimental Environment

To set up an experimental environment that would allow us to objectively study the effectiveness of TREC ad-hoc algorithms for a realistic web search task, we need the following tools:

• Queries: A set of real user queries.
• Judgments: User judgments for the ``goodness'' of a page/site retrieved by a search engine.
• TREC Implementation: An implementation of a state-of-the-art TREC ad-hoc algorithm.
• Evaluation Measure: An objective evaluation measure that accurately measures search effectiveness for the search task at hand.
• Collection: A large collection of fresh web pages.

In the following we discuss our approach to selecting and building each of these components.