Refereed Papers

Track: Search: Crawlers

Paper Title:
Recrawl Scheduling Based on Information Longevity

Authors:

• Christopher Olston(Yahoo! Research)
• Sandeep Pandey(Carnegie Mellon University)

Abstract:
It is crucial for a web crawler to distinguish between ephemeral and persistent content. Ephemeral content (e.g., quote of the day) is usually not worth crawling, because by the time it reaches the index it is no longer representative of the web page from which it was acquired. On the other hand, content that persists across multiple page updates (e.g., recent blog postings) may be worth acquiring, because it matches the page's true content for a sustained period of time. In this paper we characterize the longevity of information found on the web, via both empirical measurements and a generative model that coincides with these measurements. We then develop new recrawl scheduling policies that take longevity into account. As we show via experiments over real web data, our policies obtain better freshness at lower cost, compared with previous approaches.

PDF version

Inquiries can be sent to: