Introduction

The Internet and World Wide Web are home to vast repositories of information - from text to multimedia, from amateur opinions to expert thought, from voluntary contributions to commercial interests - on every conceivable topic. Lawrence & Giles [#!Lawrence-Giles-1999!#] estimated the publicly indexable Web at 800 million pages, 6 terabytes of text data, on 2.8 million servers, as of February 1999. Internet search engines, which serve as a gateway to this information repository, have established a crucial role in today's important society. Most studies of Internet usage find that search engines play a vital role in information retrieval over the Internet. A recent USA Today (Dec. 11, 2000) article states that 100 million queries are made on U.S. search engines each weekday, and a study of Web usage by Media Metrix found that the top 3 search engines were each visited by 61%, 56% and 40% of tracked Internet users during the past month. The widespread use of search engines has facilitated technology transfer, so that search engine technologies are now licensed to business Web sites, used in digital library systems, etc. For the purpose of this paper, the term search engine encompasses various applications of these indexing-retrieval technologies, including traditional Web search engines (e.g., Google), metasearch engines (Metacrawlers), niche search engines (e.g., DEADLINER (Gruger etal (2000)) [#!Kruger-etal-2000!#]), information portals (Yahoo!), and comparison shopping engines (mySimon). Most search engines began as university projects that focused more on development and algorithms, and less on revenue generation. Even after transitioning into commercial entities, search engines tended to operate as a free resource to content providers and users alike. However, the recent drop in supply of cheap venture capital and sweat equity has forced commercial search engines to investigate mechanisms for generating revenue from content providers. These mechanisms - which we generically label as paid placement - include a fee for inclusion in the database, an increased relevance score in response to a query, or featured listings on the results pages. A paid placement strategy usually requires a minor modification of the ranking algorithm or to the display of results, either of which can be made at very low cost. Paid placement is widespread in search engines (e.g., Google), information portals (e.g., Yahoo!, and metasearch engines (Metacrawler). Nearly all major search engines and portals employ paid placement. Table  presents data on the extent of paid placement for metasearch engines. And, as Figure  indicates, the major comparison shopping engines also employ paid placement. The focus of this paper is on a search engine's strategy regarding revenues from content providers in its database, and how this objective conflicts with its other revenue sources which are a function of its user base, such as advertising and licensing revenues. We develop a mathematical model to analyze the dilemma that search engine faces in raising revenues: it wants to charge content providers for priority placement, but this reduces the search engine's credibility, hence its market share and potential user-based revenues. Specifically, we determine the optimal paid placement policy, i.e., the optimal placement fee and the resulting percentage of sites that choose paid placement. Our longer term interest is to determine the optimal bias-level that would give a search engine the best balance between revenues from content providers and revenues based on its user base. The revenue problem is a critical one for search engines, since it impacts both current performance and future development and improvements. In spite of many years of research on information retrieval, search engines are still far from perfect in terms of the usual metrics of relevance and recall. Hence, there remains considerable research and commercial interest in refining the indexing and ranking algorithms, and user interfaces, employed by search engines. Recent research examines a variety of topics, including Web page ranking algorithms evaluation and comparison of alternative ranking algorithms (Singhal & Kaszkiel [#!Singhal-Kaszkiel-2001!#]), contextual and topic-based search (e.g.,(Bharat & Henzinger [#!Bharat-Henzinger-1998!#]), design and evaluation of metasearch engines (e.g., Dreilinger & Howe [#!Dreilinger-Howe-1997!#]), metasearch using full-text analysis of Web pages (e.g., Lawrence & Giles [#!Lawrence-Giles-1998!#]), and visualization of results (Hearst [#!Hearst-1995!#]). Since further research and development is expensive, commercial search engines need to find new revenue sources in order to balance these costs. Paid placement offers an intriguing possibility: placement revenues in one period can support research and development aimed at improving indexing and retrieval algorithms, database index, or user interface. Hence the negative impact (on users) of paid placement could be reversed by using placement revenues to improve search engine quality.

The rest of this paper is organized as follows. In §, we develop our model of the search engine's revenue problem, considering network effects, the effect of paid placement, and third-party revenues. We characterize the optimal paid placement strategy in §. In § we discuss the sensitivity of the paid placement strategy to various controllable parameters such as the extent of bias and search engine quality, and other factors such as perceived disutility and the advertising rate. We conclude with a summary of our results and possible application of our work to other forms of Internet-based information intermediaries.