(1)

Ethical Web Agents(1)

David Eichmann
Repository Based Software Engineering Program
Research Institute for Computing and Information Science
University of Houston -- Clear Lake
2700 Bay Area Boulevard
Houston, TX 77058
eichmann@rbse.jsc.nasa.gov

Abstract

As the Web continues to evolve, the sophistication of the programs that are employed in interacting with it will also increase in sophistication. Web agents, programs acting autonomously on some task, are already present in the form of spiders. Agents offer substantial benefits and hazards, and because of this, their development must involve not only attention to technical details, but also the ethical concerns relating to their resulting impact. These ethical concerns with differ for agents employed in the creation of a service and agents acting on behalf of a specific individual. An ethic is proposed that addresses both of these perspectives. The proposal is predicated on the assumption that agents are a reality on the Web, and that there are no reasonable means of preventing their proliferation.

1 - Introduction

The ease of construction and potential Internet-wide impact of autonomous software agents on the World Wide Web [1] has spawned a great deal of discussion and occasional controversy. Based upon our experience in the design and operation of the RBSE Spider [5], such tools can create substantial value to users of the Web. Unfortunately, agents can also be pests, generating substantial loads on already overloaded servers, and generally increasing Internet backbone traffic.

Much of the discussion to date has been directed towards this single perspective, the impact of an agent (and indirectly, the operator of that agent) on the Web. Attention more recently has turned to the impact that an agent can have upon its operator - both positively, when needed information resources are ferreted out, and negatively, when an agent's actions results in mail-bombings and (in the near future) substantial financial charges for its operator.

This paper addresses our work in building a spider that is both a good Web citizen and a provider of a generally-useful resource, and how such approaches can be scaled to the increasingly massive information infrastructure of the Web. I contrast this approach with the provider-focussed approach of systems such as ALIWEB [17] and the agents hinted at in such venues as the recent television commercials by AT&T. I conclude with a proposed architecture to support ethical behavior while still allowing operator-defined search in the rapidly evolving Web.

2 - Background: Agents in General and Spiders in Particular

This paper's motivation is the confluence of what was until recently, two distinct threads of research, development of intelligent software agents and development of hypermedia systems (and more specifically, distributed hypermedia systems). This confluence was first exhibited with the emergence of Web spiders(2) in 1993, when the Web grew to a sufficient size as to be interesting to study as an artifact in its own right. A brief review of recent work in software agents and Web spiders will be useful in setting a context for later sections of the paper.

2.1 - Intelligent Software Agents

An agent is a program which interacts and assists an end user. Research in this area has been driven primarily by the artificial intelligence research community, since an unintelligent agent is not of much use. The special issue of Communications of the ACM edited by Riecken [30] provides a good overview of the area. Agent research presents two useful perspectives for a discussion of ethics.

The first concerns the nature of interaction between agents themselves. Genesereth and Ketchpel [12] present an overview of the Knowledge Interchange Format (KIF) and the Knowledge Query and Manipulation Language (KQML), developed as part of the ARPA Knowledge Sharing Effort. Guha and Lenat [13] describe the Cyc project's evolution from an emphasis on addressing brittleness in expert systems to an emphasis on assisting with information management.

The second perspective concerns the nature of interaction between an agent and an end user. Kautz, et. al. [16] focus on what they refer to as "mundane tasks - setting up meetings, sending out papers, locating information in multiple databases, tracking the whereabouts of people, and so on." They distinguish between userbots (mediators for a specific user) and taskbots (responsible for carrying out a specific task). Maes [22] focuses on the reduction of information overload though agents that are trained using machine learning techniques to handle electronic mail, news filtering and meeting scheduling.

AT&T's recent television commercial involving a animated dog posting a note on a computer screen and responding to voice commands (including praise from its presumed owner!) is the natural (albeit still fictional) extension of this work. Norman [28] offers a reflective perspective on how the field might evolve.

2.2 - Web Spiders

A Web spider is a program that autonomously explores the structure of the Web and takes some action upon the artifacts thereby encountered. This action might be as simple as counting the number of artifacts found, or as complex as a full text indexing of the contents of the artifact. Given the relative ease with which a spider can be constructed, it actually somewhat surprising that there are only twenty-odd spiders documented to date [19]. A representative sample of research work in spiders follows, refer to [7] for a more complete survey.

The RBSE Spider [5] retains both the structure of the Web as a graph representation is a relational database and a full text index of the HTML documents encountered. Searches can be specified either as SQL queries against the relation, supporting information such as is displayed in Figures 1 and 2, or against the full text index, providing relevance-ranked results. McBryan's World Wide Web Worm (WWWW) [23] retains a more limited information base, comprised of the document's title and contained anchor information. Search is limited to scanning individual records using pattern matching.

Fielding's MOMspider [10] was designed as a maintenance tool for large webs of HTML documents. It reaches out only to validate the existence of a document corresponding to the URLs found in documents appearing in its maintenance list.

The fish search mechanism of De Bra and Post [3] is a completely distinct category, based upon an executing instance of a modified Mosaic, and supporting a spreading activation of URL retrievals similar in behavior to schooling fish (hence the name). Fish search results are transient, available only to the specific Mosaic user and only for the duration of that execution of Mosaic.

Spider Benefits -- Some Potential, Some Realized

There are distinct benefits to be had in Web spiders, the most obvious of which is improvement in user satisfaction through effective search of the Web -- its scale has completely outstripped an individual's ability to assess and comprehend it. The problem has become similar to that experienced by users of anonymous FTP. Archie arose as a response to the increasing difficulty with locating specific software packages among hundreds of FTP servers. Search-directed access has the potential to reduce traffic by reducing revisitation and casual browsing to "see what's there." Whether this potential will be realized is an open question. Spiders also offer the opportunity to support archivists in the construction of virtual neighborhoods of information. Currently archivists are dependent upon the suggestions of users of the archives or providers who volunteer a description of their information.

Spider Hazards - Mostly realized...

Poorly designed spiders can severely impact both overall network performance and the performance of the servers that it accesses. Many of the known spiders fail to control document retrieval rates, repetition of requests or the retrieval of low-value artifacts (e.g., GIFs). Authors of new spiders frequently fail to address the question of infinite regress, whether in the form of cycles in the graph (obvious) or in the form of dynamic documents (not so obvious). Many of the early attempts at spider construction resulted in information sinks, where information flowed in and little or no information flowed out (frequently, all that flowed out was a simple metric, such as x servers located, or y documents retrieved). Much of this was due to a mixture of naivete and the cloak of anonymity. As providers shifted from pride over access counts to load management, peer pressure and education limited the more egregious cases.

3 - A Rationale for Building Agents for the Web

Hypertext browsing is easily argued as one of the principle strengths and reasons for success of the Web. The Web has demonstrated a practical realization of much of the theoretical work in hypertext. At the same time, there were key aspects missing from the Web's technological infrastructure missing at its inception, the most critical of which was support for indexing.(3) McLeese [25] argues that browsing is central to effective hypertext -- making a distinction between navigation and browsing -- but also observes that a variety of tools are required to most effectively select where to go next within hypermedia. This isn't to say that browsing is not central to hypertext in general and the Web in particular.(4) For example, Campagnoni and Ehrlich [2] found that users of a hierarchical hypertext preferred browsing over indices for navigation. It's instead my claim that it is time that designers and providers of Web services and infrastructure to look to the literature for empirical evidence of what works and what doesn't, and act accordingly.

3.1 - Modes of User Behavior

The important question to ask here is not "what can be built in such a manner as to limit resource consumption?" which is the refrain of opponents of even the concept of Web agents, but rather "what can be built in order to make usage of the Web more effective?" The explosive growth of the Web(5) is making it increasingly difficult to accomplish an effective search for information in a specific area. Effective is defined here as finding everything that is relevant to the search and not finding things that are not relevant -- the traditional sense of the word in library science. Multiple access modes are becoming a necessity as the Web scales up.

Navigation itself has been demonstrated to have distinct modes. Monk [26] distinguishes between directed navigation, where the user traverses a known path to a known artifact, and exploratory navigation, where the user is attempting discovery of previously unknown artifacts. His personal browser foreshadows the Mosaic hotlist facility as a facility supporting directed navigation. Frisse and Cousins [11] distinguish between local and global navigation, and claim that global navigation requires an index space distinct from the document space. This is a key distinction in relating traditional results in hypermedia research, where the corpus resided on a single host, and the Web, where the basic premise is massive decentralization.

The nature of the artifacts themselves can color user preferences for access mechanisms. Dillon, et.al. [4] and Van Dyke Parunak [32] comment upon the likelihood of disorientation as nonlinearity increases. Wright and Lickorish [34], comparing an index-based navigation scheme with one more hierarchical in nature, found readers preferred the latter for book-like text, but the former for more modular information.

3.2 - Some Hard Data from the Web

Just what is the structure of the Web? One of the rationales for the separation in the RBSE Spider [5] of the discovery and storage of Web structure from the indexing and retrieval of the HTML documents thereby retrieved was the ability to generate queries concerning the characteristics of the Web itself. The February 1994 run of the RBSE Spider produced a graph comprised of approximately 36,000 HTML documents, 62,000 distinct target artifacts and 182,000 total edges. As can be seen in Figures 1 and 2, the modularity of the Web is extremely high. While the number of hyperlinks contained within a given document is fairly high (>10 links per document in our index), only a third of those hyperlinks are to other documents (~ 3.5 inter-document links per document in our index), and documents so referenced are referred to in general by only a few other documents (59% of the documents have a single inbound link and 96% have five or fewer). There are, of course, notable exceptions in the data:

    Figure 1: External URL in documents

    Figure 2: Inbound URL references to documents

Clearly, if this is a representative sample of the web, having one-half of the artifacts reachable from a single other document in the web implies that much of this traffic is likely to be redundant -- users must wander from node to node to discover information, and frequently revisit nodes in doing so.

4 - Web Ethics

The concept of an ethic for what is basically an information system might at first seem strange. In fact, we have an excellent example of how a virtual community evolves a set of ethics in a setting of potentially complete chaos -- Usenet news. The readership has evolved over the years a consensus as to how newsgroups come into existence, and what behavior is appropriate within specific groups. The only true constraints are those generated by peer pressure on the offending user and their system administrator. This section begins with a review of the consensus that has arisen amongst participating spider authors and resource providers, describes a similar manifesto originating in the intelligent software agent community, and concludes with a proposal for a web ethic.

4.1 - The Guidelines for Robot Writers

Koster's guidelines [18], authored as a means of addressing the increasing load on his server by spiders, was the first wide-ranging attempt at Web ethics. It served as a basis for discussion amongst spider authors and resource providers, and in conjunction with his "list of robots" [19], began to create the first community pressure on spider authors to act ethically. Briefly, the guidelines entail:

The guidelines were not intended to suppress legitimate research or resource discovery done as a part of the creation of an information service, but rather were intended to stem the tide of "because it's there" spider implementations. As mentioned above, the result has been an emerging consensus on appropriate Web behavior that is still operating reasonably well.

4.2 - The Robot Exclusion Standard

The difficulty with the guidelines is that they still don't provide a means for an information provider to indicate to a running spider that portions or even the entire file space of their server should be off-limits. The robot exclusion standard [20] was defined to address just this. The scheme entails the creation of a file on the server with a standard path (/robots.html) and contents detailing the nature of the desired constraints. For example, the definition appearing in Figure 3 indicates that all spiders should avoid this server, with the exception of alpha and beta, which are granted access to /private, and gamma, which is granted complete access. Further details are available in [20]. Note that while a provider can specify as many constraints on spiders as wished through an exclusion definition, it is still up to the spider itself to check for the existence of the file and adhere to its constraints.

4.3 - The Laws of Softbotics (Etzioni and Weld)

Etzioni, et.al. define a softbot as

"an agent that interacts with a software environment by issuing commands and interpreting the environment's feedback. ... [a] softbot's sensors are commands meant to provide the softbot with information about the environment... Due to the dynamic nature and sheer size of real-world software environments it is impossible to provide the softbot with a complete and correct model of its environment... "[8]

envisioning a construct very similar to a Web agent. Softbots also effect change on their environment, leading to the formulation of a collection of softbotic laws (intentionally derivative of Asimov's laws of robotics) [9, 33]:

4.4 - A Web Ethic

Any scheme of ethics for Web agents must address the motivation of the users who employ these agents, of the individuals that author agents and the providers of information resources that the agents access. Users are seeking guidance and organization in a chaotic, dynamic information framework. They are in a process of exploration when using the results of agents, since other mechanisms (i.e., hotlists and personal pages) serve only the need for directed navigation. Web agent authors respond to this need with a service that, if we believe the feedback received, far outweighs the problems created by their progeny. Even those authors who act in isolation are in a mode of exploration - the concept of "power user" carried to the extreme, where the entire world is at their fingertips. Information providers are interested in dissemination of their artifacts (why else publish them?). The issue is hence one of striking an appropriate balance between interests and concerns - accessibility for the individual balanced against accessibility for the community.

The guidelines/exclusion perspective and the softbotics perspective offer similar approaches to attempting such a balance. However, neither operates from assumptions that match completely with what the Web promises to become.

Assumptions

A Strawman Architecture

Factoring agent functionality into smaller categories offers a useful means of examining the issues. As shown in Figure 4, these categories include user-agent interaction, agent-agent interaction, and agent-server interaction. I specifically exclude user - server interaction, as this is the normal mode of activity on the Web, and doesn't involve agents. Koster's ALIWEB scheme [17] entails agent - server interaction in its accretion mode, when the server extracts current index information from participating servers and forms its aggregate index, and user - agent interaction when a user employs a Web client to search the index. Kahle's Wide Area Information Server (WAIS) system [15], entails a degenerate form of agent - server interaction when a server generates a local index for a collection of artifacts, a meta form of agent - agent interaction when a server posts a description of itself to a directory of servers, and two forms of user - agent interaction - one at a meta level then interrogating the directory of servers for information about servers and one at a normal level of interrogating a server about its contents. Other approaches include that of McKee [24], where a WAIS index is generated for the documents available from a particular server (another degenerate form of agent - server interaction), and the MORE system [6], which supports localized searching of metadata about artifacts residing anywhere in the Web, where two forms of user - agent interaction take place - one by the librarians posting information to the repository concerning already known artifacts, and one by the users interrogating the repository for referrals to specific artifacts on specific servers.

    Figure 4: Categories of Agent Functionality

The spiders discussed so far fall into the general category of task agents - accomplishing a specific task and generating a result for subsequent multiple uses. The single exception to this is the fish search mechanism, which is readily identifiable as a user agent. An architecture that is capable of scaling to millions of artifacts on tens of thousands of servers accessed by millions of users requires aspects of both of these categories. Because of this, I propose a separation of design concerns into those which relate to service agents - those interacting with servers on the Web in the formation of information bases (thereby making themselves servers in their own right), and user agents - those interacting with servers on the Web in direct support of a particular individual.

A Service Agent Ethic

A service agent should adhere to the following ethical guidelines:

Clearly a balance must be struck between the concerns of openness, moderation and respect, which limit a service agent's scope and activities, and the concern of authority which broadens them. A service agent that is not authoritative will not be employed, but one that is renegade [29] is as damaging to the Web as the impact avoided by the community it supports.

A User Agent Ethic

A user agent should adhere to the following ethical guidelines:

The current state of the Web makes the goal of appropriateness difficult to attain for a user agent. The service agents in existence at the current time are for the most part experimental, and hence still struggling with their own issues, particularly in the areas of moderation, respect and authority. Vigilance is a concern that has not yet even begun to generate the impact that it will. As commercial providers become an increasing presence on the Web and evolve towards services with strong authority, limiting the impact to a user of their agent's activity against fee-for-service agents will become a critical concern. Pottmyer has proposed some initial approaches to address this area [29].

5 - Unresolved Issues

While it's easy to argue that there really are not any resolved issues concerning Web ethics at this point, there are some key areas as yet unaddressed that deserve mention:

6 - Future RBSE Work With Spiders

Text-based indices are not a panacea for the Web. Monk observes that keyword-based indices exhibit similar problems to browsers with sufficiently large hypertexts [27]. Future work by the RBSE research group will include:

The resulting knowledge will be used to modify the spider for more intelligent access patterns, integration of exploration results into the MORE storage scheme, and definition of a conceptual (as opposed to keyword) search engine.

References

[1]
Berners-Lee, T., R. Cailliau, A. Loutonen, H. F. Nielsen and A. Secret, "The World-Wide Web," Communications of the ACM, v. 37, n. 8, August 1994, p. 76-82.
[2]
Campagnoni, F. R. and K. Ehrlich, "Information Retrieval Using a Hypertext-Based Help System," ACM Transactions on Information Systems, v. 7, n. 3, 1989, p. 271-291.
[3]
De Bra, P. M. E. and R. D. J. Post, "Information Retrieval in the World-Wide Web: Making Client-based searching feasible," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 137-146.
[4]
Dillon, A., C. McKnight and J. Richardson, "Navigation in Hypertext: A Critical Review of Concept," Proc. of IFIP INTERACT `90: Human-Computer Interaction, Detailed Design: Hypermedia, 1990, p. 587-592.
[5]
Eichmann, D. "The RBSE Spider -- Balancing Effective Search Against Web Load," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 113-120.
[6]
Eichmann, D., T. McGregor and D. Danley, "Integrating Structured Databases Into the Web: The MORE System," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 369-378.
[7]
Eichmann, D., "Advances in Network Information Discovery and Retrieval," submitted to the International Journal of Software Engineering and Knowledge Engineering.
[8]
Etzioni, O., N. Lesh and R. Segal, Building Softbots for UNIX (Preliminary Report), University of Washington, Seattle, WA, November 1992.
[9]
Etzioni, O. and D. Weld, "A Softbot-Based Interface to the Internet," Communications of the ACM, v. 37, n. 7, July 1994, p. 72-76.
[10]
Fielding, R. T., "Maintaining Distributed Hypertext Infostructures: Welcome to MOMspider's Web," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 147-156.
[11]
Frisse, M. E. and S. B. Cousins, "Information Retrieval from Hypertext: Update on the Dynamic Medical Handbook Project," ACM Hypertext `89, Information Retrieval I, 1989, p. 199-212.
[12]
Genesereth, M. R. and S. P. Ketchpel, "Software Agents," Communications of the ACM, v. 37, n. 7, July 1994, p. 48-53,147.
[13]
Guha, R. V. and D. B. Lenat, "Enabling Agents to Work Together," Communications of the ACM, v. 37, n. 7, July 1994, p. 126-142.
[14]
Hayes, B., "The World Wide Web," American Scientist, v. 82, September-October, 1994, p. 416-420
[15]
Kahle, B., Wide Area Information Server Concepts, Thinking Machines Inc., November 1989.
[16]
Kautz, H. A., B. Selman and M. Coen, "Bottom-Up Design of Software Agents," Communications of the ACM, v. 37, n. 7, July 1994, p. 143-147.
[17]
Koster, M., "ALIWEB -- Archie-Like Indexing in the WEB," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 91-100.
[18]
Koster, M., "Guide for Robot Writers," Nexor Corp., http://web.nexor.co.uk/mak/doc/robots/guidelines.html.
[19]
Koster, M., "List of Robots," Nexor Corp., http://web.nexor.co.uk/mak/doc/robots/active.html.
[20]
Koster, M., "A Standard for Robot Exclusion," Nexor Corp., http://web.nexor.co.uk/mak/doc/robots/norobots.html.
[21]
Lai, P. and U. Manber, "Flying Through Hypertext," ACM Hypertext `91, Presentation Issues, 1991, 123-132.
[22]
Maes, P., "Agents that Reduce Work and Information Overload," Communications of the ACM, v. 37, n. 7, July 1994, p. 30-40,146.
[23]
McBryan, O. A., "GENVL and WWWW: Tools for Taming the Web," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 79-90.
[24]
McKee, D., "Towards Better Integration of Dynamic Search Technology and the World-Wide Web," First International Conference on the World Wide Web, Geneva, Switzerland, May 25-27, 1994, pages 129-135.
[25]
McLeese, R., "Navigation and Browsing in Hypertext," HYPERTEXT I: Theory into Practice, 1988, p. 6-44.
[26]
Monk, A., "The Personal Browser: A Tool for Directed Navigation in Hypertext Systems," Interacting with Computers, v. 1, n. 2, 1989, p. 190-196.
[27]
Monk, A. F., "Getting to Known Locations in a Hypertext," HYPERTEXT II: State of the Art, Navigation and Browsing, 1989, p. 20-27.
[28]
Norman, D. A., "How Might People Interact with Agents," Communications of the ACM, v. 37, n. 7, July 1994, p. 68-71.
[29]
Pottmyer, J., "Renegade Intelligent Agents," SIGNIDR V - Proc. Special Interest Group on Networked Information Discovery and Retrieval, McLean, VA, August 4, 1994. Presentation slides available as http://wyww.wais.com/SIGNIDR/Proceedings/SA3/.
[30]
Riecken, D., "Intelligent Agents: Introduction to Special Issue," Communications of the ACM, v. 37, n. 7, July 1994, p. 18-21.
[31]
Schatz, B. R. and J. B. Hardin, "NCSA Mosaic and the World Wide Web: Global Hypermedia Protocols for the Internet," Science, v. 265, August 12,1994, p. 895-901.
[32]
Van Dyke Parunak, H., "Hypermedia Topologies and User Navigation," ACM Hypertext `89, Navigation in Context, 1989, p. 43-50.
[33]
Weld, D. and O. Etzioni, "The First Law of Robotics (a call to arms)," Proc. of the 12th National Conference on AI, Seattle, WA, July 13 - August 4, 1994.
[34]
Wright, P. and A. Lickorish, "An Empirical Comparison of Two Navigation Systems for Two Hypertexts," HYPERTEXT II: State of the Art, Navigation and Browsing, 1989, p. 84-93.

Biography

David Eichmann is an assistant professor of software engineering at the University of Houston - Clear Lake and director of research and development of the Repository Based Software Engineering Program. Besides normal academic duties, his responsibilities include management of a research and development group working in the areas of reuse repositories and reengineering. He joined the UHCL software engineering faculty in August of 1993 after visiting for a year in his role with RBSE. He previously held positions at West Virginia University, where he lead the Software Reuse Repository Lab (SoRReL) group, and at Seattle University.

Email: eichmann@rbse.jsc.nasa.gov

Footnotes

(1)
This work has been supported by NASA Cooperative Agreement NCC-9-16, RICIS research activity RB02.
(2)
A number of terms are used for programs which autonomously navigate Web structure: wanderers, robots, worms, even fish. My use of spider is intended to include all of these.
(3)
That is, most critical from the perspective of this paper's focus. Other key aspects relating, for example, to expressiveness - support for tables and equations in HTML, etc. are not relevant to the discussion here.
(4)
The requirement for navigation is frequently cited as being, in part, cause of the demise of network database systems and non-navigational access has been, in part, the success of relational database systems.
(5)
Just how explosive is reflected in the fact that articles appearing in such generic venues as American Scientist [14] and Science [31] discuss the phenomenon of the Web as much as they do the technology.