Fifth International World Wide Web ConferenceMay 6-10, 1996, Paris, France
Fifth International World Wide Web ConferenceAlberto Riva (ariva@ipvaim.unipv.it) Dipartimento di Informatica e Sistemistica, Università di Pavia, Italy Marco Ramoni (M.Ramoni@open.ac.uk) Knowledge Media Institute, The Open University, United Kingdom
We describe the design and the implementation of LispWeb, a specialized HTTP server, written in Common Lisp, able to deliver Distributed Artificial Intelligence (DAI) applications over the World Wide Web (WWW). In addition to implementing the standard HTTP protocol, the LispWeb server offers a library of high-level Lisp functions to dynamically generate HTML pages, a facility for creating Graphical User Interfaces on the WWW through dynamically generated image maps, and a server-to-server communication method (STSP).
LispWeb is intended to act as the front-end of a network of intelligent agents that communicate with each other using an extension to the HTTP protocol. The dynamic generation of HTML pages allows complex AI applications to be delivered to end-users without the need for specialized hardware and software support, and using a simple and homogeneous interface model.
The current trend in the evolution of the WWW (testified by products like Sun's Java and Netscape's JavaScript) aims at addressing this issue by embedding applications inside the HTML documents, and having them interpreted by the client. This approach obviously requires to put even more intelligence inside the browser, that is already by far the most complex of the two sides of the connection. We consider this solution unsatisfactory for at least two reasons. First, a piece of code that must to run on a variety of different remote machine can make very few assumptions about the execution environment. It must be small in size in order to be transmitted over slow network links. It cannot assume the presence of an interpreter for any particular language and must therefore be written in a language for which an interpreter is embedded in one or more particular browsers. It cannot make assumptions about the amount of resources that will be available (e.g. memory) and must therefore keep its requirements low. Finally, it is committed to the stateless model of the WWW, since it has no way of storing state information between one usage and the next one. The second reason has to do more closely with the kind of systems we are most interested in, that is real-world Artificial Intelligence (AI) applications. These systems rely on the interaction between sophisticated inference procedures and large amounts of structured knowledge. Since in this case only a very small portion of the knowledge-base could realistically be downloaded on the client and most of the expensive computation should be performed by the server, it makes sense to keep the application on the server itself.
We therefore believe that, in order to make them available on the WWW, large knowledge-based applications must be integrated with the server itself. To this aim, the server must provide more than simple document retrieval and script dispatching functionalities: it must be able to reply to particular HTTP requests by invoking the application functions and presenting their output to the user. In other words, it must enable the application developer to link the application code with the server and to control the way the server replies to HTTP requests. The solution implemented by LispWeb addresses the problems outlined above by putting the intelligence on the server side of the connection rather than on the client side; it also allows the application to run in a known and controllable environment, and provides a notion of state that current WWW-based applications lack. Bandwidth problems are of no concern at this state of the development, because LispWeb makes available applications that could not be possibly run on most of the client machines.
The most sophisticated implementation of an HTTP server written in Common Lisp is currently the CL-HTTP system [1], developed at the MIT AI Laboratory by John Mallery. This server does not explicitly provide features to enable the remote execution of Lisp programs, since its main goal is to provide Lisp Machines with HTTP servers capabilities rather than to redesign the way in which information is distributed on the WWW. Therefore, it does not address key technological issues, such as server-to-server communication and graphical interfaces development. Another example of the use of HTTP to access a Lisp environment is represented by the Stanford KSL Network Services [2].
In this paper we present a specialized HTTP server designed to support and to provide access to distributed AI applications. We will describe the implementation of the server, its usage as a development environment for WWW-based applications, and an extension to the HTTP protocol to support server-to-server communication.
LispWeb is an HTTP server written in Common Lisp, that offers all the typical features of the standard HTTP servers: GET, POST and HEAD methods, access control and logging, multithreading, handling of forms and image maps. The main feature of the LispWeb server is that its "pages" are not stored in disk files, but are dynamically generated whenever they are requested: the server can associate a Lisp function to a URL or to a class of URLs, and the output of its evaluation is sent to the client as the request result. The server provides a rich development environment to facilitate the task of writing page-generating functions and to link them together in a complete WWW-based application. We will first discuss the choice of Common Lisp as the server language, and we will then present the high-level tools for the generation of HTML code and graphical user interfaces available to the developer.
The choice of the Common Lisp programming language, though unusual for this kind of applications, was motivated by some very strong reasons. First, Lisp is a very high-level, dynamical and extensible language that has been used for decades in the development of Artificial Intelligence (AI) systems. Common Lisp allows large systems to be prototyped and developed quickly, provides very powerful debugging tools and sophisticated error-handling, and has a very simple syntax that makes it easy to write code generators and parsers.
Modern versions of Common Lisp are also endowed with powerful optimizing compilers, very efficient automatic memory management, support for multiple execution threads, the ability to link with external libraries and interfaces to low-level operating system functions, such as network access.
One of the reasons why Lisp was criticized in the past is the large size of the executable programs that it produces, usually one order of magnitude grater than typical compiled programs. To see why this criticism is not very relevant in our case, it must be considered that a Lisp executable is not just a program, it is a complete development environment that includes the interpreter, the debugger, the runtime-support system, and often the compiler and the graphical interface. Modern Lisp implementations have a modular structure that allows the unused parts of the environment to be discarded, thus reducing the overall size of the code, and take advantage of shared libraries and memory paging techniques to reduce the overhead involved in dealing with very large programs. Moreover, the large size of the Lisp executable is only a problem at startup time, when the image must be (at least partially) loaded in memory. For this reason, Lisp is not suited for being used as a scripting language. On the other hand, when the Lisp application is the server itself, this problem disappears (it only occurs once when the server is started), and the result is a net speed gain since the time needed to invoke a function inside the Lisp environment is usually much smaller than the one needed to spawn an external script stored in a disk file.
Finally, Lisp has been the language of choice for AI research for at least three decades; this has given the Lisp community a great amount of experience in the construction of large and complex applications, and has produced a number of highly successful systems, particularly in the fields of knowledge engineering, automated reasoning, dynamical planning and control. In the past, their use was limited by the need for expensive specialized hardware (e.g. Lisp Machines), and even today sophisticated applications must run on expensive workstations; the ability to make them accessible through the network with very little effort promises to be one of the most interesting future developments of the WWW technology.
(define-page (page-title page-keywords ...) (function-arguments ...) function-body ... )
This macro defines a function that, when called, produces a complete HTML page. The contents of the page can be customized through the page-keywords, or through the function-arguments. page-keywords is a sequence of keyword-value pairs that provide information on the HTML page that the function will generate. These keywords specify various attributes for the generated page, including the name of the Lisp function we are creating, the URL to which it must be associated, the HTTP response code that must appear in the response header, optional meta fields to be included in the header, the background color, etc. Function arguments is the formal parameter list of the function, and function-body is a sequence of arbitrary Common Lisp forms. Together, they constitute the definition of the function that will be generated by the define-form macro. Once invoked, the function will first generate an HTTP header according to the options supplied through the page-keywords, and then will execute the function-body.
Although the body of the function can contain arbitrary Lisp forms, all output operations must take place through the html-out function, whose purpose is to produce a fragment of HTML code. The fragments are put together in the order they are generated (that is, according to the flow of execution inside the function body), and constitute the text of the HTML page. The define-page macro takes care of wrapping the appropriate title, head and body tags around the text itself, and of sending the resulting HTML page to the network stream. The rest of the library contains functions that generate HTML fragments; Lisp expressions are therefore used as a high-level representation of the desired HTML text. For example, the bold function, called as follows:
(bold "This is bold text")returns the following string:
Another example regards the anchor function:"<B>This is bold text</B>".
(anchor "http://ipvaim.unipv.it:3000/hall" "This is a useful link")produces:
"<A href="http://ipvaim.unipv.it:3000/hall">This is a useful link</A>".
Of course, the arguments passed to the HALL functions need not be simple strings, but can be the result of other Lisp expressions such as function applications or variables. For example, the two preceding examples can be combined to produce a bold link:
(anchor "http://ipvaim.unipv.it:3000/hall" (bold "This is a bold link"))produces:
Moreover, the library takes advantage of powerful features of the Lisp evaluator such as optional and keyword parameters, as the next example shows:"<A href="http://ipvaim.unipv.it:3000/hall"><b>This is a bold link</b></A>".
(inline-image "file.gif" :alt "A GIF image" :align *default-image-align*)produces (assuming that the value of the global variable *default-image-align* is "middle"):
Note that the :alt and :align keywords are used to specify optional components of the IMG tag.<IMG src="image/file.gif" ALT="A GIF image" ALIGN=MIDDLE>
The HALL library is completely self-documented: a special purpose page lists all the available functions, and presents an example of the usage of each one of them.
End-user interfaces are recognized as playing a central role in the success use of AI systems. Many AI applications, as well as other advanced computer programs, require complex interaction processes and they need to pay special attention to their end-user interface, often relying on sophisticated graphical capabilities.
LispWeb includes CL-FLY, an integrated environment to develop graphical user interfaces (GUI) on the WWW. CL-FLY implements most of the capabilities of current stand-alone GUI development environments, including graph drawing, pop-up menus, and mouse events handling; its design is based on an object-oriented approach in order to foster fast prototyping and reusable coding.
CL-FLY translates the basic concepts of graphical interfaces into objects suitable to be delivered over the WWW. The concept of "window" is translated into an active image, that is generated and updated on-the-fly. LispWeb keeps the image in memory, updates it according to the input coming from the user and delivers it back to the client. The image is never physically saved on the server.
CL-FLY has been implemented in CLOS, the standard Object Oriented programming language for Common Lisp. There are two basic CLOS classes in CL-FLY: a class of "canvases", representing windows, and a class of "graphical objects" representing active regions on a canvas. Graphical objects can be nested at any depth in chains of parentship. CL-FLY also provides a library of predefined graphical objects for the most common user interface components, such as lines, rectangles, circles, strings, and so on. Graphical output is realized using Gd-Lisp, a library of graphical functions, based on the Gd C library [3], to dynamically create images in "gif" format.
Mouse input is handled in the usual way, for object oriented graphical interfaces: each class of object in CL-FLY, either a canvas or a graphical object, owns a method that is called whenever the mouse in clicked inside its region. A CL-FLY application is therefore developed by specializing the standard classes and the input methods can be customized by overwriting the standard method for handling mouse inputs.
Figure 1 shows the snapshot of a CL-FLY application for the interactive drawing of direct graphs. When the user clicks on the background of the window, a node is created, while clicking on the node brings up a pop-up menu. Using the menu, the user can choose either to draw a direct link between the current node and another one, or to delete it.
As an example of an alternative approach to the development of GUIs in a mixed Lisp-HTTP environment, we may cite the CWEST (CLIM-WEb Server Tool) system developed at the Stanford Research Institute [4]. CWEST is based on CLIM, a proprietary package to build graphic interfaces in Common Lisp, and works by directly translating CLIM output to image-maps in "gif" format. Therefore, it does not provide any form of active interaction capabilities, while CL-FLY is designed to provide a LispWeb application with all the graphical capabilities of a stand-alone environment.
In this section we will briefly outline other interesting features of the LispWeb server. In particular, the following three paragraphs discuss the error-handling capabilities, a source-code viewer for HALL-generated pages and the server maintenance tools.
Effective error handling capabilities are essential for a server that executes user-defined functions instead of just retrieving files. Errors can arise during the application debugging phase, or during normal execution (due to incorrect input or unexpected changes in the state of the server). In all such cases, the server must be able to signal the error in an appropriate way and to resume normal operation. Common Lisp treatment of run-time errors is based on a powerful condition system, that defines the hierarchy of error types (called conditions) and permits to dynamically establish condition handlers. A condition handler is a function that deals with the error in place of the default interactive debugger. The LispWeb server uses this facility to protect itself from errors in the user-defined functions: if an error occurs during the execution of any such function, control is transferred to a handler that sends an appropriate "error page" to the client.
Most Web browsers provide the user with the ability to view the HTML source of a page. This is feature is very useful for beginners, since it gives them an opportunity to "learn by example" by matching the contents of the page with its HTML source. The use of Lisp to generate the HTML code adds an extra layer of complexity to the task of the developer. The LispWeb server provides a tool to access the Lisp function that generated a given page, for debugging of didactic purposes. When this feature is enabled, the define-page macro adds a special icon to the page; clicking on the icon invokes a function that displays the Lisp code that produced it.
The LispWeb server can be interactively configured while running, using a special-purpose form that allows the server operator to view and set various global parameters. For example, the operator can enable or disable request logging, shut down the server, control memory management, and execute arbitrary Lisp forms (for example, to load patches and additions to the Lisp environment). Access to the configuration form is controlled by a password that is stored in encrypted form by the server.
The main goal of the LispWeb server is to provide access to Common Lisp applications through the WWW. This means that when a user connects to it, he is really interacting with a Lisp system running inside the server, using the Web-browser as the graphical user interface of the program. It is apparent that the interpretation of the URL performed by our server must be different from the usual case. In particular, when a URL is used to reference a Lisp function, its path component must be interpreted as indicating the name of the Lisp function to invoke and the arguments to which the function must be applied. URLs are also used to reference static images, dynamical images created with CL-FLY, forms, image-maps and so on.
LispWeb uses a uniform hierarchical naming scheme, that represents an extension of the file-system model, to associate pathname-like request strings to server objects. The directory components of the request are parsed from left to right, and are used to select the appropriate dispatching function to apply to the rest of the pathname. For example, all requests whose first component is "/image" are assumed as referencing static images (i.e., images stored in a disk file). In this case, the dispatching function simply interprets the rest of the request as the pathname of the image file and retrieves it. In the case of URLs that reference Lisp functions, the dispatching function splits the path into a function name and a set of arguments, and subsequently applies the function to the arguments. The server assumes that the return value of the function is an HTML page, and therefore sends it back to the client browser.
This scheme is totally open and extensible: the developer can define new object classes and new dispatching functions, or override system defaults to customize the behavior of the server.
To conclude, we will show some examples of the use of HALL to define new pages in the server's object space. The following expression defines a page that counts how many times it was accessed.
(define-page ("A page with a counter" :func-name counter-page
:url "/examples/counter")
()
(incf *access-counter*)
(html-out "This page was accessed ~a times." (bold *access-counter*)))
The previous expression defines a function of no arguments called counter-page, associated with the "/examples/counter" URL. The first form in the function body uses the Common Lisp incf function to increment the global variable *access-counter* (that we suppose previously initialized). It does not produce HTML, but it is used for its side-effect on the state of the server. The second one produces a string to be inserted in the HTML page. When invoked, it evaluates the current value of the *access-counter* variable, wraps a <B> ... </B> tag pair around it, and inserts it in the supplied string in place of the "~a" format directive. The behavior of the html-out function is equivalent to that of the general string-formatting Common Lisp function format except that its output is directed to the HTTP stream.
In conclusion, the function created by the above definition sends the following text to the client when invoked (assuming that the previous value of the counter was 97):
The second example shows how the combination of partial URLs and argument passing allows to hold state information. This page displays the value of a numerical variable, and offers the user the chance to increment or decrement it.
(define-page ("A page with an argument" :func-name arg-page
:url "/examples/args")
(&optional (state 0))
(setq state (read-from-string state))
(html-out "The current value of the state is ~a." (bold state))
(html-out "You can ~a it or ~a it."
(anchor (html-build "/examples/args/~d" (1+ state))
"increase")
(anchor (html-build "/examples/args/~d" (1- state))
"decrease")))
The first html-out form merely prints the current value of the state variable. The second one contains two anchors that associate the words "increase" and "decrease" with two URLs obtained by appending the increased or decreased state, respectively, to the URL of the page. By clicking on one of them, the user would invoke the same function with a different argument. Note that in this case the state is not explicitly stored in a variable in the server's environment.
The assumption lying at the base of the new discipline called Distributed Artificial Intelligence (DAI) is that, in general, a single isolated intelligent agent does not possess enough knowledge and resources to complete a problem-solving task. On the contrary, it is envisioned that complex problem-solving in real-world domains will be performed by communities of specialized intelligent agents, able to collaborate with each other towards the accomplishment of a given common goal and to negotiate the use of resources in the most effective way. The ability to communicate with other agents is therefore considered essential for an agent to exhibit intelligent behaviors [5].
Since the LispWeb server is intended to be used as the development environment for distributed AI systems, we have provided it with an extension to the HTTP protocol designed to support server-to-server communication. In particular, we have introduced a new method that enables a client agent to invoke a specific service on a server agent. Note that, in this case, the distinction between client and server agents is no longer fixed: a LispWeb server can act both as a client agent and a server agent at the same time.
The services that a server can offer are described in terms of tasks and actions. An action is an atomic operation that usually corresponds to the invocation of a single function on the server; it can take arguments and it physically coincides with an HTTP connection. A task is a set of actions related to the same application domain.
The general form of an STSP request is the following:
STSP action-descriptor HTTP/1.0 Header-Field-1: Header-1 Header-Field-2: Header-2 ...The action-descriptor is a string that encodes the requested action and its parameters using the usual pathname-like naming scheme:
/task-name/action-name/action-arguments...The main advantage of STSP-based communication is that it allows for a bi-directional exchange of information between the two agents. This makes it possible to use the extended HTTP protocol as the basis for a distributed computing environment, in which communication between intelligent Lisp agents exploits the STSP mechanism for remote action invocation, and the interaction with the user can take advantage of the simplicity and power of the WWW paradigm.
In this paper we have described the LispWeb system, a full-featured HTTP server written in Common Lisp. Our aim in developing it was to bring together the best of two apparently remote worlds: the power of a dynamical, object oriented, high level language such as Common Lisp, and the World Wide Web interface model, based on hypertextual and multimedia capabilities.
We showed how the Common Lisp tools provided by the LispWeb environment allow the developer to make complex Lisp applications available over the WWW, thereby increasing their effectiveness and acceptability. On the other hand, the ability to embed intelligent, knowledge-based services in the WWW will provide a chance to turn it from a mere information retrieval infrastructure into a true Distributed Artificial Intelligence environment. Therefore, we expect this new technology to foster a brand new way of delivering information over the Internet and, possibly, a new way of interacting with computer programs.
LispWeb will be used as the core of a distributed system for chronic patient management, that is currently under development in the Medical Informatics Laboratory of the University of Pavia[6]. LispWeb already supports the distribution of some AI applications on the WWW: an implementation of Game Theory classic example, known as The Prisoner Dilemma, a collaborative Elevator Designer, and Luigi, an intelligent meeting scheduler.