Providing Data on the Web: From Examples to Programs

The World-Wide Web is based on the HyperText Transfer Protocol, HTTP, and the HyperText Markup Language, HTML. HTTP is a generic object-oriented stateless protocol to transmit information between servers and clients [Berners-Lee 1992]. HTML is a simple, yet powerful platform-independent document language [Berners-Lee and Connolly 1993].

When the documents to be published are dynamic, like those resulting from queries to databases, the hypertext needs to be generated. For this purpose, there are scripts, which are programs that perform conversions from different data formats into HTML on-the-fly. Even tough for fixed data formats these scripts may be simple, providers need them to be able to publish their data on the Web. Furthermore, even basic changes to the data formats or the generated HTML, imply changes to these scripts.

To overcome these problems, Zelig generates scripts that base their HTML generation on schemata [Varela 1994]. In this research we extended Zelig, to additionally produce code for performing conversions from fixed data formats into HTML. There are two main stages in this conversion process: extracting database, record and field information from your traditional database report; and instantiating that categorized information along with the query information into a particular schema.

Using Zelig to provide access to dynamic information in a fixed format, providers only need to give examples of their current text reports and the desired hypertext to be generated for those particular examples. Our system, Zelig, produces the program to extract relevant data from the reports and the schemata to drive the hypertext generation process. Thus, it becomes easier to provide effective user interfaces to dynamic information in the World-Wide Web.

In the next section, we elaborate more on the server-client model used by the World-Wide Web, and the functionality of scripts. In section 3, we highlight the problems faced by providing WWW access to dynamic data. In section 4, we explain the architecture of Zelig, our system that performs schema-based HTML generation. In section 5, we demonstrate the ideas presented with a gateway to the CCSO ph/qi nameserver databases. Finally, in section 6, we give some conclusions and results.

Fig 2.1. Server-Client Architecture [Berners-Lee and Cailliau 1992].

This communication is performed under the stateless HTTP protocol. In a stateless protocol, connections are created, processed and closed without keeping state information. The server actions depend on predefined methods such as GET, POST, PUT, and DELETE.

The resulting information can be served in different format types and it is the client's responsibility to present it in a consistent and clear manner. The most common format is HTML, which contains information, and its logical structure; but leaves out those details particular to specific browser implementations.

It is important to note that a server can provide static documents to the clients, but it could also provide transparent access to databases or other information sources. In other words, the clients can also request for specific queries that should be processed by the server. Scripts or gateways take care of this processing. These are programs that communicate with the WWW server software under a predefined interface. The most common currently used interface is the NCSA's Common Gateway Interface, CGI [McCool 1993].

Fig 2.1. Purpose of a script and a WWW server [Berners-Lee and Cailliau 1992].

These scripts are written in any programming language (like C, C++ or PERL) and their main functions together with the WWW server are:

• To receive the information from the WWW client under the hypertext transfer protocol, HTTP.
• To perform a query for the database server, allowing the WWW server to act as a database client.
• To parse the database server results.
• To generate an HTML document and send it to the WWW client.

• To provide access to many information sources that are currently using non-hypertext formats. There are many fixed data formats for which we would need to create different scripts. For example: phone information, bibliographic databases (BibTeX), papers (LaTeX), electronic mail, server administration usage statistics (logs), UNIX manual pages, file directories.
• To more easily design hypertext interfaces to databases by making changes at the level of schemata, as opposed to modifying, recompiling and retesting scripts.
• To create evolving user interfaces, by instantiating schemata to most common accessed fields for queries, and changing the order or level of the different user interface actions.
• To increase the functionality of the data management system. For our phone example, in section 5, we have included sorting records, which was not in the original ph/qi functionality.
• To reuse the schemata across different databases to provide similar look-and-feels for users.

Fig 4.1. A General Framework for Zelig.

In section 4.1, we see how the HTML instantiation process takes place. In section 4.2, we see how to extract Query Info from the HTML Query Form, how to extract Categorized Output from the traditional DB Report, and how to abstract a schema from user-given HTML Examples.

In this section, we will explain the information that is provided by the schemata to the scripts for document generation. Then, we will give a description of ZHTML, the language to write these schemata, which is an enhancement to HTML incorporating directives for database interface generation.

• Current database transaction results. These are the results generated by a query to a database. We will see in section 4.2 how to interact with the database server, parse its results, and standardize the query information to the following taxonomy:
  • On an application level.
    The most general database information. For example: name, number of tables, default query table.
  • On an object level.
    The information for a specific object of our application. For example the table PEOPLE may have: name, number of records, number of fields, default field for queries.
  • On a field level.
    The most specific information about a query for a given record. For example: field names, default field values, current field values, length, type, and access policy.

• Existing ZHTML schemata. These are generated either from HTML report examples given by the interface designer, or taken from a library. Some library examples are:
  • ISINDEX-based user interface schema.
    This schema is based on the HTML <ISINDEX> tag. It is important to note that there are many indexed databases in the Web that use WAIS as the basis for database search.
  • Forms-based user interface schema.
    This schema makes use of the forms-based WWW browsers to provide a more friendly user interface with menus and widgets to perform the most common database operations.
  • Application-specific user interface schema.
    This schema is usually an advanced interface tailored to the specific needs of a database user interface. It still allows evolution, in the sense that certain constructs don't imply any order or level of access in the generated HTML documents.

The ZHTML comments are similar to HTML constructs. They are generally of the form:

<!ZTAG> ZHTML body <!/ZTAG>

There are several Zelig directives with different functionality, including: print a variable value, run an external function printing its output, conditionally include the ZHTML body, traverse all the current database records invoking Zelig recursively on the ZHTML body and traverse all the fields in a specific object.

Following are the main constructs of this Document Type, even though a formal Document Type Definition (like the DTD shown for HTML in [Berners-Lee & Connolly 1993]) is still in progress:

• <!ZPRINT variable>
  returns the current value of an HTML form variable, or an application-level variable. If the variable is object-level or field-level, then it needs to be in the scope of a ZFOR tag.
• <!ZRUN external_fn>
  runs the script function external_fn and returns its output.
• <!ZIF cond-expr> ZHTML body <!/ZIF>
  returns the output from ZHTML body if cond-expr is true. It returns the null string otherwise.
• <!ZFOR TYPE=traversal_type> ZHTML body <!/ZFOR>
  traverses all the tables, records or fields of the current query, depending on the traversal_type (which can be the value TABLES, RECORDS, or FIELDS) and returns the output from ZHTML body instantiated to each of the loop elements in the query.

In a file listing example, the first line has application-level information, the total space occupied by the directory. Then, we see records (files) that in turn can be divided into fields (name, size, owner, date...) What we do, is to guess where these separators lie and confirm them with the user, prompting her for any unknown information. Then, we proceed to generate the data structure, necessary to instantiate the ZHTML file once new queries get requested.

Note that the query info described above, can contain information that will not be processed by the DBMS, but instead it is functionality provided additionally by Zelig, such as sorting by a specific field.

Here in this subsection, we work on generating the database query from the form variable bindings and the given query example.

In our examples, we mainly have to create:

• % ls <directory>/<mask> or
• % ph <name> return <fields>

We do this by querying the user when we aren't sure if the information parsed is relevant (needs to be categorized to subsequently be used by the schema instantiation algorithm) or it is just a separator.

In the following section, we show an example illustrating a schema, and a couple of its possible instantiations depending on the database query.

The following is an HTML Query Form to access those databases:

The following link contains a schema for instantiating the categorized database information, once a query has been made. We will show two different instantiations depending on two different user queries for this same schema:

• The first query example asks for the names and phone numbers of all UIUC entries with the name Hardin sorted ascendently. Our schema instantiation for such a query results in the following HTML file, which in turn, is displayed by the browser as the hypertext shown in the next figure.

  Each of the records is a hyperlink to more information on that particular person. Also, links previously visited have a different color.

• Our second query example asks for the same entries, but it also queries their address. Additionally the sort order is descendent, as opposed to ascendent. Following the links, you can see the HTML file generated and its corresponding hypertext output.

The functionality for our system, Zelig, was described in this paper. Its main improvements over previous technology include:

• providing code generation for converting fixed data formats into hypertext;

• allowing evolving user interfaces for more effective human-computer interaction;

• increasing the functionality of applications owning the data, by offering additional operations such as sorting; and

• reusing HTML schemata to provide similar look-and-feel interfaces across different applications.

We provided a hyperlinked example giving WWW access to the CCSO ph/qi nameserver software. This gateway running at NCSA, as of September 1994, provides phone directory information for about 250 academic institutions around the world, and receives more than a thousand queries per day.

Ultimately, Zelig offers the user an effective way to generate fully customized interfaces to dynamic data, further closing the gap between information generation, provision and use.

Carlos has also been a research assistant at the National Center for Supercomputing Applications (NCSA) since 1991. At NCSA he has worked in different projects including an alpha shapes visualizer (NCSA Walvis), a World-Wide Web browser ( NCSA Mosaic for X/Windows), and a World-Wide Web server (NCSA httpd for Unix).

In the past, Carlos has been a Math and Computer Science teaching assistant for classes up to differential equations and information systems at the University of Los Andes, Bogota, Colombia. He has also been a consultant for Arthur Andersen & Co., and an Artificial Intelligence fellow at the Beckman Institute for the Advancement of Science and Technology.

Currently she is an assistant professor at the Department of Computer Science and at the Beckman Institute of the University of Illinois at Urbana-Champaign.

Her research interests include artificial intelligence, specially planning, design, abstraction, and knowledge-based systems; as well as computer-aided manufacturing and design. Professor Hayes is particularly interested in tools evaluating, criticizing and optimizing designs in areas from machined parts, intersection design, roofing design and software design.