WebMake: Integrating distributed software development in a structure-enhanced Web

Abstract: In this paper, a technique for structuring large amounts of interdependent data is presented. This approach which facilitates graph-based hierarchical structuring and allows for the definition of arbitrary views on graph structures can be applied to a broad range of very different application areas. Based on this concept we implemented a distributed software development environment supporting cooperative work on top of the World Wide Web. In general, the approach is intended to serve as a basis for decentralized efforts to tame the immense and hardly manageable collection of data accessible in the Web.

Keywords: Software development environment; Computer supported cooperative work; World Wide Web; Structuring Approach

1. Introduction

In the following chapter, the initial version of the distributed software development environment is presented. Section 3 introduces our generic graph-based environment for structuring large and complex systems followed by a comparison with related research projects. The paper concludes with a short summary and some implications for possible future development directions of the World Wide Web in general and WebMake in particular.

Since we aimed to make the central sections of this paper as independent as possible, the chapters may be read in different order: Everyone interested in the application WebMake, i.e., its architecture, the way to interact with it and the experiences gained with its implementation on the Web may continue with chapter 2 before proceeding to the sections on related work and the conclusion. All those primarily interested in the basics underlying the approach for structuring large systems and the seamless integration of this approach into the existing Web, are recommended to read chapter 3 first.

The current C++ implementation of WebMake features the possibility to modify software subsystems, e.g. source code or object libraries, as well as software updating, like compilation or linking of code, both remotely, i.e., on different internet sites. WebMake is built up from a set of service programs offering full interoperability with existing WWW clients and servers.

2.1 The Architecture of WebMake

The hyperlinks of the WWW have been extended to Context-sensitive Links (CLinks) giving configurable access to the various typed documents as required by the activities and tools used in the development process. In WebMake, CLinks are used to manipulate and retrieve highly dynamic cluster information needed in the different phases of the software development cycle. They provide a kind of `access polymorphism' in that they return the matching document type to the given request, such as dependency relationships, source code, or object code. The mechanism used to implement them basically consists of a cgi-bin that can process HTTP protocol requests in a flexible way. The queries encapsulated in the protocol messages are evaluated and, if a cluster local to the server processing this request is given as an argument, a function as specified by the given cluster is executed. As also several arguments can be transmitted in this request, each cluster's CLink can react appropriately. If the HTTP command contains no such cluster information (as it is the case if a standard WWW client is used for traversing the context-sensitive link), a default reaction is triggered, which consists currently just of returning a URL to our home page. Since all (modifying) accesses to cluster-specific information are handled by CLinks, they are also the place where concurrency control mechanisms are located: Several users' altering accesses to the same document are coordinated, e.g. by rejecting or delaying later accesses until the earlier ones have committed..

Figure 1: Sample of a collaboration between different sites participating in the WebMake approach to globally distributed software development

As an example for the cooperation between the different components making up WebMake, consider the scenario presented in figure 1: At a WWW client site (either running an unmodified Mosaic capable of handling forms or a new kind of client offering direct manipulation access to typed documents) a user issues the command `Make' aimed at updating a given software component; this request is transmitted via a CLink to the cluster at which its code resides (`www.uni-kl.de'). At first, this link evaluates the context of the query, e.g. for which host architecture the requested executable has to be created (`Sun Sparc') and what the date of the last executable known to the client is. If this information indicates that a new executable has to be created (e.g. some source code modifications took place since then), a recompilation is initiated. First, all dependencies are checked recursively: As WebMake depends upon the WWWLibrary, a request for the most up-to-date version of its code is issued (to `www0.info.cern.ch'). If the CLink at that site finds out that a new version of the library for the given architecture must be built, it may transmit several compile jobs in parallel to locally adjacent hosts of the appropriate architecture (`www1 - www3.info.cern.ch') in order to make use of their computing resources. If not all code can be compiled on the WWW sites known to the library provider (because these hosts are heavily loaded), the mere source instead of the object code can be returned to the requester, i.e., the WebMake cluster. Since the host on which it resides may have the wrong architecture, it delegates the job of creating machine dependent object code to another site known to have the appropriate architecture (in this example `aldebaran.cs.uni-sb.de'). After this site returns the desired object code, it can be linked together and returned to the client that triggered the request in the first place..

Figure 2: The cooperation scheme for server-site induced data transmission is a two part process: registering with the ICLink (black lines - 1., 2., 3.) and server-side initiated communication (grey lines - a., b., c.).

In order to provide server-side induced data transfer and to keep backward compatible to an unmodified XMosaic 2.4, a Command Converter influencing the WWW client-site has been developed. It accepts remote connections (RFC 931-secured) and transmits commands asynchronously to the Mosaic process and possibly to other client-side programs like editors accepting external stimuli (e.g. via the gnuserv extension of Emacs). Unlike the programs listed above, this remote-control (R/C) application is spawned by Mosaic in response to a certain MIME [5] message [BoF94] (preliminarily typed `application/x-spawn-rc') from a server when the user registers with an interactive CLink (ICLink), i.e., a context-sensitive link which actively sends messages to clients. The port at which the R/C program may accept external connections is transmitted to the server, which can then asynchronously send messages to client-side processes, e.g. forcing Mosaic to load a certain document, or to make an editor display a transmitted file, thus fostering interaction between several users of the same cluster. This way, the traversal of an ICLink can induce actions on each registered user's clients. This approach has deliberately been chosen as long as the CCI and DTM interfaces of Mosaic 2.5 [NCSA94] have not been made sufficiently secure. With the approach presented in this section, the external R/C program can easily be reconfigured to only accept connections and remote commands either authenticated with a magic number or coming from certain trusted hosts and/or users.

2.2 Interacting with WebMake

After starting the GUI with a default document already displayed, additional typed documents can be inserted into the shown graph structure (rf. figure 3) by entering their URLs in the input field (1). In an options menu (2), the user is presented a selection of recently entered URLs for repeated instantiation of typed documents. The respective icons (3) and edges (4) representing their relationships, such as dependency links, are drawn in the GUI's graphing area.

Figure 3: Screenshot of WebMake's graphical user interface

By dragging the icon corresponding to a design document onto an Editor icon (6), the user can edit the respective document; in an analogous way, all WWW objects represented as icons can be displayed in a (possibly local) Mosaic client (7) attached via the aforementioned secured remote control mechanism. With this, a rudimentary cooperation facility has been created: As the `Mosaic' icon on the desktop might represent a remote user's WWW client, the local developer can send documents, resp. their URLs, directly to the remote site, e.g. to foster discussion about its contents. Dependencies can be added in an analogous way, namely by dragging the required object's icon onto the dependent document's symbol. The user can now make a selection among several alternative actions presented in a dynamic menu, namely create a source, link-time or documentation dependency; these dependencies are used for example during update operations that can be started by selecting the Make command in the dynamic menu (5) on a cluster. The entry `Expand' in the menu is the only visible hint to another feature of WebMake: As soon as it is selected (and the appropriate command sent to the typed document) all documents somehow connected, e.g. all those to which links appear on the same page, are also displayed in the GUI, thus effectively creating a graphical mapping of the WWW as traversed by the user.

The HTTP protocol underlying the World Wide Web is based on short synchronous data transmission and stateless servers. Although the possibility to asynchronously trigger operations on remote sites is suggested (HTTP return code 202, `Accepted'), no real applications for this scheme have come up. In the context of WebMake, an asynchronous mode of operation is absolutely necessary for a parallel processing of independent compile and link jobs transferred to remote clusters' sites, because the results from these jobs arrive in a completely unpredictable order (by traversing the CLink of the originating cluster). At last, the next stage, e.g. linking, may only start after all jobs of the previous step, i.e., compiling all sources, could be checked off correctly by the cluster initiating the update operation. To this end, we developed a simple state-preserving approach based on the use of shared memory stored within a server's local filesystem accessed by the context-sensitive links. This approach will be enhanced with techniques ensuring fault tolerance like checkpointing in order to be able to recover from possible server crashes. Additionally, server-side induced data transfer is necessary in WebMake to give an immediate visual feedback, for example about the progress of all independent update operations. To achieve this, a connection with a long-running cgi-bin has to be maintained and asynchronously arriving packets must be processed. To this end, we had to develop a specialized protocol on top of HTTP transmitting the packet and packet size of each message for this case. This proprietary approach will be replaced as soon as this feature is integrated into HTTP.

In order to ensure a minimum security level when transferring source and object code back and forth between clients and servers as well as between servers and servers, some thoughts about authentication were in order. As long as no final decision on a security scheme within the WWW has been made (rf. discussion in chapter 4), a simple security mechanism was realized, which identifies the host from which the data has been sent and compares it against a list of trusted hosts; moreover, the user that initiated the data transfer is identified via an (RFC 931) ident server running on her (trusted) host; finally, it is checked if a correct magic cookie, i.e., a large random number, was transmitted, thus realizing a simple capability-based security scheme. This method allowed us to implement a first basic accounting scheme absolutely necessary for the global software development environment we have in mind.

The execution of a cgi-bin when a CLink is traversed, as well as the use of machines running HTTP daemons as compile servers may induce a prohibitively high load on server machines. In our opinion, the following two partial solutions together solve this performance problem: For one, hosts dedicated to developing software could run a simplified httpd just implementing what is necessary to realize this concept, e.g. the ability to spawn cgi-bins; this would be a very simple WWW server, i.e., a so-called gateway intended exclusively for this purpose. Second, threaded httpds which need not spawn a full-fledged process but only a light-weight thread when answering an HTTP request, e.g. like the native VMS httpd (URL=http://kcgl1.eng.ohio-state.edu/www/doc/serverinfo.html) , can also reduce the server load considerably. If no such widely accepted, secure multi-threaded server is available within short time, we intend to develop it based on the CERN httpd and a public domain threads package - a feat made easier by the availability of the WWWLibrary 3.0 (Prerelease 2 announced in December 1994) redesigned to handling multithreaded I/O. Such a server can also drastically reduce the load on any heavily used WWW site. Another advantage of multithreading will be exploited when implementing specialized gateways to cluster sites: By not having to fork a process on every incoming call to a CLink and the determination of the correct cluster-specific and context sensitive operation, the overall responsiveness of WebMake will be drastically improved.

All in all, the general mechanisms of the WWW suffice to support the presented paradigm of a fully distributed, world-wide software repository. The enhancements proposed in the paragraphs above would just make our idea much more portable and efficient.

As a validation test for WebMake, after realizing the basic mechanisms, we put the complete WebMake source and documentation under control of WebMake itself. This equally applies to the on-line documentation about WebMake, which is accessible over the World Wide Web as well.

As an approach to solve these problems, a generic concept for managing large collections of related documents has been developed which we implemented using the mechanisms provided by the World Wide Web for document storage, presentation, and retrieval. The Web is used as underlying medium for the implementation, providing for the scalability and openess needed; on the other hand, as it is a large system itself, it serves as a validation example for our structuring concept..

Figure 4: Structure of the inheritance hierarchy underlying WebMake in Booch notation [Boo94]

The approach facilitates graph-based decomposition for structuring complex systems hierarchically. The basic abstractions introduced to this end are shown in figure 4: General documents in the context of the World Wide Web are represented by Nodes, and hyperlinks on the Web are mapped to typed Edges in our graph model. They specify both the location where a certain document is stored, as well as the type of relationship between the interconnected nodes. Context-Sensitive Links (CLinks) are a special form of edges containing functional properties. They may for example be used to transform data passed through them. Description Nodes are abstractions for any kind of application-specific information, e.g. software development artifacts such as source or object files and documentation, as well as for all other documents in the Web which have no special semantics in the given application field. The Structural Node abstraction extends the flat graph structures, made up of ordinary nodes and edges, to hypergraphs with arbitrary subgraph hierarchies. Two kinds of structural nodes are currently defined: Cluster Nodes contain, i.e., encapsulate other nodes; they provide abstract interfaces to the contained subsystems while hiding their internals. By expanding a Cluster Node, the next level of abstraction within the graph is exposed for traversal and manipulation. Views are the second kind of Structural Nodes; they are arbitrary projections on the overall set of nodes and edges without physically containing them. Customized views can be defined to represent aspects of the system under development which are relevant in a certain context. As cluster nodes and views are nodes themselves, they can of course be grouped by other structural nodes (describing a higher level of abstraction in the graph model). This allows the recursive definition of complex hypergraph structures, which is the technique we use to make the complex structure of globally distributed software projects manageable and comprehensible.

Figure 5: Structural nodes offer and implement a certain functionality. The expansion of a structural node by the user yields its internal structure with the nodes and their relationships making up the next level of abstraction, which can then be accessed and modified. Used by CLinks, the same structures provide - for example in the context of WebMake - the information for the make mechanism. A View as shown in the lower part is made up of projections to certain other nodes at different levels of abstraction (Documents in this example).

As an example, consider figure 5: On one level of abstraction, it may only be known that the structural node 1 depends somehow on node 2; if the representations of these nodes are expanded, the internal structure is exposed yielding the reason for the dependency: Source A depends upon, i.e., imports the definitions of, source B. Moreover, since the component nodes belonging to the respective structural nodes are now not only visible, but also accessible, their internals may now be displayed if the appropriate permissions are given. Another aspect of structural nodes, resp. views, can also be made clear on this figure: It is possible to export some documents to the upper layers by attaching the respective document to the interface of a structural node in order to make it visible externally.

Although a simple security scheme based on a combination of capabilities, RFC 931 authentication and a list of permitted well-known hosts has been realized, further efforts have to be undertaken to make this concept generally agreeable. In this context, the work of Schiffman on S-HTTP [Sch94], Hallam-Baker on Shen [Hal94] and Netscape on SSL [And94] should just be mentioned to stress that a widely accepted security scheme is just down the road.

Unlike Hypercode [DBE+94] which aims at providing a hyperlink-enriched source code repository via a centralized database, our concept uses the mechanisms of the Web to provide a fully distributed and configurable software development environment, especially adressing later development phases such as compiling and linking of software.

Finally, we will discuss the various structuring approaches trying to sort out the huge amount of information currently accessible over the World Wide Web: On the one side, there are tools for the creation and management of per-user views of the data available on the Web, like Simon [JoM95] or WebMap [Döm94]. On the other hand, there are efforts to chart the WWW globally like the one undertaken by ALIWEB [Kos94] or the numerous search-engine/Web-robot sites, like MomSpider [Fie94]. While the latter naturally have the problem that they are not simply customizable for each user's needs, the first lack the ability to keep up-to-date with the developments on the Web, e.g. the creation or relocation of information. In this respect, our ideas for structuring the Web itself combine the strengths of the two approaches while reducing their weaknesses: By giving each user as well as each information provider the opportunity to set up views (possibly traversable via CLinks), each user is given the chance to stay informed about the latest developments in her research area by subscribing to, for example, a view maintaining links to the major research institutions as well as a gateway to the various newsgroups dealing with this topic. If not everybody is content with a given view, a personalized projection presenting less (or more) information can be built without losing touch with the original view that is continuously updated. This mechanism can be compared with the usenet news system: As soon as a new (widely visible) newsgroup is created, everybody interested is informed. In our scenario, everyone who has derived a personal view from a server-side provided view can be notified about changes in this globally accessible view. Since one's personal views are structural nodes and as such documents on the Web, they can be accessed by whoever (with the appropriate permissions) and may thus foster the collaboration of people with the same interests.

The mechanisms developed in the course of this project may also be used in new applications independent from what we initially had in mind: One service that is very interesting in this context is the possibility to use dedicated servers to compile all kinds of source code upon request in parallel. As the client side is completely independent from the graphical front ends like Mosaic, these servers might be triggered by a slightly modified make(1) program thus offering the full capability of WebMake to any non-Web-based software distribution, like the X-Window library.

In order to facilitate a data migration mechanism as useful for example for load distribution between clusters managing source and object code, some problems with the currently too inflexible addressing scheme of the WWW via URLs have to be overcome. New addressing methods, like that of URNs [6] in combination with name servers, have to be considered. In our approach, a first solution to the problem of accessing potentially mobile (source) documents consists in using the cluster information, which is made up of hyperlinks to the -virtually- contained nodes, resp. to their proxies, if the actual nodes have been migrated. In this way, (multiply) relocated documents will be found by recursively traversing the chain of their proxies.

The current progress in multimedia technology as well as the market penetration of affordable multimedia hardware make the integration of tools like on-line audio and video conferencing programs into a distributed software development environment look feasible. The abstractions and concepts introduced can effectively be used when implementing the necessary extensions of this development environment to achieve the closest possible integration of multimedia collaboration technology in our project. We are convinced that other tools useful in the context of distributed software development can be fitted easily into the given framework as well.

This paper should be concluded with a proposal for the future development of the World Wide Web: We think that it would make sense to create an infrastructure, e.g. within the WWWLibrary, for fully asynchronous communication between clients and servers, i.e. server side induced data transfer. This, combined with the future capabilities of clients concerning function shipping, aka agent-based behavior, would result in a further boost to the WWW, its user community as a whole and the applications easily created on and of the Web.

6. Abbreviations

[1]

CGI = Common Gateway Interface

[2]

URL = Uniform resource locator

[3]

HTML = Hypertext Markup Language

[4]

HTTP = Hypertext Transfer Protocol

[5]

MIME = Multipurpose Internet Mail Extensions

[6]

URN = Uniform Resource Name

7. References

[And94]:

M. Andreesen (editor): Internet Data Security: An Overview of Implementation and Plans from Netscape Communications Corporation;draft document announced November 1994

[BDH94]:

C.M. Bowman, P.B. Danzig, D.R. Hardy, U. Manber, M.F. Schwartz: The Harvest Information Discovery and Access System; 2nd International Conference on the World Wide Web, Chicago, October 1994

[BoF94]:

N. Borenstein, N. Freed: MIME (Multipurpose Internet Mail Extensions): Mechanisms for Specifying and Describing the Format of Internet Message Bodies; Internet RFC 1521

[Boo94]:

G. Booch: Object-Oriented Analysis and Design with Applications, second edition; Benjamin/Cummings, 1994

[DBE+94]:

A. DeHon, J. Brown, I. Eslick, L. Karbiner, T.F. Knight, Jr.: Hypercode; 2nd International Conference on the World Wide Web, Chicago, October 1994

[Döm94]:

P. Dömel: WebMap - A Graphical Hypertext Navigation Tool; 2nd International Conference on the World Wide Web, Chicago, October 1994

[Fie94]:

R.T. Fielding: Maintaining Distributed Hypertext Infostructures: Welcome to MOMspider's Web; 1st International Conference on the World Wide Web, Geneva, May 1994

[FLF94]:

T.J. Frivold, R.E. Lang, M.W. Fong: Extending WWW for Synchronous Collaboration; 2nd International Conference on the World Wide Web, Chicago, October 1994

[Hal94]:

P.M. Hallam-Baker: Shen: A Security Scheme for the World Wide Web

[JoM95]:

M.J. Johnson, E.H. Mamdani: Feedback in Internet Resource Discovery Systems: SIMON; submitted for publication in IEE Electronics Letters

[Kos94]:

M. Koster: Aliweb - Archie-like Indexing in the Web; 1st International Conference on the World Wide Web, Geneva, May 1994

[KPT94]:

M.F. Kaashoek, T. Pinckney, J.A. Tauber: Dynamic Documents: Extensibility and Adaptability in the WWW; 2nd International Conference on the World Wide Web, Chicago, October 1994

[LaK94]:

M.G. Lavenant, J.A. Kruper: The Phoenix Project: Distributed Hypermedia Authoring; 1st International Conference on the World Wide Web Geneva, May 1994

[NCSA94]:

National Center for Supercomputing Applications: NCSA Mosaic; URL:http://www.ncsa.uiuc.edu/SDG/Software/Mosaic/NCSAMosaicHome.html.

[NLi94]:

H.F. Nielsen, H.W. Lie: Towards a Uniform Library of Common Code; 1st International Conference on the World Wide Web Geneva, May 1994

[OPW94]:

K.A. Oostendorp, W.F. Punch, R.W. Wiggins: A Tool for Individualizing the Web; 2nd International Conference on the World Wide Web, Chicago, October 1994

[Sch94]:

A. Schiffman: Secure HTTP: Safe Transactions for the World Wide Web; Presentation at the InterOp 1994

[WoR94]:

T.K. Wo, M.J. Rees: A Synchronous Collaboration Tool for World Wide Web; 2nd International Conference on the World Wide Web, Chicago, October 1994

8. About the Authors

• Michael Baentsch is a member of the system software group at the University of Kaiserslautern since he received his diploma in computer science in 1993. His research interests include graphical software development environments, parallel and distributed systems in general and the development of configurable and generic distributed operating systems in particular.

• Georg Molter received his diploma in computer science in 1994; he is currently working at the University of Kaiserslautern system software group. His research interest is focused on the field of generic support mechanisms for distributed systems, especially in the context of the problems and possibilities of the rapidly developing high-speed networking technology.

• Peter Sturm is a member of the system software group at the University of Kaiserslautern since 1987. He received his diploma (1987) and Ph.D. (1993) both in computer science. His current research interests include distributed operating systems and applications, user interface design, and software engineering.