Sankar Virdhagriswaran, Mike Webb, Karen MacArthur,
and Gordon Dakin
Crystaliz, Inc., 9 Pond Lane, Unit 4D-B, Concord, MA 01742, U.S.A
{sv, mjw, kmm, gad}@mail.crystaliz.com
PowWow provides a system for configuration management over the web. Using the framework of the web for naming and sharing data, and using their existing web development tools and environments, developers can create, access, and change web objects between local development environments and remote sites. Over this, PowWow provides a structure for tracking and managing this information, using version control, workspace management, release management, and configuration management. By implementing these systems using metadata over a shared data space distributed across the web, PowWow takes advantage of ongoing web developments in the naming and sharing of data, and of the body of useful tools for developing web objects.
The web has become a worldwide system for sharing information and objects. Its benefits as a distributed, platform-independent network need no further explanation here. As a result, there is a huge development effort taking place around the web sites. Developers need to work in a distributed information space, authoring objects which are shared across many locations, coordinating efforts on jointly-held data, and collaborating on group projects.
Efforts are under way in the web community to develop universal naming schemes and representation of web objects in order to facilitate the sharing of information and the functioning of distributed systems. A wide array of tools provides ever better methods of creating information and web objects. A system to aid developers should not try to reinvent existing web tools or replace the web; rather, it should take advantage of their capabilities and continue to work within the framework of future developments by the web community. The primary need of developers in this environment is the coordination of ongoing distributed development and the management of a heterogeneous shared data space. This is the vision behind our change management system, which works across the web to tie together and manage information and processes from this conglomeration of tools, locations, and platforms, growing as the web grows.
A web site development project is a good representative example of the need for change management. Members of a development team create documents, programs, and graphics, working on a variety of platforms and using a number of tools including word processors, layout tools, and code debuggers. The web site itself is a heterogeneous collection of these items, distributed among many locations and platforms. Over time documents grow through many versions, and objects are rearranged in new configurations. The web site often includes several versions in different stages of release. At any time, a developer may need to get at a particular version of an item, use a tool on their local machine to modify it, and put it into the appropriate place in the shared set of items. Developers need to track changes and versions over time, and they need to be able to map and transfer items between their local tools and the shared base. Furthermore, they likely need management and handling over existing sites, configurations, and development environments.
PowWow addresses this need to manage change, by providing:
All of these function on distributed sites in both a 1 to n and an n to n fashion, i.e., one team on a LAN authoring for multiple web sites or n authoring teams on n different LANs authoring for n web sites. Distributed version control allows multiple team members to edit the same copy of a web object identified by an URL from heterogeneous client machines connected to the control site only with HTTP; no distributed file system is necessary. Furthermore, version control allows users to share binary objects across projects and users. Workspace management allows individuals and projects to create their own private workspaces to support in-place editing. By managing the workspace, PowWow tracks locations on the workspace and maps between current and final locations (via URL), so no changes or reconfiguring is necessary on check-in or release. (Again, PowWow uses URLs to identify the web objects it manages, in order to work within the framework of the web and take seamless advantage of editing tools that support HTTP get and put.) Release management supports creation of named releases on distributed web sites and supports distributed and parallel make of software. Configuration management allows users to create and recreate configurations of versioned files, such as those comprising a web site. Furthermore, configuration management allows web site authors to back-out and freeze named configurations. Task management allows users to implement a process model to manage their site development process.
PowWow was originally developed to address problems arising in ongoing development of web sites which integrate information from many different sources. Several factors make the continuous improvement problem an intractable problem for existing web site authoring, testing, and maintenance teams (in what follows we describe these teams as authoring teams). These teams have certain requirements, which we address in PowWow:
In-place editing refers to the ability to check-out/edit/check-in a web object (HTML file, source code, etc.) from/to a control site from a client machine without changing the URL of the web objects(s). Current systems used for version control and configuration prevent users from taking advantage of the worldwide distributed, easy-to-set-up nature of the web.
A web site is made up of many thousands of objects of many content types. The contents can range from HTML pages to Java programs to sound files, all hyper-linked through the URL. Furthermore, this content is managed using different version control systems. Any continuous improvement system needs to deal with the variety of version managers used by customers and variety of types of web objects managed by the system.
Programmers and documentation specialists typically work in different sets of files based on the task they are working on. These tasks may be grouped into projects and managed. A continuous improvement system needs to provide support for tasks and projects. Tasks provide a view mechanism that users or managers can use to group files into meaningful conceptual entities and can set up constraints on the way the files will be used by the build and/or release process. For example, users can create tasks for development, testing, integration, etc. and can specify different rules on how files that are under these tasks need to be processed during the build and/or release process.
Web sites are authored on multiple platforms using various tools. Usually graphic arts for the web site is prepared using Macintosh platforms, software development to tie in databases through CGI scripts is done on UNIX systems, and documents are prepared using Windows platforms. Therefore, the change management system has to deal with different platforms. Existing approaches to version control systems that use LAN file systems for moving data for check-in/check-out are usually limited to one or at best two platforms. PowWow uses the WWW as the file transfer mechanism. Therefore, it covers all the platforms on which WWW servers and clients execute. This includes Macintosh, Windows, and UNIX platforms.
Usually, material for web site(s) is authored by multiple teams. Since many web sites are integration points for various departments or functions, each of these departments use their own authoring teams to prepare the information for the web site(s). However, the web site has to present a consistent, navigable web that is made up of material supplied by these teams. The problem is similar to the integration and testing step performed in software development. Teams may want to install or stage delivery of information in web sites by swapping different configurations for different uses at different times. PowWow supports such a swapping of configurations and staging the release of sites without having to change the URL of documents.
Building target web sites from information contained in a user's (i.e., client) host and control server host should be supported at different levels of complexity. A prototype build on client host before delivery to the target host is one requirement. An integrated build on a server host before delivery to the target host is another. A partition build on to a logical host which gets bound to a target host or to a cluster of hosts based on their capabilities (e.g., memory size or disk space) before delivery is a final requirement. PowWow supports all of these requirements.
Additionally, rebuilding of a built site is a key requirement. To support this requirement, PowWow maintains a build audit report that can be used for construction at a later time.
Distributed change management systems have been difficult to use. While some of them address ease of use for the administrator through Graphical User Interface (GUI) functionalities, others address ease of use for the authors through virtual file system approaches. For change management of WWW sites, the ease of use problem can be divided up into five broad areas:
Functionalities that address each of the first five in an intuitive way contribute to (6). However, there is a system wide ease of use feature that needs to be addressed also.
A key assumption of the system is that authors use the WWW access and update. As was presented above, authors and project managers establish workspaces to isolate activities. These workspaces in addition to providing a view onto the global web used by different project teams also establish concurrency control policies. Therefore, the view and concurrency control policy mechanism needs to work with the WWW. This problem is similar to the ones faced by configuration management systems that use a virtual file system approach. However, here the analog to the file system is the WWW itself. These configuration management systems allow users to set up the views using cryptic commands typed through shell scripts. This approach is unsuitable for graphic artists who would like to organize their workspaces. A GUI based approach needs to be developed to address this problem.
As was presented earlier, different teams author/update different portions of a distributed web. Therefore, an organization needs to be developed to manage version, configuration, and interaction issues. This organization needs to mirror the tree-like organization that is familiar to the web users (even though what is being created is a graph, tree is the navigation metaphor). A GUI tool needs to address the problem of organizing the tree. Furthermore, in order to support search-like queries and querying for report generation, this tree organization tool should allow the administrator to set up the tree as a database.
In addition to organizing and administering the web, the administrator and users also need to administer information about configurations, users, access control policies, file transaction history, etc. A GUI based tool needs to be provided to support this administration need.
Establishment of the workflow policies and administering them on a distributed basis is a time consuming task. GUI based tools need to be developed to address this issue.
Over the course of using this tool, much historical and administrative information will be collected by the change management system. Furthermore, the change management system will manage many, many files and objects. Therefore, users and administrators would be interested in creating reports on data contained in the system. Additionally, managers would be interested in creating tracking reports. The system should provide GUI based tools to support this need.
Finally, functionalities developed to address the ease of use should appear as an integrated whole. A key problem to be addressed is an overall metaphor of interacting with the system for authors, administrators, and managers.
PowWow supports all of these needs by providing a GUI built as a Java applet. Using this applet, users and administrators can perform all the functions described above from their web browsers.
In addition to these problems introduced by the special nature of the web, additional problems such as:
will be addressed by future versions of PowWow.
Secure access to the change management system for the WWW involves not only distributed access control, but also authentication and encryption. Centralized version management systems provide ways of controlling access to the files, by managing directory/file level access over a LAN. The access control information is usually maintained in a central repository that administrators can modify. On sites distributed over the WWW, issues such as administrative domains and corporate autonomy will prevent access control from being managed by setting file/directory access control. The problem is similar to a subset of the electronic commerce problem where a buyer and seller have to authenticate each other before data can be transmitted or modifications made (for check-in and check-out).
Different types of workflow processes are used by organizations to support collaboration between authoring team members. The proposed solution needs to be flexible enough to support establishment of process control systems. Furthermore, the process control system closely interacts with the functionalities provided for secure access (e.g., roles played by individuals) and concurrency management policies.
Different teams or the same team in different stages of the life cycle of web site development use different concurrency control policies. A team in the authoring stage will typically allow individuals to derive off of checked out versions. On the other hand, a team in the testing stage will allow read-only access for checked-out versions. Therefore, the concurrency control mechanism should be customizable by users and projects.
Furthermore, users or projects in order to isolate themselves from their projects or superior projects will set up their own private workspaces (usually called sandboxes). In these workspaces, they will not only want to create their own view of the controlled information, but also their own concurrency control policies. The proposed solution must address setting up a hierarchy of workspaces and manage the concurrency control policies appropriately. PowWow uses a concurrency control model that uses a sophisticated tree search mechanism to support flexible concurrency control management.
We use many innovative solution approaches to address the above requirements. They are:
The following paragraphs explain in detail our approach.
The first key innovation in our system is the support for different version control systems executing in different platforms. For example, let us say that a development team is developing software to deliver information through web browsers for data stored in relational databases running on Unix servers and NT servers through one HTTP server.
Also, let us say that these are two different teams: one for a Unix server and one for an NT server. Furthermore, let us assume that the Unix server web objects are being controlled by one or more CVS version control system and the NT web objects are being controlled by one or more SourceSafeTM or StarBaseTM version control systems. Figure 1 presents this picture. However, from a web site development perspective, objects managed by these version control systems have to be related into a top-level project. Additionally, from a configuration management perspective, the user should not have to be aware of the different version control systems. PowWow supports this functionality. Users can create folders in a top-level project (called Spindles in PowWow) and specify the folders to be managed by different version control systems. When the user checks in or checks out web objects from these version control systems, PowWow does the appropriate mapping of commands, arguments, etc.
The second innovation consists of using a URL as the identity of a web object instead of its location. Since URL is the identity of a web object, it does not change and our Meta Object Base (MOB) takes care of the mapping as an object is physically moved from a control site to support cooperative authoring to a test site to the final delivery site. Typically, users assign the URL of the final resting-place of the web object - its release site, when they start with the development project. Once the web object(s) is released, viewers can get at that object using normal WWW accesses mechanisms and using the URL assigned when the document was created.
PowWow uses a distributed configuration manager as scaleable as the web itself and unifies the concept of configurations with the personal information space of the author. An administrator or an author can create many configurations that a user may access. A configuration is a named view on the web of documents that an author wants to interact with. Each URL contained in a configuration is mapped onto the real location, pathname, or version number using view functions provided with the system. The view functions are automatically updated when a web object moves from one host to another. For example, when a document is checked out from a control server to a client workstation for editing, an editing tool that supports HTTP GET or PUT can access the checked-out document with the same URL as was used when the document was in the control server. Figure 2 presents this picture.
In order to support delivery management of software and associated web objects (HTML pages, AVI files, etc.) on multiple web sites, assuming only HTTP as the distribution infrastructure, we support two key functionalities:
Distributed and parallel builds are supported through a partition manager which is used to logically specify where the software is to be built -- at the authoring workstation (i.e., client), control server, or some other host. PowWow then moves the appropriate files to the target host and builds the software there. In addition, the partition manager can be told to select a set of hosts from a cluster based on criteria such as available memory, disk space, etc. in those machines.
Bundling and delivery is supported through a build audit facility which creates and manages a build audit report consisting of the normalized locations of web objects identified by URLs, their dependencies, tools used to compile software, etc. This information is stored in the PowWow system for later use. The data contained in the build audit report can be used to recreate the site or recreate the build process in a different host.
Process management is supported in many different ways in PowWow. First, users can tag arbitrary attributes, objects, and values to these attributes and objects to any of the information managed by PowWow. Information managed by PowWow includes files, folders, projects, etc. Second, a post event notification system allows users to create arbitrary notifications. These notifications can be used for administrative purposes (e.g., when a check-in occurs send email to the administrator) or for process purposes (e.g., recompile dependent files when a check-in of a file in a module occurs). Third, tasks that correspond to change requests from customers can be established and managed using state transitions. The state transitions combined with the notifications can be used to verify correctness and to perform integrity checks when a task moves from one state to another. For example, when a task moves from debugging to testing state, a notification can be set up to make sure that the configuration being tested does not contain files from the configuration tagged as development to avoid confusion with copies that may contain debugging information included in them.
In this paper, we described PowWow, a distributed configuration management working over the World Wide Web. We described its unique characteristics and the problems addressed by these characteristics.