Abstract. SOXT is an XML manipulation language based on the core of the XSL processing model, but extending it to handle XSDL-style type hierarchies and polymorphism. In particular, it allows for the equivalent of methods for complexTypes, with dynamic dispatch to the right method, regardless of an element's type. SOXT also seamlessly integrates Java as an embedded procedural language, so navigation around the document is controlled by markup including an XPATH extended to handle polymorphism, while the language within a template is Java, mixed (as in XSL) with XML intended for output. While the Java/XSL combination is explored here in depth, the same approach could be applied to other languages, such as Perl, Python, C#, or C++.
Keywords. XSL, Java, XML, XML Schema, XSDL, legacy integration, Web Services, polymorphism, object-oriented programming, SOX.
Word Count. approx. 4900, including markup.
Over the past few years, XSLT [8] has become an extremely important, widespread technology for building XML applications. However, XSL has two unresolved issues we wish to address:
At the same time, XSL is ideally suited for evolution. At the core of XSL is a very simple mechanism for iterating over a document and dispatching based on predicates. The rest of XSL hangs on this skeleton. The issues above are addressable by extending this core. In fact, as we will see, this core can be repurposed to add XSL functionality to a variety of languages.
Our investigations in extending this core mechanism started with an attempt to extend XSL for SOX [9], the schema language for XML that introduced the object-oriented features inherited by XSDL. In the course of this we developed SOXT, an almost seemless blend of the capabilities of XSL and the Java programming language in a single scripting language.
We believe the results are significant both for the future evolution of Web Services [3], where a deeper integration between manipulating the incoming XML document and the Java code of the Web Service itself (or other implementation language) can simplify implementation, and the future of XSL itself, where some of our work in exploiting type hierarchies (as are now in XSDL) in an XSL-style language can point the way to incorporating the same features in XSL itself.
If we look at the core of XSL, we find a mechanism with the following elements:
There are a variety of ways to build on these basic mechanisms - the particular way XSL does only represents one such extension. For example, in an XSL stylesheet the set of predicates can only be changed by explicitly changing modes. We'll show a different way in which the set of active predicates shifts over time, but without the same need for explicit user intervention.
The abstract iterator engine takes a list of elements to process and a list of templates. A "run" of the engine goes something like this:
Processing a document starts by calling the iterator with the root element and an initial set of templates.
Given this core behavior, it is possible to branch out in many directions, one of them being the XSL we currently know. However, that is far from the only one. The order of iteration could be affected, the set of predicates matched at any point could be determined in a variety of ways, the interface between the code and the iterator could be built in different ways, and, of course, the code itself could be in any of a variety of languages other than XSL. SOXT changes every one of these except the order of iteration - an implicit in-order traversal seems to be unquestionably the most natural and intuitive to users.
SOXT had two goals:
In the next section we will discuss the structure of SOXT as a language, including extensions to XPATH and the interplay of XSL-style markup and Java source code within a single SOXT file. Then we'll discuss the Java runtime underlying SOXT.
The goal of SOXT, then, was to use to extract this core mechanism from XSL and rebuild it around Java as the core language for template bodies and SOX as the XML Schema language. This meant we wanted to both exploit SOX's contributions to XML, such as:
as well as Java's language characteristics, such as:
in interesting and compelling ways. Of course, Perl, Python, or C++ would have been legitimate alternative languages (C#, of course, did not yet exist), and we'd be very interested in seeing similar efforts for those (and other) languages.
As a generalization of XSL, one can view SOXT as being in the same relationship to XSL as Embedded SQL is to SQL.
To cut to the chase, the following fragment shows a SOXT script showing many of the innovative aspects of SOXT. The script itself is an XML document containing a mix of markup and Java code. The biggest concession to XML from the Java side is the need to use the built-in character entities (&, and <) for "&" and "<". To explain how the script behaves, we will first briefly describe our extension to XPATH to accomodate types. We will then explain the various elements SOXT, after which we'll describe the Java runtime underlying it.
<soxt:transform
xmlns:soxt="urn:x-commerceone:document:com:commerceone:xdk:soxt:Stl.sox$1.0"
xmlns:html="..."
package = "com.commerceone.xdk.soxt" class = "MkHtml">
<soxt:imports>import java.util.*;
import java.sql.*;
</soxt:imports>
<soxt:namespace prefix = "po" ns = "http://www.example.com/PO.sox"/>
<soxt:method name = "doAddr" type = "po:Address">{
<html:li>{
<html:pre>
<soxt:text match = "po:organization/text()"/>"\n"
<soxt:text match = "po:street/text()"/>"\n"
<soxt:text match = "po:locality/text()"/>", "
<soxt:text match = "po:department/text()"/>" "
<soxt:text match = "po:postalCode/text()"/>
</html:pre>
}</html:li>
}</soxt:method>
<soxt:method name = "doAddr" type = "po:UKAddress">{
<html:li>{
<html:pre>
<soxt:text match = "po:organization/text()"/>"\n "
<soxt:text match = "po:street/text()"/>"\n"
<soxt:text match = "po:locality/text()"/>" GB "
<soxt:text match = "po:postalCode/text()"/>
</html:pre>
}</html:li>
}</soxt:method>
<soxt:template match = "root">{
... jdbc startup code ...
<html:html>{
<html:title>"Purchase Order"</html:title>
<html:body>{
<soxt:template match = "text()"/>
<soxt:template match = "po:shipto">{
<html:h2>"Shipping Addresses"</html:h2>
<html:ul>{
<soxt:template match = "^po:BaseAddress" method = "doAddr"/>
<soxt:apply-templates/>
}</html:ul>
}</soxt:template>
<soxt:template match = "po:lineItems">{
<html:h2>"Items"</html:h2>
<html:ul>{
<soxt:template match = "^po:LineItem">{
<soxt:members>
private void findItem(String ivalue) throws Exception {
string statement = "get name from catalog where partno = " + ivalue;
ResultSet nameSet = catalog.executeQuery(statement);
while (nameSet.next()){
return value.getString("NAME");
}
return "";
}
private String makeTotal(String amount,String price) throws Exception {
double p = Double.valueOf(price).doubleValue();
double a = Double.valueOf(amount).doubleValue();
final double total = p * a;
<html:em>Total: </html:em>
<soxt:text>Double.toString(total)</soxt:text>
return "";
}
</soxt:members>
final String partNumber = <soxt:text match="po:partNumber/text()"/>
<html:li>
<em>"Part: "</em> partNumber
<em>" Name: "</em> findItem(partNumber)
<em>" Amount: "</em> <soxt:text match="po:requested/text()"/>
<em>" Price: "</em> <soxt:text match="po:price/text()"/>
makeTotal(<soxt:text match="po:requested/text()"/>
<soxt:text match="po:price/text()"/>)
</html:li>
}</soxt:template>
<soxt:apply-templates/>
}</html:ul>
}</soxt:template>
<soxt:apply-templates/>
}</html:body>
}</html:html>
}</soxt:template>
</soxt:transform>
The key point to extending XPATH is to provide a mechanism for distinguishing between an element's name and its type. Since XPATH matches against an element's name by default, we need a mechanism to establish when we want to indicate it's type. We do this by adding a circumflex (^) before the name to indicate a type name. For example, in an XPATH expression, "po:Address" will match any Address element in the po namespace, while, "^po:Address" will match any element of type Address, including (in our schema) UKAdress. "po:shipTo/^po:Address" will match any element of type Address (or subtype of Address) within an element with name "shipTo". Since SOX does not have XSDL's distinction between elements and complex types at the top level,n the top-level Address element is also of type Address. Both "po:Address" and "^po:Address" will match the same top-level element. In XSDL, there need be no relationship between an element and its declared type, so "po:Address" and "^po:Address" might match different elements entirely.
The root element of a script is <transform>. All SOXT elements are in the Stl.sox namespace. Because the script will eventually be translated to Java, it has two attributes indicating the package and class for the output.
Immediately within the root element is an optional <imports> element. This contains a list of Java import statements corresponding to the output statements required for the output Java code. The <namespace> elements following that provide prefixes for the namespaces to be accessed by the XPATH expressions. These can be dropped in favor of better access to the current namespace prefix map.
The next element, <method>, begins SOXT's first innovation. A method is similar to an ordinary method in an object oriented language. A method has a name and an associated type. Methods, like methods, are called by name, but dispatch is done polymorphically, according to the inheritance hierarchy of the current element's type.
For example, we have two methods here, both called doAddr. One is for type Address and the other for UKAddress. In the schema, USAddress and UKAddress both extend Address. Now suppose (as we will see later) we have a match expression that matches against the Address type and executes the "doAddr" method. This expression will also match any subtype of Address, such as USAddress and UKAddress. It must then execute the appropriate action. This is done by finding the most specific "doAddr" method. In the case of UKAddress, there is a method defined for it, so it is the body of that method which would be executed. For USAddress, however, there is no method defined. The runtime would then look for a method of the appropriate name defined for its parent, Address. Therefore it would execute the same action as for Address. This polymorphic dispatch does not exist in XSL and goes beyond matching by type.
The fragments below show how the different methods would handle a USAddress vs. a UKAddress:
Input: <USAddress> <organization>Acme Inc.</organization> <street>111 First St.</street> <locality>New York</locality> <department>New York</department> <postalCode>12345</postalCode> <phone>456</phone> </USAddress> Output: <li><pre>Acme Inc. 111 First St. New York, New York 12345</pre></li>
Input: <UKAddress> <organization>Acme Inc.</organization> <street>111 First St.</street> <locality>London</locality> <postalCode>12345</postalCode> </UKAddress> Output: <li><pre>Acme Inc. 111 First St. London GB 12345</li></pre>
At this point, we finally start seeing templates comparable to XSL templates. SOXT templates, along the lines of [2], nest lexically. The body of a template may introduce one or more other templates that are active uniquely within the body of that template. In this script, for example, there is only one top level template - the one matching the root element.
A SOXT template consists of the following:
An example of the use of the method attribute is found in the template for the <shipTo> element. In our toy example, all Addresses show up within shipTo, so the template to match Addresses (which dispatches to the doAddr method) appears inside shipTo.
This method call demonstrates the importance of having some mechanism, such as our methods, for polymorphic dispatch in the presence of types. Because of the type extension and modularity mechanisms in XSDL, copied from SOX, it is not possible for the author of a style sheet to know all the subtypes which may appear in an instance. Therefore it is impossible to:
For the latter issue, we can exploit our expanded XPATH to match unknown subtypes. However, having managed to match all possible subtypes, it is unlikely that the same template action is appropriate for elements of all these types - one would not expect UKAddress to appear just like a USAddress. Therefore it is essential to provide a mechanism to implicitly find the appropriate code for the type. This is provided by our methods. Methods go beyond just finding the code matching the type and look up the inheritance hierarchy for a most specific method. This frees the developer from needing to create a specific template for every use of every type, but allows for dynamic reuse of existing code when it is sufficient for subtypes.
If we had Addresses showing up elsewhere, they could be matched to completely local templates wherever that occurred. Aside from personal preference, this approach clearly enables greater encapsulation and will work well with a more extensive modularity mechanism. In future developments, we'd like to provide a dynamic loading mechanism for methods similar to the dynamic loading provided by modern languages like Java.
The template for matching the LineItem type provides a good example of the SOXT's marriage of XML and Java. In it, we see Java code and XML markup freely mixed, with both SOXT elements to structure the script and HTML elements to be reflected in the output. The first thing to note (and an indication of the heavy use of anonymous inner classes in the Java translation) is the <soxt:members>element. This allows for the declaration of methods and members to be accessible from the body of the template. In this case we have two methods, findItem and makeTotal. The first method takes the partNumber, queries a database, and return the part name. The second method takes the amount and price and adds the total to the output. Note that the value of the partNumber element is assigned to a Java variable. The extensive use of final variables comes from Java restrictions on the use of variables in nested inner classes. The following shows the execution of this template on a single line item:
Input: <LineItem> <partNumber>8</partNumber> <requested>3</requested> <price>5.60</price> </LineItem> Output: <li><em>Part: </em>8<em> Name: </em>socks<em> Amount: </em>3 <em> Price: </em>5.60 <em> Total: </em>16.80</li>
The body of the template calls these methods to generate little chunks of HTML to appear in the output. In the generated Java code, the context in the output document is carried around so the fragments of HTML created in the method calls appear in the right place in the output.
The requirement to support inlined Java code meant SOXT would need to spend some part of its life as a Java source code file. That source code could then be compiled to a Java .class file (the Java version of an executable) which is, in turn, actually executed at run time. In the first version of SOXT, one programmed entirely in Java - there was no XML at all. The current version translates the XML script into Java code.
The difficulty of programming in Java was the amount of boilerplate code required for communicating with the runtime, in particular for creating the inline XML elements. For example, each HTML element added to the document requires the creation of a new object, the managing of the various elements in scope (there's the current element as well as its parents), balancing the open and close parts, etc. There's also a lot of boilerplate code associated with creating the templates, each of which contains at least one inner class extending from the base template action class. In other words, there's extensive management required to make sure the XSL part of the application works. As a result we chose to rewrite the system as an XML application, with the XML oriented parts of the system in markup, and the Java oriented parts of the system in Java, as in the script above.
The current translation to Java makes extensive use of inner classes and nested lexical scoping. Some adjustments have needed to be made to support different Java compilers, as support for various obscure parts of the Java language specification, not to mention compiler bugs, vary. Basically, for every template and every output XML element appearing in the script, the associated Java code is wrapped up and placed in a few methods (described below) of an anonymous inner class. These all nest one within each other, so inner actions are capable of referencing members and methods in outer elements of the script (implemented as enclosing anonymous classes), as well as local variables. Each template consists of a predicate and an object from one of these anonymous classes representing the associated action. (This power is tempered by Java's insistence that such members and variables be declared final. In Java this indicates that the value of the member may be initialized once but may never be changed.)
The base class for actions has two methods of interest. One, startup, is called when the template itself comes into scope. A template comes into scope when the immediately surrounding template is active and that template creates its nested templates (including the one we are concerned with). The startup method is a way for the template to execute any initialization code, and is is constructed from the body of the <init> element of the source code. An example of such code might be the beginning and end of an SQL statement whose column names might come from the individual elements matched by the template (or by its children). There is also an execute method that is called every time the template matches an element. The body of the execute method is generated from the rest of the body of the template (i.e., excluding the <init> and <members> elements). The parameters to these methods keep track of the template's environment, particularly with regards to the current input element and the current output element.
The code to create output XML elements also uses inner classes. In this case there is a single method to execute the code that appears between the start and end tags of the element. The constructor for each of these classes is given the name of the element and a pointer to the parent element and it executes its own code. This allows us to exploit the Java runtime's own stack to ensure that everything is pushed and popped in the right manner. However it requires that start and end tags in the script be balanced and always in the same scope. The may turn out to be too onerous a restriction and eventually require a more complex mechanism, but we've not found this to be burdensome yet.
The other major constituent is the iteration mechanism. The iterator takes a list of elements and a structure of nested templates. The latter is the more interesting. As noted, each template action can introduce its own set of new templates. When an action is entered, it is passed the current set of templates. It creates a new structure with the current structure as parent and adds new templates as required, similar to the way environments are created in languages allowing nested scopes.
The iterator takes its list of elements and matches them one after another against the templates. This starts with the innermost set of predicates and works its way out towards the global predicates, looking for the first match. A more sophisticated arbitration mechanism, such as found in XSL, may be substituted once there's enough experience to determine what the best mechanism would be.
When a template executes <apply-templates>, there is a recursive call to the iterator, so it then walks over a different set of elements with a new template structure. Because the calls are recursive, pushing and popping lists of elements and templates is automatic.
We have already compared the functionality of SOXT to to XSL. Similar systems of interest include the Schema Adjunct Framework [1] (SAF), XDuce [4], and XQuery [6].
SAF is a very simple, elegant, mechanism for attaching processing semantics to document elements. A Schema Adjunct is a separate XML file allowing, for each input element or attribute, arbitrary information to be passed to an application. This may include fragments of code, such as Javascript, or the names of Java classes, to be executed for those particular items. It is probably possible to build a system of the sophistication of SOXT on SAF, but this would require sigificant effort. However, there are a number of ways to implement SOXT, and the SAF could be an interesting skeleton on which to hang the Java version of the code. In its current version, however, SAF does not support enough XPath for our purposes.
XDuce is a complete language for manipulating XML based on modern type-inference mechanisms.
XQuery is another powerful language for manipulatig XML developed by the W3C Query Working Group. XQuery has an extensive array of XML manipulation facilities
SOXT (and XSL) distinguish themselves from XDuce and XQuery by being entirely data-driven. An XSL-style language sets up the possible actions and then allows the input document itself to drive which actions are executed. For languages like XDuce or XQuery, the program determines which parts of the input are to be examined and what is to be done with them. These are two different styles, and over time the XML community will have the experience to determine which applications fit one style over the other. SOXT is particularly adept at integration of XML processing with a non-XML environment increasingly written in Java and support of inheritance hierarchies among the XML types - neither of which is directly supported by XDuce or XQuery.
SOXT represents a new language exploiting the core "style" of XSL while integrating support for the object-oriented features of XSDL and Java. On the one hand, it indicates some directions for the further evolution of XSL. On the other hand, it can lead the way to integrating the data-driven style of XSL into other languages. What SOXT does for Java can just as easily be done for C#, Perl, Python, C++, or any other programming language. On both accounts, SOXT can be seen as a precursor of the future of XML processing. SOXT is also ideal for data-driven Web Services requiring access to legacy systems or other programming libraries best accessed through Java.
Future work in this area would include embedding XSL functionality in other programming languages. There are a number of other areas for investigation, such as support for dynamic as well as lexical scoping for templates, and breaking the current symmetry requirements for introducing XML output elements into the script.
The work described here was done while the author was employed at Commerce One.