Relaxed—on the Way Towards True Validation of Compound Documents

The Web is a pervasive platform for sharing various kinds of documents and for providing remote access to applications. Since its birth in the beginning of 90s, Web has been developed as a heterogeneous system that should connect agents (browsers) and servers developed by different software vendors. In order to maintain interoperability, the Web was built around a set of standardized protocols and data formats which together define the Web environment. By adhering to standards, authors of Web pages should be assured that their pages will display correctly in browsers, because browsers should correctly implement support for Web standards. The situation here is not yet perfect—there are plenty of non-conforming pages available on the Web and at the same time not all features of the Web standards are correctly supported by all browsers.

But certainly there was a big movement for Web standards during the last years. However, the will of authors to adhere to Web standards is usually not sufficient—standards are complex and there are just few people around the world who know all details of (X)HTML language from a top of their head. The rest of the Web authors should check their pages using some sort of an automatic validation or a conformance testing tool. Several such tools exist, the most known is the W3C Markup Validation Service.

The problem of the W3C validator (and of many other similar services) is its sole dependency on a formal validation against DTD (Document Type Definition). But HTML and XHTML are not defined only in terms of DTD. Respective specifications [10][2] contain a lot of constraints which can not be expressed using DTD. This means that if some document passes validation against DTD it is not necessarily conforming to the (X)HTML standard.

To overcome limitations of DTD based validation we propose usage of more powerful schema languages that can cover more constraints than DTD. From the (X)HTML validation point of view the following features are the most missing ones:

In this paper we describe how can modern schema languages be utilized for this task (Section 2). Then we describe how we have reformulated XHTML constraints using RELAX NG and Schematron (Section 3). The Section 4 describes Relaxed validator that uses previously mentioned techniques to provide better validation service to Web content developers. In the last part of the article we outline directions for future development of our validation approach and the Relaxed application.

Shortly after launching the XML 1.0 standard it became apparent that DTDs are lacking several critical features needed in many XML applications. The two most important and missing features were support for data types and namespaces. DTD does not have the concept of data types. Every element or attribute value is considered to be almost an arbitrary string. It is not possible to define content to look like a number, a date or a string with a given length.

Namespaces allow to combine several XML vocabularies in a single XML document. The resulting document is often called a compound document. Compound documents promote information reuse and encourage to invent new, previously unknown ways to process information stored inside documents.

Several new schema languages were created to overcome DTD limitations. Many of them were just prototypes or proprietary ones. Only two new schema languages get broader acceptance—W3C XML Schema [11] and RELAX NG [6]. Both of those languages have very good support for data typing and namespaces. At the same time there are also big differences between those languages. Formal comparison of DTD, W3C XML Schema and RELAX NG can be found in [9]. To summarize briefly, RELAX NG is the most expressive language and offers the greatest flexibility in modularizing and combining schemas. This is the reason why RELAX NG is very popular for creating complex document oriented schemas like TEI or DocBook. W3C XML Schemas are enforcing unambiguity of schema^[1] and thus they are very popular in scenarios where unambiguous mapping from XML to object or database representation is required.

All previously mentioned schema languages are so-called grammar based languages. They define grammar of the XML vocabulary by enumerating all elements and their content models. However, this approach is not sufficient in all situations. Some more complex constraints and relations between values in a document can not be captured in a grammar based approach. This problem can be solved by using rule based schema languages like Schematron. Schematron schema consists of a set of XPath expressions that are evaluated against the validated document.

Each schema language is good only in constraining of some document facets. In order to gain better validation results it is reasonable to combine several schema languages and validate document against all of them. Combination of Schematron with RELAX NG or W3C XML Schema is an example of such powerful constraint language. Moreover extensibility of both W3C XML Schema and RELAX NG allows to embed Schematron rules directly into principal, grammar based schema.

To prove the power of RELAX NG and Schematron for Web documents validation we created a validation application called Relaxed which utilizes RELAX NG and Schematron schemas for XHTML.

Relaxed is an open source project which aims to help authors of HTML documents to achieve better international standard compliance using an automated validation service. Modern state-of-the-art XML technologies allow Relaxed to provide better and more detailed validation results than other similar services including W3C validator. The project consists of two essential areas. The first is an HTML schema written in RELAX NG with embedded Schematron patterns which is further discussed in this section. The second but nevertheless important area is a validation engine which is discussed in Section 4.

The W3C's HTML specifications usually consist of two different parts. The first part is the specification text which is basically a set of restrictions, recommendations and explanations provided in a verbal form. The second part is a DTD which is another source of restrictions written in a formalized form. To write standard compliant documents authors need to follow both parts, but only the DTD based part may be validated automatically. This means that authors still need to be familiar with the verbal specification, but most of them aren't. This leads to a huge amount of non-standard documents in today's Web environment which causes many interoperability problems.

The aim of Relaxed is to express most of the verbally formulated restrictions in a form that can be automatically validated to reduce the authors' knowledge requirements and to achieve a better HTML specification compliance. To reach this goal, Relaxed uses one of today's most expressive combination of schema languages, RELAX NG and Schematron. The biggest advantage over other similar approaches (including W3C XML Schema combined with Schematron) is aside from expressivity also some sort of elegance of use, possibility to easily integrate both languages and a great support for modularity.

<define name="Inline.class" combine="choice">
  <ref name="a"/>
</define>

<define name="otherNamespaceElement">
  <element>
    <anyName>
      <except>
        <nsName ns="http://www.w3.org/1999/xhtml"/>
      </except>
    </anyName>
    <zeroOrMore>
      <choice>
        <attribute>
          <anyName>
            <except>
              <nsName 
                ns="http://www.w3.org/1999/xhtml"/>
            </except>
          </anyName>
        </attribute>
        <text/>
        <ref name="otherNamespaceElement"/>
      </choice>
    </zeroOrMore>
  </element>
</define>

<element 
    xmlns="http://relaxng.org/ns/structure/1.0">
  <anyName>
    <except>
      <choice>
        <nsName 
          ns="http://www.w3.org/1999/xhtml"/>
        <nsName 
          ns="http://www.w3.org/1998/Math/MathML"/>
      </choice>
    </except>
  </anyName>
  <zeroOrMore>
    <choice>
      <attribute>
        <anyName>
          <except>
            <choice>
              <nsName 
                ns="http://www.w3.org/1999/xhtml"/>
              <nsName 
                ns="http://www.w3.org/1998/
                                     Math/MathML"/>
            </choice>
          </except>
        </anyName>
      </attribute>
      <text/>
      <ref name="otherNamespaceElement"/>
    </choice>
  </zeroOrMore>
</element>

<grammar ns="http://www.w3.org/1999/xhtml">

<include href="/www2006/xhtml-strict.rng">
  <define name="otherNamespaceElement">
    <externalRef 
        href="/www2006/exclude/ns-xhtml-mathml.rng"/>
  </define>
</include>

<define name="Block.class" combine="choice">
  <externalRef href="../mathml/mathml2.rng" 
       ns="http://www.w3.org/1998/Math/MathML"/>
</define>

<define name="Inline.class" combine="choice">
  <externalRef href="../mathml/mathml2.rng" 
       ns="http://www.w3.org/1998/Math/MathML"/>
</define>

</grammar>

<define name="tabindexNumber.datatype">
  <data type="nonNegativeInteger">
    <param name="pattern">[0-9]+</param>
    <param name="minInclusive">0</param>
    <param name="maxInclusive">32767</param>
  </data>
</define>

<define name="Target.datatype">
  <data type="string">
    <param name="pattern">_(blank|self|parent|top)
                         |[A-Za-z].*</param>
  </data>
</define>

<sch:rule context="html:select">
  <sch:report test="not(@multiple) and 
                count(html:option[@selected]) > 1">
    Select elements which aren't marked 
    as multiple may not have more 
    then one selected option.
  </sch:report>
</sch:rule>

<sch:rule context="html:form">
  <sch:report test="descendant::html:form">
       form element can not have any nested 
       form elements
  </sch:report>
</sch:rule>

<sch:rule context="html:param">
 <sch:assert test="@name">
  The name attribute of param element is mandatory.
 </sch:assert>
</sch:rule>

<html>
  ...
  <body>
    ...
    <h1>Datatype tests</h1>
    <table border='10%'>
      <tbody>
        <tr>
         <td><font color='nougat'>B</font></td>
        </tr>
      </tbody>
    </table>
    ...
    <h1>Nested form</h1>
    <form action='process.form'>
      <div>
        <form action='process.subform'>
         <p>Somethings wrong</p>
        </form>
      </div>
    </form>
    ...
    <h1>Id and name are not the same</h1>
    <form name='name1' id='name2' 
          action='process.form'>
      <p>Somethings wrong</p>
    </form>
    ...
  </body>
</html>

Relaxed application was designed to be fully configurable, extensible and to provide a transparent and easy-to-use validation interface. Beside the application programming interface (API), which may be used to embed Relaxed validation functionality into other applications or tools, Relaxed features a Web-based user interface and a command-line user interface as well. Where command-line interface is mainly intended for testing and administration, the Web-based interface may serve for two different purposes. An HTML font-end may be rendered by a Web browser to allow a broad variety of users to interact with the validation engine, whereas an XML result output is intended for automated processing by another service or application.

Figure 1. Relaxed user interface

4.1. Web-based user interface

Users may access Relaxed validation functionality through an HTML user interface with their Web browser. The only mandatory parameter to start a validation process is the URL of the source document. But the user can specify a number of additional validation parameters as well. Documents are implicitly validated against XHTML1.0/HTML4.01 specification, but as you can see on Figure 1 there are several other possibilities to choose from the “restriction” select box. For instance the user may include WCAG 1.0 [5] restrictions, choose to disallow elements from different namespaces, to validate compound documents etc…

By default the document type declaration is automatically detected by Relaxed. The “doctype” select box allows user to force Relaxed to handle his document as a different document type than declared. XHTML1.0/HTML4.0 specifies three language subsets with different level of strictness. One reasonable approach to use this validation parameter is to check your document's validity for a less or more restrictive language subset. In case your document is valid for a different document type than declared, you may think about changing your document type declaration. You may also use this parameter to bypass the Relaxed autodetection mechanism for some reason.

At the bottom in Figure 1 you see three check boxes. They represent another validation parameters which can be also specified. The check box labelled “view source” appends the complete document's source at the bottom of the validation output and links the error messages to the corresponding source lines. “Brief output” hides messages with a low severity level when checked. The last check box called “dirty parser” makes Relaxed convert the current document into well-formed XML before the validation takes place. This helps authors to focus on the structural problems rather than on syntactical ones. The purpose of those three parameter's is mainly to rearrange the validation output to make it more readable and easy to understand.

In Figure 2 you see an example of the validation output. The demonstrational document is considered to be invalid. There is one info message at the top telling the user that Relaxed has successfully detected the document type. This is followed by four error messages. The first one complains about a mismatched name and id attribute within the scope of the same element. The second error highlights a form element nested into another form element. And the third and fourth error reflects an HTML datatype violation. All of the mentioned errors are XHTML1.0/HTML4.0 specification violations and none of them is reported when using the current version of W3C validator (see Section 3.5).

Figure 2. Validation output

4.2. Relaxed internals

Relaxed validation engine is written in Java and uses a number of third-party libraries. One of those libraries is the Sun‘s Multi-Schema Validator (MSV), which is used to validate XML against RELAX NG schemas. Another reason for using MSV is that it implements a common validation interface called JARV. This interface allows Relaxed to easily switch to a different validation library or to configure Relaxed to use schemas written in several different schema languages and combine them.

Relaxed was also designed to validate Schematron patterns embedded in RELAX NG schemas. This step is achieved through several XSLT transformations. Those are processed using the Saxon library. One of the reasons to use Saxon are its extension capabilities. As the standard transformation mechanism doesn't provide information about current source file line numbers, one of the Saxon's extensions is used to provide this information, which is essential for validation results usability and convenience.

Using current XML technologies gives Relaxed the power to validate more than the others, but not all real-life HTML documents are XML instances. Despite XML brings many advantages to HTML, there is still a significant number of documents adhering to the HTML 4.x specification, which is an application of SGML. As SGML is less restrictive than XML, those documents mostly don't meet the requirements for a well-formed XML which makes processing of those documents using common XML tools impossible. Enabling Relaxed to validate XHTML documents only would make its applicability in current heterogeneous Web environment very limited. That's a reason why Relaxed uses another library called TagSoup. A specially modified version of TagSoup brings the possibility to convert HTML documents to well-formed XML (basically to XHTML) and to keep users informed about most SGML syntax violations at the same time. TagSoup architecture always guarantees a well-formed output which makes possible to validate almost all of the current real-life HTML documents using Relaxed validation engine.

4.2.1. Application architecture

Relaxed architecture consists of several integrated components. The centerpiece of the application is a validator container, which contains a set of different validation components. The container basically manages the validation process. It invokes all registered validation components in a sequence and finally aggregates their validation results.

A validation component is designed to handle a large number of validation requests against a small set of prearranged schemas. Preparing a schema for validation may be a quite expensive operation. For that reason the component caches every prepared schema when it is first requested in order to make it ready for future immediate use.

Figure 3. Application architecture

Currently there are two different validation components implemented in the Relaxed project. One of them is basically a wrapper around the JARV validation interface. The component can be be configured to use any schema language supported by any validator which implements the JARV interface. This validation component is used in the Relaxed project to validate the modular RELAX NG schemas.

The second component does the embedded Schematron pattern validation. The schema preparation process is a series of three XSLT transformations. First transformation extracts the embedded Schematron patterns from RELAX NG schema modules. After that, Schematron is transformed into an XSLT stylesheet. Finally the third transformation adds Saxon line number extension elements. The outcome of those transformations is again an XSLT stylesheet which can be directly used for validation of the individual documents. Such validation process is again nothing else than another XSLT transformation.

Figure 4. Schematron validation

Another part of the architecture is a filter chain mechanism. Before any document is processed by the validator container, it is first passed into a filter chain. The chain can consist of a number of filters. Filters may modify the validated document content with respect to the current validation properties. As well as the validation components, filters also have the ability to issue validation errors. One example of a filter implementation is the HTML to XHTML conversion filter.

4.2.2. Schema selection

As there are several types of HTML documents and every type needs a special schema to be validated, Relaxed validator needs some mechanism to map document types to relevant schemas. An eligible unique document type identifier is the public identifier used in the document type declaration. Occurrence of such declaration in HTML documents is required by the W3C HTML specification.

Relaxed validation components use a special configuration file to map document types to schemas. Such mapping may be further structured into different validation options which are basically different mapping subsets. It is possible to specify which option to use at each validation request. For instance a user may specify such an option, using a select box in a user interface. Grouping mappings into options allows Relaxed users to select schemas or a group of schema modules involved in the particular validation process and thereby for instance adjust the level of validation strictness.

4.2.3. Validation dispatching and validation result aggregation

When a validation request is retrieved, the validation container component performs a sequence of predefined steps. At first the specified document is retrieved, filtered by the filter chain and stored in a reusable form to be accessible among several validation components. After that the validator container triggers all registered validation components to start their individual validation tasks.

During this process, the validator container propagates a shared error handler instance into all validation components and even into all chained filters. The error handler collects all messages issued by different components. It may also perform additional operations e.g. message sorting or grouping on them.

A message contains an explanatory text, severity level and when it makes sense also a position of the described issue in the source document (usually specified by a line number and possibly by a column). The info severity level is used to inform the user about the validation process stages. Warnings are used in case some observed issue is not directly violating the specification, but instead its use in such a context is not recommended or questionable. The validated document is labelled as invalid in a case that the error handler retrieves any error or fatal error messages. The error severity level is used in case of a recoverable schema violation, while fatal error announces a serious problem, which makes further document processing impossible (e.g. document isn't well-formed).

4.2.4. Legacy HTML parser

As discussed in Section 4.2, for Relaxed it is very important to keep backward compatibility with HTML4.x documents. For that reason the TagSoup library has been integrated into the Relaxed project. TagSoup is a SAX-compliant parser which allows standard XML tools to be applied to the real-life HTML documents. What's important, TagSoup architecture guarantees a well-formed output under all circumstances without any syntax error thrown. The TagSoup motto is “Just Keep On Trucking​”. This means that from any HTML4.x document Relaxed gets always a well-formed input.^[2]

From the Relaxed point of view the TagSoupes philosophy is both, its biggest advantage and its biggest disadvantage. Relaxed requirement is to convert the SGML syntax into the more strict XML syntax but to stay notified about any SGML violations at the same time. A TagSoup patch was necessary to achieve that.

TagSoup is basically a simple state-machine. When parsing a document, every important unit of the document transits TagSoup into a different state by invoking an action. A simple example of such behavior is an end tag without the start tag. This transits TagSoup into a special state where a repair action is invoked and the omitted tag is set right. Other violations e.g. missing end tag, unknown entities, attribute minimization, overlapping tags (see Example 10) etc. are repaired in a similar manner. The problem is that fixed SGML violations stay hidden to Relaxed and the user doesn't get notified about them. Overlapping tags are a fatal violation of XML as well as SGML. If TagSoup wouldn't fix such a problem the XML parser used in Relaxed would throw a fatal error, the validation process stops and the user gets finally notified. As you can see one solution is to modify TagSoup to fix just those errors which violate XML but not SGML.

Example 10. Overlapping elements

before TagSoup:

<p> <i> Hello,</p> world! </i>

after TagSoup:

<p> <i> Hello,</i></p><i> world! </i>

Another approach currently used in Relaxed is to let TagSoup fix all problems, but force him to report those, which are an SGML as well as an XML violation, with a specific error message. This approach is more comfortable for the user. As the validation process doesn't stop, the user gets most of the errors during a single validator run.

4.2.5. REST Interface

Representational State Transfer (REST) is a model for building Web-services based solely on the HTTP protocol and the URL specification. The client accesses a URL (a resource identifier) and gets a response (a resource representation) in some understandable form, for instance XML, HTML, PNG etc... REST is a light-weight alternative to SOAP, XML-RPC and other protocols which may mean an overhead in some simple situations. This is exactly the case of Relaxed as it features just one method with a small number of parameters.

The general URL for Relaxed REST Interface looks like this:

http://servername/context/method?parameter1=value&
      parameter2=value&...&parameterN=value.

The validation method is called validate and the context name depends on the particular server deployment. The validate method has a set of parameters. They basically correspond to the validation parameters introduced in Section 4.1.

Validation parameters

uri

the document to be validated (mandatory to start the validation process)

option

name of the validation option, specified in the document type to schema mapping, see Section 4.2.2 (if not specified a default option is used)

source (true / false – default)

specifies whether to include the full document's source within the validation output

severity (true / false – default)

filters low severity level messages from the output

filterForcedDoctype (... / autodetect – default)

forces an explicit document type, if not default

filterDirtyParser (true / false – default)

if true, document is converted to well-formed XML before validation

xml (true / false – default)

choose between HTML and XML output

The xml parameter determines the output format. If not specified the output is by default HTML which can be rendered by a Web browser. If set to true, the response is an XML output which is tailored for further automated processing by some other tools, applications or different services (see Example 11).

Example 11. Relaxed XML output

<relaxed>
  <source 
    url="http://nalevka.com/resources/
	 relaxed/poc.html"/>
  <output result="Your document is invalid.">
    <message severity="INFO">
      <text>Forced document type: 
            -//W3C//DTD XHTML 1.0 Strict//EN</text>
    </message>
    <message severity="ERROR">
      <locator line="9" column="25" />
      <text>attribute "border" has a bad value: 
           "10%" does not satisfy 
           the "nonNegativeInteger" type</text>
      <source>...&lt;table border='10%'&gt;</source>
   </message>
   ... more messages ...
   </output> 
</relaxed>

There is no doubt that compound documents are the future of the Web. Compound documents combine elements from several vocabularies in one XML document. In the Web paradigm, compound document is usually a combination of XHTML with SVG, MathML, XForms, SMIL or VoiceXML fragments. In Section 3.2 we have shown how to manually create a schema which combines several vocabularies together and which can be used for validation of compound documents. We plan to create schemas for additional vocabulary combinations in the future. However, this approach does not scale well because there are simply too many possible vocabulary combinations.

In order to create even better support for compound documents in Relaxed we plan to implement our own NVDL based validator. NVDL (Namespace Validation and Dispatching Language) [1] is an upcoming ISO standard for validation of compound documents. In NVDL you can specify how to split compound document into several fragments and how to validate these fragments against separate schemas.

<rules xmlns="http://purl.oclc.org/dsdl/
                     nvdl/ns/structure/1.0">
  <namespace ns="http://www.w3.org/1999/xhtml">
    <validate schema="xhtml.rng">
      <mode>
        <namespace ns="http://www.w3.org/1999/
                       02/22-rdf-syntax-ns#">
          <validate schema="rdfxml.rng">
            <mode>
              <anyNamespace>
                <attach/>
               </anyNamespace>
             </mode>
           </validate>
         </namespace>
       </mode>
     </validate>
  </namespace>
</rules>

As far as we know Relaxed is the only validator that provides validation of compound documents.^[3] There are validators like XML schema validator and Validation Service for RELAX NG which are based on W3C XML Schema or RELAX NG and are able to check data types better then DTD validators. However, none of those validators is supporting embedded Schematron rules and thus it is far beyond Relaxed validation capabilities. This is also supported by the fact that our RELAX NG schemas for XHTML are being used by another validation services, not only by Relaxed.

Currently Relaxed is also used for evaluating Web page accessibility in European Internet Accessibility Observatory project. We cooperate with MedIEQ project. The aim of this project is to develop tools for quality labelling of medical Web sites. Relaxed will be used there to check validity and some aspects of accessibility.

This article has shown that our approach of using RELAX NG combined with Schematron for Web documents validation is in all aspects superior to currently used DTD based validation. Implementation of our validation approach—Relaxed validator—is mature enough for production use. Significance of Relaxed will grow up with adoption of compound documents on the Web because currently there are no other validators able to validate compound documents.

Relaxed is an open-source project hosted on SourceForge. Relaxed on-line validation service is available at http://badame.vse.cz/validator/.