XProc, or the XML Pipeline Language ( http://www.w3.org/TR/xproc/), is an upcoming W3C standard for pipeline processing of XML data. Emphasizing a declarative approach to XML data manipulations, XProc can greatly simplify the development of XML applications, providing application developers with an easy access to many XML technologies such as schema validation, XInclude processing, or XQuery.
This article provides a brief introduction to XProc for solution architects and developers who are interested in learning about the benefits of using a declarative XML processing model in their XML applications. Basic familiarity with the common XML technologies (XSL transformations, schema validation, etc.) is assumed.
XProc has every potential to become one of the most useful new XML technologies aroundand and is attracting growing interest in the XML community, both from users and implementers.
XProc is a declarative language for describing sequences, or pipelines, of operations to be performed on XML documents. It deals with the task of composing XML processes, a common problem that has to be dealt with in almost every XML application. Be it a simple XML data manipulation or a more convoluted process that involves, for instance, running an XQuery over a set of documents in an XML database, validating the results with XML Schema, and generating the final XHTML report using XSLT, XProc makes implementing this kind of functionality remarkably easy and transparent — a feature that is often quite difficult to achieve with traditional “program-everything-yourself” approaches.
This article provides a brief introduction to XProc. It discusses the benefits of XProc over traditional approaches to XML processing and describes the main principles and concepts of the language.
Declarative XML Processing
Manual XML programming has always been a mundane and an error-prone task, with quite a steep learning curve as well: the beginning developer has to get familiar with the available XML processing data models and APIs (DOM, XOM, SAX, StAX, ...), pick the most appropriate one for his needs and then, after some experimentation, often rely on the help of others when things don't work as expected (“How do I move a DOM node from one document to another?”).
When an application needs to integrate multiple technologies or tools for different kinds of XML processing (querying, applying an XSLT stylesheet, etc.), additional problems may arise. In order to make these technologies work together, the developer often needs to write code that converts the output of one tool to structures that can by accepted by another. The need for this conversion is obvious in cases when heterogeneous data models are used (such as a combined relational and XML-based storage), but a translation of some kind is often unavoidable also in pure XML environments: the different tools may be based on different paradigms or processing models, or their APIs are just not directly compatible.
Integrating different XML processing tools can be a non-trivial task, and there is also a great danger of mistakes that can lead to unnecessary performance bottlenecks — or worse, to fatal error conditions in the application.
Another problem is that one-to-one mapping between different models is not always possible. Some information may get changed — or even lost — during the conversion, because there is no natural representation for it in the target domain. This phenomenon is often referred to as impedance mismatch and has always been a well-known source of problems when integrating different technologies, not only in the XML area.
With XProc, many of the issues described above can be reduced significantly. Emphasizing a declarative approach to XML data manipulations, XProc shields the developer from the complicated (and — from the XML perspective — often unimportant and distracting) details of the underlying XML frameworks and tools. In XProc, the developer specifies what actions (and in what order) are to be performed on XML data, but the implementation of how this is done are left to the XProc processor.
This is a very important aspect of XProc, since it can make the XML applications much easier, and therefore cheaper, to develop. Looking from a different perspective, XProc-based applications will also likely be more stable and reliable, simply because the amount of actual XML programming is reduced, and therefore the risk of bugs in the application code is lower. To put it bluntly, XProc can protect the applications from poor XML programming practices and bad-quality coding.
Using a declarative XML processing model has also a positive effect on the maintainability of the applications — it is usually easier to detect and fix problems in an XProc pipeline than in the application code, which is often hard to understand or entangled with other pieces of the application.
Another interesting benefit of using XProc is that it can make the applications much easier to customize and extend. Very often, new functionality can be introduced by simply adding new pipelines to the application (or by modifying the existing ones), with no or very little changes to the application itself. In the traditional model, especially with larger applications or application frameworks, customizations are often quite costly (if possible at all), simply because the internal model is not flexible enough.
This section provides a brief overview of XProc and discusses some of its main principles. For more detailed information, see the XProc specification. Another useful resource is XProc.org, a more informal website about XProc, its background, and the existing implementations. It also contains an FAQ section that answers some of the common XProc-related questions.
XProc is a declarative, XML-based language for pipeline processing of XML documents. The individual operations to be performed on XML documents are expressed in the form of steps that apply some well-defined transformations to the XML data. Steps can be organized into (possibly non-linear) sequences, called pipelines, to implement more complex behavior.
An XProc step can be thought of as black box with a set of named input ports and output ports. The step expects zero or more XML documents to appear on its input ports and produces zero or more result XML documents on its output ports.
The behavior of some XProc steps can be influenced by specifying options (for instance, an XSLT version to use in a step that transforms a document) or by passing parameters to the underlying technology (for instance, parameters to the XSLT stylesheet). Parameters are different from options in that their names may not be known in advance to the pipeline author. Parameters are provided to the step through a special type of an input port: so-called parameter input port. (To distinguish between the two input port types, the non-parameter input ports are sometimes referred to as document input ports.)
Input and output ports can be declared to accept/produce sequences of XML documents, instead of only a single XML document. Parameter input ports always accept sequences of documents.
Input ports of a step can be connected to output ports of other steps, to external sources, or to inline data specified in the pipeline document itself. Outputs of a step are either consumed by other steps or, if not used, discarded.
Steps in XProc can be of two main types: atomic and compound. Atomic steps perform indivisible (from the XProc perspective) operations on input XML documents, such as, for instance, performing schema validation or applying an XSLT stylesheet. Compound steps contain sequences — sub-pipelines — of other steps (atomic or compound). A pipeline that validates the input XML document and then applies an XSLT stylesheet is an example of a compound step.
The XProc specification defines a standard library of steps that can be used as basic building blocks when writing pipelines. See the section called “Standard Step Library” for a complete list of available XProc steps.
By design, XProc is primarily an XML processing language. Although there are standard steps and language constructs that provide some support for dealing with non-XML and binary data (mostly in terms of reading from and writing to external locations), XML is always used as the underlying model: what flows between the steps in an XProc pipeline are XML documents; the result of running an XProc pipeline is a sequence of XML documents.
In order to execute an XProc pipeline, an XProc processor is necessary. An XProc processor is an application that interprets XProc pipelines and applies them — evaluating the individual steps in proper order — to the input XML documents.
Despite XProc still being a relatively new technology, a number of XProc processors are already available (or is in active development). Calabash, the reference XProc processor implementation, implements most of the language, and is available as an open source project. EMC Documentum Dynamic Delivery Services uses an embedded XProc processor for most of the XML-related processing. You can follow the progress of the implementations at the XProc Test Suite page.
Figure 3, “Validate/Transform Pipeline” shows an example XProc pipeline that takes an XML document, validates it using XML Schema, and transforms it using an XSLT stylesheet.
The pipeline consists of two steps: the first step validates the input XML document, and the second step applies an XSLT stylesheet. The pipeline has four input ports:
source– The document to be processed by the pipeline;
schema– XML Schema documents to be used during schema validation;
stylesheet– The XSLT stylesheet to be used for transforming the input document
parameters– A parameter input port used for supplying parameters to the pipeline (in this case, parameters for the XSLT stylesheet).
The pipeline has a single output port named
result. It is connected to the
result output port of the XSLT step and it will provide access to the result document after running the pipeline.
The example below shows the source code of the pipeline:
Example 1. Validate/Transform Pipeline Source
<p:declare-step name="main" xmlns:p="http://www.w3.org/ns/xproc"> <p:input port="source"/> <p:input port="schema" sequence="true"/> <p:input port="stylesheet"/> <p:input port="parameters" kind="parameter"/> <p:output port="result"> <p:pipe step="transform" port="result"/> </p:output> <p:validate-with-xml-schema name="validate"> <p:input port="source"> <p:pipe step="main" port="source"/> </p:input> <p:input port="schema"> <p:pipe step="main" port="schema"/> </p:input> </p:validate-with-xml-schema> <p:xslt name="transform"> <p:input port="source"> <p:pipe step="validate" port="result"/> </p:input> <p:input port="stylesheet"> <p:pipe step="main" port="stylesheet"/> </p:input> <p:input port="parameters"> <p:pipe step="main" port="parameters"/> </p:input> </p:xslt> </p:declare-step>
As can be seen from the source code, the main namespace of the XProc vocabulary is
http://www.w3.org/ns/xproc. In addition to that, XProc pipelines can use two other XProc-related namespaces:
http://www.w3.org/ns/xproc-step (used for documents that are inputs or outputs of some standard XProc steps) and
http://www.w3.org/ns/xproc-error (used for reporting errors).
By convention, the namespace prefix “
p:” is used further in this article to indicate elements from the
http://www.w3.org/ns/xproc namespace. Similarly, the namespace prefix “
c:” is used for elements from the
The pipeline in the example is declared using the
p:declare-step element. The pipeline first specifies its input and output ports (
p:output elements). What follows are the actual steps in the pipeline:
As declared in the XProc standard step library (see the section called “Standard Step Library”), the
p:validate-with-xml-schema has two input ports (
schema) and one output port named
p:xslt has three input ports (
parameters) and two output ports (
The steps in the pipeline are given names (
main for the pipeline itself) so it is possible to refer to the steps when binding them together and creating the desired flow. This is achieved by connecting the ports of the steps by using
Aids for Pipeline Authors
The source code in Example 1, “Validate/Transform Pipeline Source” is unnecessarily verbose. XProc provides a number of convenience features for pipeline authors that both reduce the amount of coding and increase the overall readability of the code. Most importantly:
Input and output ports can be designated as being primary, which indicates to the XProc processor that it can apply some default processing to these ports. More specifically, if a step has an unconnected primary input port, it will be automatically connected to the primary output port of the preceding step (or, in some cases, to the primary input port of the step's container step). A step can have at most one primary document input port, at most one primary parameter input port, and at most one primary output port.
If a step has exactly one document input port, and that port is not explicitly set to be non-primary, it will become the step's primary input port. The same rule applies to parameter input ports and output ports.
p:pipelinecan be used as a shortcut for
p:declare-step. A pipeline declared using
p:pipelinealways has one
sourceprimary non-sequence input port, one
parametersprimary parameter input port, and one
resultprimary non-sequence output port by default. Additional ports can be added to the pipeline, if necessary.
By applying these rules, the pipeline can be rewritten as follows:
Example 2. Validate/Transform Pipeline — Simplified Version
<p:pipeline name="main" xmlns:p="http://www.w3.org/ns/xproc"> <p:input port="schema"/> <p:input port="stylesheet"/> <p:validate-with-xml-schema> <p:input port="schema"> <p:pipe step="main" port="schema"/> </p:input> </p:validate-with-xml-schema> <p:xslt> <p:input port="stylesheet"> <p:pipe step="main" port="stylesheet"/> </p:input> </p:xslt> </p:pipeline>
This version of the pipeline does not declare the
result ports (because of the
p:pipeline shorthand). Furthermore, because the
result ports of
p:xslt are declared as primary in the standard step library, some of the explicit connections between the steps in the pipeline can be left out.
Creating Dynamic Pipelines
XProc pipelines are not restricted to only simple linear sequences of steps: XProc provides language constructs that make it possible to create more complex and dynamic flows. Most of these constructs are based on evaluating XPath expressions, which XProc uses as its main expression language.
XProc contains the following built-in steps that can be used for creating more dynamic pipelines:
p:for-each– Applies a sub-pipeline to each XML document in the input sequence of documents.
p:viewport– Applies a sub-pipeline to selected portions of the input XML document.
p:choose– Selects one of the contained sub-pipelines based on evaluating XPath test expressions over the input XML document.
p:group– Wraps a sup-pipeline in a single compound step.
p:try– Wraps a sub-pipeline and makes it possible to catch dynamic errors that occur within it in a recovery sub-pipeline.
As mentioned in the section called “XProc Basics”, XProc steps can take options. Option values can be either static or computed dynamically, by evaluating XPath expressions.
Similar to options are variables. Variables can be used within compound steps to store string results of evaluating XPath expressions, and can be referred to in other XPath expressions.
The usage of options and variables is illustrated in Example 3, “Options and Variables”. The pipeline in the example contains a single
p:add-attribute step, which adds an attribute to a set of matching elements in the input document. According to the declaration in the standard step library, the step has three required options:
match (identifies the elements to add the attribute to),
attribute-name (name of the attribute), and
attribute-value (value of the attribute).
attribute-name are specified as attributes on the
p:add-attribute element; the values of these attributes represent the static values for the options. The
attribute-value option is specified using the
p:with-option element. The
select attribute contains an XPath expression that will be evaluated against the input XML document during run-time. The XPath expressions refers to the
root-name variable defined earlier in the pipeline.
Example 3. Options and Variables
<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"> <p:variable name="root-name" select="local-name(/*)"/> <p:add-attribute match="/*" attribute-name="cnt"> <p:with-option name="attribute-value" select="count(//*[local-name() = $root-name])"/> </p:add-attribute> </p:pipeline>
XPath can also be used for selecting portions of XML documents to be passed to steps, or to iterate over, as shown in Example 4, “Dynamic Pipeline”. The pipeline in the example takes an XHTML document and resolves all resources the document refers to (using
Example 4. Dynamic Pipeline
<p:declare-step xmlns:p="http://www.w3.org/ns/xproc" xmlns:c="http://www.w3.org/ns/xproc-step"> <p:input port="source"/> <p:output port="result" sequence="true"/> <p:for-each> <p:iteration-source select="//a"/> <p:choose> <p:when test="/a[@href]"> <p:variable name="target" select="/a/@href"/> <p:add-attribute match="/c:request" attribute-name="href"> <p:input port="source"> <p:inline> <c:request method="GET"/> </p:inline> </p:input> <p:with-option name="attribute-value" select="$target"/> </p:add-attribute> <p:http-request/> </p:when> <p:otherwise> <p:wrap-sequence wrapper="unresolvable"/> </p:otherwise> </p:choose> </p:for-each> </p:declare-step>
The main part of the pipeline is a for loop that iterates over all
a elements. Inside the loop, there is a choose/when statement that tests whether the
a element contains an
href attribute. If so, it invokes the
p:http-request step to fetch the resource; otherwise it just wraps the
a element in an
unresolvable document element.
The result of the for loop — and the result of the entire pipeline — is a sequence of XML documents of two types: successfully resolved resources and
unresolvable documents that represent unresolvable references.
The pipeline uses the
p:http-request step to fetch the resources. XML resources (or resources of a type that is derived from XML) are returned “as is”, with no modifications. If any of the resources is binary, the
p:http-request automatically applies Base64 encoding to the data and wraps the resulting character sequence in a
c:body document element. The input to the
p:http-request step is a
c:request document that specifies the properties of the request. In the example, the
c:request document specifies the request method (
GET) and the target URI (which is inserted in the document dynamically using the
One of the great features of XProc is its natural extensibility. In XProc, pipelines can be imported in other pipelines and used as any other steps there. For even higher degree of modularity and reusability, XProc makes it possible to create pipeline libraries, collections of multiple pipelines with related functionality.
Before a pipeline (either standalone or contained in a pipeline library) can be imported, it must be given a type (which is an XML qualified name) so it can be referred to. Example 5, “Pipeline Library” below shows an example library that contains a single pipeline of type
Example 5. Pipeline Library
<p:library xmlns:p="http://www.w3.org/ns/xproc" xmlns:myext="http://developer.emc.com/ns/xproc-ext"> <p:pipeline name="main" type="myext:validate-transform"> <p:input port="schema"/> <p:input port="stylesheet"/> <p:validate-with-xml-schema> <p:input port="schema"> <p:pipe step="main" port="schema"/> </p:input> </p:validate-with-xml-schema> <p:xslt> <p:input port="stylesheet"> <p:pipe step="main" port="stylesheet"/> </p:input> </p:xslt> </p:pipeline> </p:library>
Suppose the library is available at URI
library.xpl (Note the “
.xpl” extension which is commonly used for XProc pipelines and libraries). The library can be imported as illustrated in Example 6, “Importing a Pipeline Library”.
Example 6. Importing a Pipeline Library
<p:pipeline name="main" xmlns:p="http://www.w3.org/ns/xproc" xmlns:myext="http://developer.emc.com/ns/xproc-ext"> <p:input port="schema"/> <p:input port="stylesheet"/> <p:import href="library.xpl"/> <myext:validate-transform> <p:input port="schema"> <p:pipe step="main" port="schema"/> </p:input> <p:input port="stylesheet"> <p:pipe step="main" port="stylesheet"/> </p:input> </myext:validate-transform> </p:pipeline>
Pipelines and pipeline libraries represent an elegant way of extending XProc in XProc itself. While this can be considered the preferred approach (because it guarantees interoperability between different XProc processors), there are also other possibilities how to extend the language.
The XProc specification allows pipeline authors to specify extension attributes on elements from the XProc namespace. Extension attributes are XML attributes that provide additional, processor-specific information about the element. This mechanism can be used, for instance, for influencing the behavior of some XProc steps when using a particular XProc processor. Despite being processor-specific, extension attributes do not necessarily break interoperability if used properly. The presence of extension attributes in a pipeline doesn't alter the overall flow of the pipeline (determined by the connections between steps), nor does it prevent other XProc processors from running the pipeline: extension attributes that are not supported are simply ignored.
Finally, XProc processors can extend the standard step library by introducing processor-specific steps to provide functionality that is not available — or is difficult or inefficient to implement — in XProc. While this may break interoperability (pipelines written for one processor may not run with another), extensions steps can offer significant value for XML application developers. For example, the XProc processor used in EMC Documentum Dynamic Delivery Services provides integration with the EMC Documentum xDB native XML database as well as other EMC technologies.
Standard Step Library
There are three types of steps that are available in XProc by default: built-in steps, required steps, and optional steps.
Built-in steps are part of the language and provide the pipeline author with the basic constructs for creating dynamic pipelines: conditionals, loops, try/catch blocks, etc.
Required and optional steps are defined in a so-called standard step library and can be used, in addition to the built-in steps, in every XProc pipeline. Required steps are guaranteed to be supported by every compliant XProc processor, whereas the availability of optional steps may differ between individual XProc processors.
Table 1. Built-in Steps
|Applies a sub-pipeline to a sequence of input documents.|
|Applies a sub-pipeline to selected portions of the input document.|
|Selects one of the contained sub-pipelines based on an XPath test expression.|
|Wraps a sub-pipeline in one compound step.|
|Allows for catching and handling dynamic errors that occur in a sub-pipeline.|
Table 2. Required Steps
|Adds an attribute to a set of elements in the source document.|
|Exposes the base URI information in the source document via inserting explicit |
|Compares two documents for equality.|
|Counts the number of input documents.|
|Deletes items in the source document.|
|Lists the contents of a specified directory.|
|Generates a dynamic error.|
|Serializes (“escapes”) the children of the document element in the source document.|
|Selects portions of the source document based on an XPath select expression.|
|Provides support for HTTP and related protocols.|
|Produces a copy of the source document.|
|Inserts content at specified locations in the source document.|
|Adds a label attribute to specified elements in the source document.|
|Loads an XML document from an external location.|
|Absolutizes URI values of selected elements or attributes.|
|Rebinds a namespace URI to a new URI.|
|Merges two document sequences pair-wise.|
|Generates a document from a parameter set.|
|Renames elements, attributes, or processing instructions in the source document.|
|Replaces portions of source document with given replacement.|
|Sets multiple attributes on selected elements in the source document.|
|Discards the input documents.|
|Splits the sequence of two documents into two based on an XPath expression.|
|Stores the source document to an external location.|
|Replaces selected nodes in the source document by a dynamically constructed string value.|
|Deserializes (“unescapes”) the content of the document element in the source document.|
|Replaces selected elements with their children.|
|Wraps selected nodes in the source document with given wrapper element.|
|Wraps a sequence of input documents into a single document with given document element.|
|Applies XInclude processing to the source document.|
|Applies an XSLT stylesheet to the input document.|
Table 3. Optional Steps
|Runs an external command.|
|Generates a hash for a given value and inserts it in the source document.|
|Generates a UUID and inserts it in the source document.|
|Validates the source document against a RELAX NG grammar.|
|Performs Schematron validation on the source document.|
|Performs XML Schema validation on the source document.|
|Decodes a URL-encoded string into a set of parameters.|
|Encodes a set of parameters as a URL-encoded string and inserts it in the source document.|
|Executes an XQuery on the sequence of source documents.|
|Renders an XSL-FO document and stores the result to an external location.|
This article presented a brief introduction to , a declarative language for pipeline processing of XML data, and discussed some of its benefits over traditional approaches: XProc streamlines the development of XML applications and makes their architecture cleaner and more robust. It bridges the gaps between different XML technologies — and in turn, it also bridges the gap between XML and application developers by making XML processing simple and transparent.