9 querying xml data and documents

SDPL 2002 Notes 9: XQuery 1

9 Querying XML Data and Documents9 Querying XML Data and Documents

XQueryXQuery, W3C XML Query Language, W3C XML Query Language– "work in progress", Working Draft, 30 April 2002"work in progress", Working Draft, 30 April 2002– joint work by XML Query and XSL Working Groupsjoint work by XML Query and XSL Working Groups

» together with XPath 2.0 (together with XPath 2.0 ( XPath 1.0 and XQuery) XPath 1.0 and XQuery)

– influenced by many research groups and query influenced by many research groups and query languageslanguages

» QuiltQuilt, , XPathXPath, XQL, XML-QL, SQL, OQL, Lorel, ..., XQL, XML-QL, SQL, OQL, Lorel, ...

– Goal: a query language applicable to any XML-Goal: a query language applicable to any XML-represented data: both documents and databasesrepresented data: both documents and databases


Functional Requirements Functional Requirements (1)(1)

Support operations (selection, projection, aggregation, Support operations (selection, projection, aggregation, sorting, etc.) on all data types: sorting, etc.) on all data types: – Choose parts of data based on content or structureChoose parts of data based on content or structure– Also operations on document hierarchy and orderAlso operations on document hierarchy and order

Structural preservation and transformation:Structural preservation and transformation: – Preserve the relative hierarchy and sequence of input document Preserve the relative hierarchy and sequence of input document

structures in the query resultsstructures in the query results– Transform XML structures and create new XML structuresTransform XML structures and create new XML structures

Combining and joining:Combining and joining:– Combine related information from different parts of a given Combine related information from different parts of a given

document or from multiple documentsdocument or from multiple documents


Functional Requirements Functional Requirements (2)(2)

Cross-references:Cross-references:– must be able to traverse intra- and inter-document references must be able to traverse intra- and inter-document references

Closure property:Closure property: – The result of an XML query is also XML (usually not a valid The result of an XML query is also XML (usually not a valid

document, but a well-formed document document, but a well-formed document fragmentfragment))– The results of a query can be used as input to another queryThe results of a query can be used as input to another query

Extensibility:Extensibility:– The query language should support the use of externally The query language should support the use of externally

defined functions on all data types of the data modeldefined functions on all data types of the data model


XQuery in a NutshellXQuery in a Nutshell

Functional expression languageFunctional expression language Strongly-typedStrongly-typed: : (XML Schema) types may be assigned (XML Schema) types may be assigned

to expressions staticallyto expressions statically Includes XPath 2.0 Includes XPath 2.0 (says(says Draft, but not all XPath axes Draft, but not all XPath axes included!) included!)

– XQuery 1.0 and XPath 2.0 share extensive functionality:XQuery 1.0 and XPath 2.0 share extensive functionality:» XQuery 1.0 and XPath 2.0 Functions and Operators, XQuery 1.0 and XPath 2.0 Functions and Operators,

WD 30/4/2002WD 30/4/2002

Roughly: XQuery Roughly: XQuery XPath' + XSLT' + SQL' XPath' + XSLT' + SQL'


XQuery: BasicsXQuery: Basics

A query is represented as an A query is represented as an expressionexpression Expressions operate on and return Expressions operate on and return sequencessequences of of

– atomic valuesatomic values (of XML Schema simple types) and (of XML Schema simple types) and– nodesnodes (same 7 node types as in XPath/XSLT) (same 7 node types as in XPath/XSLT)– an item an item a a singleton sequencesingleton sequence– sequences are sequences are flatflat: no sequences as items: no sequences as items

» (1, (2, 3), (), 1) = (1, 2, 3, 1)(1, (2, 3), (), 1) = (1, 2, 3, 1)

– sequences are sequences are orderedordered, and can contain duplicates , and can contain duplicates Unlimited combination of expressions, often with Unlimited combination of expressions, often with

automatic type conversions (e.g. for arithmetics)automatic type conversions (e.g. for arithmetics)


XQuery: BasicsXQuery: Basics

Query applied to an Query applied to an input sequenceinput sequence,, accessed through functions ...accessed through functions ...– input()input()

» implementation-dependent input sequenceimplementation-dependent input sequence

– collection("collection("URIURI")")» sequence of nodes from sequence of nodes from URIURI

– document("document("URIURI")") » root of the XML document available at root of the XML document available at URIURI

» this also in XSLTthis also in XSLT


Some Central XQuery ExpressionsSome Central XQuery Expressions

path expressionspath expressions creation and combination of sequences (creation and combination of sequences (unionunion, ,

intersectintersect, , exceptexcept)) element constructors (element constructors ( XSLT templates) XSLT templates) FLWR (”flower”; for-let-where-return) expressionsFLWR (”flower”; for-let-where-return) expressions sort expressionssort expressions quantified expressions quantified expressions ((somesome//everyevery … … satisfiessatisfies …) …)

testing and modifying datatypes; validationtesting and modifying datatypes; validation


Path ExpressionsPath Expressions

Extend the syntax of XPath: [/]Expr/… /ExprExtend the syntax of XPath: [/]Expr/… /Expr– arbitrary (node-sequence-valued) expressions as stepsarbitrary (node-sequence-valued) expressions as steps– only 6 (of 13) axes: only 6 (of 13) axes: child, descendant, attribute, self, child, descendant, attribute, self,

descendant-or-self, parentdescendant-or-self, parent» combined with order comparison operators (combined with order comparison operators (precedesprecedes, ,

followsfollows) sufficient for queries (?)) sufficient for queries (?)

– produce ordered sequence of nodes in document order, produce ordered sequence of nodes in document order, without duplicate nodeswithout duplicate nodes

– dereferencingdereferencing for ID/IDREF values: Figure(s) referenced for ID/IDREF values: Figure(s) referenced by by refidrefid attribute of the first attribute of the first figreffigref child: child:

figref[1]/@refid figref[1]/@refid =>=> figure figure


Element ConstructorsElement Constructors

Similar to XSLT templates:Similar to XSLT templates:– start and end tag enclosing the contentstart and end tag enclosing the content– literal fragments written directly, literal fragments written directly,

expressions enclosed in braces expressions enclosed in braces { { and and }} often used inside another expression that binds often used inside another expression that binds

variables used in the element constructorvariables used in the element constructor– See nextSee next

ExampleExample

Generate an Generate an empemp element containing an element containing an empidempid attribute and nested attribute and nested namename and and jobjob elements; elements; values of the attribute and nested elements given values of the attribute and nested elements given in variables in variables $id$id, , $n$n, and , and $j$j::

<emp empid="<emp empid="{$id}{$id}">">

<name><name>{$n}{$n} </name> </name>

<job> <job> {$j}{$j} </job> </job>

</emp></emp>


FLWR ("flower") ExpressionsFLWR ("flower") Expressions

Constructed from Constructed from forfor, , letlet, , wherewhere and and returnreturn clauses (~SQL clauses (~SQL selectselect--fromfrom--wherewhere))

Form: Form: ((FForClause | LetClause)+ WhereClause? "orClause | LetClause)+ WhereClause? "returnreturn" Expr" Expr

a FLWR expression binds values to one or more a FLWR expression binds values to one or more variables, and uses these variables to construct a variables, and uses these variables to construct a result (in general, an ordered sequence of nodes)result (in general, an ordered sequence of nodes)


Flow of data in a FLWR expressionFlow of data in a FLWR expression


forfor clauses clauses

forfor $V $V11 inin ExpExp11 (, …) associates each variable (, …) associates each variable

VVii with an expression Exp with an expression Expii that returns a list of that returns a list of

items (e.g. a path expression)items (e.g. a path expression) Result: list of tuples, each containing a binding for Result: list of tuples, each containing a binding for

each of the variableseach of the variables can be though of as loops iterating over the items can be though of as loops iterating over the items

returned by respective expressionsreturned by respective expressions


letlet clauses clauses

letlet also binds one or more variables to one or also binds one or more variables to one or more expressionsmore expressions– each variable to the entire value of its expression each variable to the entire value of its expression

(without iterating through items of the sequence)(without iterating through items of the sequence)– results in binding a single sequence for each variableresults in binding a single sequence for each variable

Compare:Compare:– forfor $x $x inin /library/book /library/book many bindings (books) many bindings (books)– letlet $x := /library/book $x := /library/book single binding (to a single binding (to a

sequence of books)sequence of books)


forfor//letlet clauses clauses

A FLWR expr may contain several A FLWR expr may contain several forfors and s and letletss– each of these clauses may refere to variables bound in each of these clauses may refere to variables bound in

previous clausesprevious clauses

the result of the the result of the forfor//letlet sequence: sequence:– an ordered list of tuples of bound variablesan ordered list of tuples of bound variables– number of tuples = product of the cardinalities of the number of tuples = product of the cardinalities of the

node-lists returned by the expressions in the node-lists returned by the expressions in the forfor clausesclauses

for/let clauses - Example 1for/let clauses - Example 1

forfor $i$i inin (1,2), (1,2), $j$j inin (3,4) (3,4)returnreturn <tuple> <tuple>

{{$i$i} <j>{} <j>{$j$j}</j></tuple>}</j></tuple>

Result:Result:

<tuple>1<j>3</j></tuple><tuple>1<j>3</j></tuple>

for/let clauses - Example 2for/let clauses - Example 2

letlet $s := (<one/>, <two/>, <three/>) $s := (<one/>, <two/>, <three/>)returnreturn <out>{$s}</out> <out>{$s}</out>

Result:Result:

<out><out> <one/><one/> <two/><two/> <three/><three/></out></out>


wherewhere clause clause

binding tuples generated by binding tuples generated by forfor and and letlet clauses clauses filtered by an optional filtered by an optional wherewhere clause clause– those with a those with a truetrue condition are used to instantiate the condition are used to instantiate the

returnreturn clause clause

the the wherewhere clause may contain several predicates clause may contain several predicates connected by connected by andand, , oror, and , and notnot()()– usually refer to the bound variablesusually refer to the bound variables

– sequences treated as Booleans similarly to node-lists sequences treated as Booleans similarly to node-lists in XPath: empty ~ in XPath: empty ~ falsefalse; non-empty ~ ; non-empty ~ truetrue


wherewhere clause clause

Variables bound by a Variables bound by a forfor clause represent a clause represent a single itemsingle item scalar predicates, e.g. scalar predicates, e.g. $p/color = ”Red”$p/color = ”Red”

Variables bound by a Variables bound by a letlet clause may represent clause may represent lists of nodeslists of nodes list-oriented predicates, e.g. list-oriented predicates, e.g. avgavg($p/price) > 100($p/price) > 100– a number of aggregation functions available: a number of aggregation functions available: avgavg(), (), sumsum(), (), countcount(), (), maxmax(), (), minmin()()

(also in XPath 1.0)(also in XPath 1.0)


returnreturn clause clause

The The returnreturn clause generates the output of the clause generates the output of the FLWR expressionFLWR expression

instantiated once for each binding tupleinstantiated once for each binding tuple often contains element constuctors, references to often contains element constuctors, references to

bound variables, and nested subexpressionsbound variables, and nested subexpressions

Examples Examples (from "XML Query Use Cases")(from "XML Query Use Cases")

Assume: a document named ”Assume: a document named ”bib.xmlbib.xml” ” containing of a list of containing of a list of bookbooks:s:

<book><book>++

<title><title><author><author>++

<publisher><publisher><year><year><price><price>

<recent-MK-books> {<recent-MK-books> {

} </recent-MK-books>} </recent-MK-books>

List the titles of books published by List the titles of books published by Morgan Kaufmann after 1998Morgan Kaufmann after 1998

forfor $b $b inin documentdocument(”bib.xml”)//book(”bib.xml”)//book

wherewhere $b/publisher = ”Morgan Kaufmann” $b/publisher = ”Morgan Kaufmann”

andand $b/year $b/year gtgt 1998 1998

returnreturn <book year="{$b/year}"> <book year="{$b/year}">

{$b/title}{$b/title}

</book></book>

Result could be...Result could be...

<recent-MK-books><recent-MK-books> <book year=”1999”><book year=”1999”> <title>TCP/IP Illustrated</title><title>TCP/IP Illustrated</title> </book></book> <book year=”2000”><book year=”2000”> <title>Advanced Programming in the Unix <title>Advanced Programming in the Unix environment</title>environment</title> </book></book></recent-MK-books></recent-MK-books>

List each publisher with the average List each publisher with the average price of its booksprice of its books

forfor $p $p inin distinct-valuedistinct-value(( documentdocument(”bib.xml”)//publisher)(”bib.xml”)//publisher)

letlet $a := $a := avgavg(document(”bib.xml”)/book[(document(”bib.xml”)/book[

publisher = $p]/price)publisher = $p]/price)

returnreturn

<publisher> {<publisher> {

<name> {$p/text()}</name> ,<name> {$p/text()}</name> ,

<avgprice> {$a} </avgprice><avgprice> {$a} </avgprice>

} </publisher>} </publisher>

Invert the book-list structureInvert the book-list structure

<author_list>{ {-- group books by authors --}<author_list>{ {-- group books by authors --}

forfor $a $a inin distinct-valuesdistinct-values((documentdocument(”bib.xml”)//author )(”bib.xml”)//author )

returnreturn <author><author>

<name> {$a/<name> {$a/texttext()} </name>()} </name>

forfor $b $b inin documentdocument(”bib.xml”)//book[(”bib.xml”)//book[

author = $a]author = $a]

returnreturn $b/title $b/title

</author></author>

} </author_list>} </author_list>

Make an alphabetic list of publishers; for each publisher, list Make an alphabetic list of publishers; for each publisher, list books (title & price) in descending order of pricebooks (title & price) in descending order of price

forfor $p $p inin distinct-valuesdistinct-values((documentdocument(”bib.xml”)//publisher )(”bib.xml”)//publisher )

returnreturn <publisher><publisher>

<name> {$p/text()} </name><name> {$p/text()} </name> {{forfor $b $b inin documentdocument(”bib.xml”)/book[(”bib.xml”)/book[

publisher = $p]publisher = $p] returnreturn <book> {$b/title , <book> {$b/title , $b/price}$b/price} </book> </book> sortbysortby(price (price descendingdescending)})} </publisher> </publisher> sortbysortby(name)(name)


Queries on Document OrderQueries on Document Order

Operators Operators precedesprecedes and and followsfollows to express to express conditions on document orderconditions on document order

The next example involves a surgical report with The next example involves a surgical report with elements elements procedureprocedure, , incisionincision and and anesthesiaanesthesia

return a "critical sequence" that contains all return a "critical sequence" that contains all contents between the 1st and 2nd incisions of the contents between the 1st and 2nd incisions of the 1st procedure1st procedure

Computing a "critical sequence"Computing a "critical sequence"

<critical_sequence> {<critical_sequence> { letlet $p := //procedure[1] $p := //procedure[1] forfor $n $n inin $p/node() $p/node() wherewhere $n $n followsfollows ($p//incision)[1] ($p//incision)[1]andand

$n $n precedesprecedes ($p//incision)[2] ($p//incision)[2] returnreturn $n $n} </critical_sequence>} </critical_sequence>


FilteringFiltering

Function Function filter(Expr) filter(Expr) – in XQuery core function libraryin XQuery core function library

returns shallow copies of the nodes selected by returns shallow copies of the nodes selected by ExprExpr– mutual order and hierarchy btw nodes the same as in mutual order and hierarchy btw nodes the same as in

input sequenceinput sequence


Hierarchical FilteringHierarchical Filtering

Only selected nodes remain; other levels are Only selected nodes remain; other levels are collapsedcollapsed

Table of contents for”cookbook.xml”, consisting of Table of contents for”cookbook.xml”, consisting of section titles (at any level of nesting)section titles (at any level of nesting)

letlet $b := $b := documentdocument(”cookbook.xml”)(”cookbook.xml”)

returnreturn

<toc> {<toc> {

filterfilter($b//(section | ($b//(section |

section/title | section/title |

section/title/text()) ) section/title/text()) )

} </toc>} </toc>


Querying relational dataQuerying relational data

Lots of data is stored in relational databasesLots of data is stored in relational databases should be able to access also this datashould be able to access also this data Example: suppliers and partsExample: suppliers and parts

– Table S: supplier numbers (sno) and names (sname)Table S: supplier numbers (sno) and names (sname)– Table P: part numbers (pno) and descriptions (descrip)Table P: part numbers (pno) and descriptions (descrip)– Table SP: relationships between suppliers and the Table SP: relationships between suppliers and the

parts they supply, including the price (price) of each parts they supply, including the price (price) of each part from each supplierpart from each supplier


A possible XML form for relationsA possible XML form for relations

**

**

**


SQL vs. XQuery SQL vs. XQuery

SQL:SQL:

XQuery:XQuery:

SELECTSELECT pno pnoFROMFROM p pWHEREWHERE descrip descrip LIKELIKE ’Gear%’ ’Gear%’ORDER BYORDER BY pno; pno;

forfor $p $p inin documentdocument(”p.xml”)//p_tuple(”p.xml”)//p_tuplewherewhere starts-withstarts-with($p/descrip, ”Gear”)($p/descrip, ”Gear”)returnreturn $p/pno $p/pno sortbysortby(.)(.)


GroupingGrouping

Many relational queries involve forming data into Many relational queries involve forming data into groups and applying some aggregation function groups and applying some aggregation function such as such as countcount or or avgavg to each group to each group

in SQL: in SQL: GROUP BYGROUP BY and and HAVINGHAVING clauses clauses Example: Find the part number and average price Example: Find the part number and average price

for parts that have at least 3 suppliersfor parts that have at least 3 suppliers


Grouping: SQLGrouping: SQL

SELECTSELECT pno, pno, avgavg(price) (price) ASAS avgprice avgprice

FROMFROM sp sp

GROUP BY GROUP BY pnopno

HAVING countHAVING count(*) >= 3(*) >= 3

ORDER BYORDER BY pno; pno;

Grouping: XQueryGrouping: XQuery

forfor $pn $pn in distinct-valuesin distinct-values((documentdocument(”sp.xml”)//pno)(”sp.xml”)//pno)

letlet $sp := document(”sp.xml”)//sp_tuple[ $sp := document(”sp.xml”)//sp_tuple[

pno = $pn]pno = $pn]

where countwhere count($sp) >= 3($sp) >= 3

returnreturn

<well_supplied_item> {<well_supplied_item> {

$pn,$pn,

<avgprice> {<avgprice> {avgavg($sp/price)} </avgprice>($sp/price)} </avgprice>

} <well_supplied_item> } <well_supplied_item>

sortbysortby(pno)(pno)

JoinsJoins

Example: Return a ”flat” list of supplier names and their Example: Return a ”flat” list of supplier names and their part descriptions, in alphabetic orderpart descriptions, in alphabetic order

forfor $sp $sp inin documentdocument(”sp.xml”)//sp_tuple,(”sp.xml”)//sp_tuple, $p $p inin documentdocument(”p.xml”)//p_tuple[(”p.xml”)//p_tuple[

pno = $sp/pno],pno = $sp/pno], $s $s in documentin document(”s.xml”)//s_tuple[(”s.xml”)//s_tuple[

sno = $sp/sno]sno = $sp/sno]returnreturn <sp_pair>{ <sp_pair>{

$s/sname ,$s/sname , $p/descrip$p/descrip

}<sp_pair> }<sp_pair> sortbysortby (sname, descrip) (sname, descrip)

XQuery vs. XSLTXQuery vs. XSLT

Could we express XQuery queries with XSLT?Could we express XQuery queries with XSLT?– In principle In principle yes,yes, always, (but could be tedious) always, (but could be tedious)

Ex: Partial XSLT simulation of FLWR expressionsEx: Partial XSLT simulation of FLWR expressions

XQueryXQuery:: forfor $x $x inin Expr …Expr …

can be expressed incan be expressed in XSLT XSLT asas::

<xsl:<xsl:for-each for-each select="select="ExprExpr">"><xsl:<xsl:variable variable name="name="xx" select=".">" select=".">

……

</xsl:</xsl:for-eachfor-each>>

XQueryXQuery:: letlet $y := $y := Expr …Expr …

corresponds directly tocorresponds directly to:: <xsl:<xsl:variable variable name="name="yy" select="" select="ExprExpr" />" />

andand wherewhere Condition …Condition … can be expressed ascan be expressed as:: <xsl:<xsl:if if test="test="ConditionCondition">"> ……

</xsl:</xsl:ifif>>

XQueryXQuery:: returnreturn ElemConstructorElemConstructorcan be simulated with a corresponding XSLT template:can be simulated with a corresponding XSLT template:

• static fragments as suchstatic fragments as such• enclosed expressions in element content, e.g. enclosed expressions in element content, e.g. {$s/sname}{$s/sname} becomebecome

<xsl:<xsl:copy-ofcopy-of select="select="$s/sname$s/sname" />" />

XQuery vs. XSLT: ExampleXQuery vs. XSLT: Example

forfor $b $b inin documentdocument(”bib.xml”)//book (”bib.xml”)//book wherewhere

$b/publisher = ”MK” $b/publisher = ”MK” andand $b/year $b/year >> 1998 1998

returnreturn <book year="{$b/year}"> <book year="{$b/year}">

{$b/title} </book>{$b/title} </book>

XQueryXQuery::

<xsl:template match="/"><xsl:template match="/"> <xsl:for-each select=" <xsl:for-each select="documentdocument('bib.xml')//book('bib.xml')//book">"> <xsl:variable name=" <xsl:variable name="bb" select="." /> " select="." /> <xsl:if test=" <xsl:if test="$b/publisher='MK' $b/publisher='MK' andand $b/year $b/year >> 1998” 1998”>>

<book year="{$b/year}"><book year="{$b/year}">

<xsl:copy-of select="<xsl:copy-of select="$b/title$b/title" /> " /> </book></book> </xsl:if> </xsl:if> </xsl:for-each> </xsl:for-each> </xsl:template></xsl:template>


XSLT for XQuery FLWR ExpressionsXSLT for XQuery FLWR Expressions

NB: The sketched simulation is not complete:NB: The sketched simulation is not complete:– Only two things can be done in XSLT for Only two things can be done in XSLT for result tree result tree

fragmentsfragments produced by templates: produced by templates:» insertion in result tree, and conversion to a string insertion in result tree, and conversion to a string

(with (with xsl:value-ofxsl:value-of))

– Not possible to apply further operations to results (like, Not possible to apply further operations to results (like, e.g., sorting in XQuery)e.g., sorting in XQuery)


SummarySummary

XQueryXQuery– W3C working draft of an XML query languageW3C working draft of an XML query language– vendor support??vendor support??

» http://www.w3.org/XML/Queryhttp://www.w3.org/XML/Query mentions about 20 prototype implementations or mentions about 20 prototype implementations or products that support (or will support) some version products that support (or will support) some version of XQueryof XQuery

– future??future??– promising confluence of document and database promising confluence of document and database

researchresearch

9 querying xml data and documents

Documents

xml document available

xml schema types

querying xml data

document hierarchy

document order

data types

w3c xml query languagework

given document