LORELight Object Repository

by

Othman Chhoul

CSC5370 Fall 2003

Outline

IntroductionWhat is Lore?HistoryLore’s ForensicConclusionQuestionsDemo

Introduction

Limitations faced by traditional Databases: force all data to adhere to an explicitly

specified schema Data Elements may change Structures may change along the execution

path of an application Head ache when it comes to decide on a

fixed schema for irregular or unstable data

SemiStructured Data

Widespread SemiStructured Data: “Self-describing” “Schemaless”

Examples: Data from the web

Overall site structure may change often. It would be nice to be able to query a web site.

Data integrated from multiple, heterogeneous data sources.

Information sources change, or new sources added.

What is Lore?

Lore is a DBMS designed specifically for managing semistructured information, such as XML

Among the Pioneers in this domain

History

Built, from scratch, by the DB Group at Stanford University, with research funding from DARPA, NASA and others.

Introduced in 1995, with the first version of the query language called Lorel, and used OEM as data model.

A lightweight system, because it was designed for a single-user, read-only access.

1999 - changed to support XML

Lore’s Forensic

Lore’s Data model

Lore’s Query Language

Lore’s General Architecture

When XML gets into action

OEM (Object Exchange Model)

Simple, self-describing, nested object model for semi structured data (XML???)

Data in this model can be thought of as a labeled directed graph

Vertices in graph are objects. Each object has a unique object identifier (oid),

such as &5. Atomic objects have no outgoing edges and are

types such as int, real, string, gif, etc. All other objects that have outgoing edges are

called complex objects.

OEM (Summary)

An OEM object has:

Label: a character string, object aliases OID: Object unique identifier Type: Atomic (int, real, string), Complex Value: If it is a complex object list of OIDs

If it is an atomic object atomic value of type int, real, string…

OEM (Example)

Lorel (Lore’s Query Language)Lorel is an extension of OQLLorel supports path expressions for

traversing graph dataA simple path expression is a name

followed by a sequence of labels. DBGroup.Member.Office: Set of objects that

can be reached starting with the DBGroup object, following edges labels member and then office.

Lorel

Range variables can be assigned to path expression

Path expression are used directly in queries in an SQL style:

select DBGroup.Member.Office

where DBGroup.Member.Age > 30

Lorel

Result:

Office “Gates252”Office

Building “CIS”

Room “411”

Lorel (Behind the scenes)Previous query rewritten to OQL style:

select Ofrom DBGroup.Member M, M.Office Owhere exists y in M.Age : y > 30

Comparison on age transformed to existential condition: A user can ask DBGroup.Member.Age < 30

regardless of whether Age is single valued, set valued, or unknown.

Lorel (More examples) select DBGroup.Member.Name

where DBGroup.Member.Office(.Room%)?like “%252”

Result: Name “Jones” Name “Smith”

Update: update P.Member +=( select DBGroup.Member where DBGroup.Member.Name = "Clark" ) from DBGroup.Project Pwhere P.Title = "Lore" or P.Title = "Tsimmis"

Lore’s General Architecture

Lore’s General Architecture

Query and Update Processing

External Data

DataGuides

Query and Update Processing

Queries

Data Engine

(A Set of OEM objects)

Query Plan Generator

select Ofrom DBGroup.Member M, M.Office Owhere exists y in M.Age : y > 30

Query Iterators

Use recursive iterator approach:

execution begins at top of query plan each node in the plan requests a tuple at a time

from its children and performs some operation on the tuple(s).

pass result tuples up to parent.

Tuples (Object Assignment)

OA is a data structure containing slots for range variables with additional slots depending on the query.

Each slot within an OA will holds the oid of a vertex on a path being considered by the query engine.

We should end up at the end of a query with complete OAs

Query Operators

The Scan operator returns all oids that are sub-objects of a given object following a specified path expression: Scan (StartingOASlot, Path_expression, TargetOASlot) For each oid in StartingOASlot, check to see if

object satisfies path_expression and place oid into TargetOASlot.

For each returned OA of the left child, the join operator calls exhaustively the right child until no more OA is returned

Query Operators (cont)

The aggregation operator (Aggr) adds to the target slot the result of the aggregation.

The Join, Project and Select are almost identical to their corresponding relational operators

Other operators: CreateSet, GoupBy, ArithOp

Query Operators (Visualize the Words)

Query Operators (Visualize the Words)

Query Optimizer

Does only a few optimizations: Push selection ops down query tree. Eliminate/combine redundant query operators.

Explores query plans that use indexes when possible. Two kinds of indexes: Lindex (link index): returns all parents OIDs of a

given OID via a label, impl. as hashing. Vindex (value index): returns all atomic objects of a

label that satisfies a condition, impl. as B+-trees

Vindexes

Because of non-strict typing system, have String Vindex, Real Vindex, and String-coerced-to-real Vindex.

Separate B-Trees of each type are constructed for each label.

Using Vindex for comparison If type is string, do lookup in String Vindex If can convert to real the do lookup in String-

coerced-to-real Vindex. If type is real or int, do almost the same thin

Vindexes (cont)

Arg2

Arg1

String Real Int

String -- Stringreal

Bothreal

Real Stringreal

-- Int real

int Bothreal Intreal --

Index Query plans

If the user’s query contains a comparison between a path expression and a value + appropriate Vindex and Lindex exist generate an index query plan

Previous query: select Ofrom DBGroup.Member M,

M.Office Owhere exists y in M.Age : y > 30

Index Query plans (cont)

Update Query plans

update P.Member +=( select DBGroup.Member where DBGroup.Member.Name = "Clark" )

from DBGroup.Project Pwhere P.Title = "Lore" or P.Title =

"Tsimmis"

External Data

Enables retrieval of information from other data sources, transparent to the user.

An external object in Lore is a “placeholder” for the external data and specifies how lore interacts with an external data source.

External Data During query processing

Scan operator notifies the external data manager whenever an external object is encountered

The spec for an external object includes: Location of a wrapper

program to fetch and convert data to OEM,

timeout interval a set of arguments used

to limit info fetched from external source.

DataGuides

A DataGuide is a concise and accurate summary of the structure of an OEM database (stored as OEM database itself, kind of like the system catalog).

Very Helpful: No explicit database schema difficult to formulate

meaningful queries Query processor may perform unnecessary work

with no knowledge of the database structure. What if a path expression doesn’t exist (waste).

Each possible path expression is encoded once.

DataGuides (cont)

DataGuides are dynamically generated and maintained over an existing database

Can store statistics in DataGuide For example, the # of atomic objects of each type reachable by p.

DataGuides (example)

When XML gets into Action

Little reminder: Lore first proposal in 1995 XML new standard for data representation and

data exchange over the WWW. Public class XML_data extends

Semi_structured_data Lore among the pioneers to integrate XML in

their DBMS architecture

From Semistructured Data to XML

Data Model

Query Language

DataGuides

Changes in The Data Model

Similar to an OEM, an XML element in Lore is a pair of < EID , VALUE >

EID: is a unique element identifier VALUE: is either an atomic string text or a complex

value containing: A String value: tag XML tag An ordered list of attribute-name/atomic-value An ordered list of crosslink subelements of the form

<label,EID>, reachable via IDREF or IDREFS An ordered list of subelements of the form <label,EID>

Changes in The Data Model (cont)Comments are ignoredWhen an XML document is mapped into

this new data model, it can be seen as a directed labeled graph

Example

Query Language

Extended path expression to distinguish between subelements and attributes, by using qualifiers: DBGroup.Member.>Name &6, use > to

implicitly specify a subelement DBGroup.Member.@Name “Smith”, use @

to implicitly specify an attribute DBGroup.Member.Name &6 “Smith”, when

no @ or > qualifier is used, both attributes and subelements are matched

DataGuides

Provide a DTD from which Lore builds the corresponding DataGuide

Otherwise if no DTD is provided, a DataGuide is generated from the XML document

Problems when updating: With a DTD is provided, validity is assured With no DTD, DataGuide is updated as the XML

document is updated

Conclusion

Lore was originally developed for OEM data model since 1995, XML was integrated later in 1999

Lore Provided a clear and robust solution for storing, querying, and updating semistructured data (XML came after)

The Lore project was declared pretty much out of business in 2000 by The Stanford Database Group

Questions???????