ADVANCED DATABASE INDEXING

The Kluwer International Series on ADVANCES IN DATABASE SYSTEMS

Series Editor Ahmed K. Elmagarmid

Other books in the Series:

Purdue University West Lafayette, IN 47907

MULTILEVEL SECURE TRANSACTION PROCESSING, Vijay Atluri, Sushil Jajodia, Binto George ISBN: 0-7923-7702-8

FUZZY LOGIC IN DATA MODELING, Guoqing Chen ISBN: 0-7923-8253-6 INTERCONNECTING HETEROGENEOUS INFORMATION SYSTEMS, Athman

Bouguettaya, Boualem Benatallah, Ahmed Elmagarmid ISBN: 0-7923-8216-1 FOUNDATIONS OF KNOWLEDGE SYSTEMS: With Applications to Databases

and Agents, Gerd Wagner ISBN: 0-7923-8212-9 DATABASE RECOVERY, Vijay Kumar, Sang H. Son ISBN: 0-7923-8192-0 PARALLEL, OBJECT-ORIENTED, AND ACTIVE KNOWLEDGE BASE

SYSTEMS,Ioannis Vlahavas, Nick Bassiliades ISBN: 0-7923-8117-3 DATA MANAGEMENT FOR MOBILE COMPUTING, Evaggelia Pitoura, George

Samaras ISBN: 0-7923-8053-3 MINING VERY LARGE DATABASES WITH PARALLEL PROCESSING, Alex

A. Freitas, Simon H. Lavington ISBN: 0-7923-8048-7 INDEXING TECHNIQUES FOR ADVANCED DATABASE SYSTEMS, Elisa

Bertino, Beng Chin Ooi, Ron Sacks-Davis, Kian-Lee Tan, Justin Zobel, Boris Shidlovsky, Barbara Catania ISBN: 0-7923-9985-4

INDEX DATA STRUCTURES IN OBJECT-ORIENTED DATABASES, Thomas A. Mueck, Martin L. Polaschek ISBN: 0-7923-9971-4

DATABASE ISSUES IN GEOGRAPHIC INFORMATION SYSTEMS, Nabil R. Adam, Aryya Gangopadhyay ISBN: 0-7923-9924-2

VIDEO DATABASE SYSTEMS: Issues, Products, and Applications, Ahmed K. Elmagarmid, Haitao Jiang, Abdelsalam A. Helal, Anupam Joshi, Magdy Ahmed ISBN: 0-7923-9872-6

REPLICATION TECHNIQUES IN DISTRIBUTED SYSTEMS, Abdelsalam A. Helal, Abdelsalam A. Heddaya, Bharat B. Bhargava ISBN: 0-7923-9800-9

SEARCHING MULTIMEDIA DATABASES BY CONTENT, Christos Faloutsos ISBN: 0-7923-9777-0

TIME-CONSTRAINED TRANSACTION MANAGEMENT: Real-Time Constraints in Database Transaction Systems, Nandit R. Soparkar, Henry F. Korth, Abraham Silberschatz ISBN: 0-7923-9752-5

DATABASE CONCURRENCY CONTROL: Methods, Performance, and Analysis, Alexander Thomasian, IBM T. J. Watson Research Center ISBN: 0-7923-9741-X

ADVANCED DATABASE INDEXING

by

Yannis Manolopoulos Aristotle University, Greece

Yannis Theodoridis Computer Technology Institute, Greece

Vassilis J. Tsotras University of California, Riverside, U.S.A.

~.

" SPRINGER SCIENCE+BUSINESS MEDIA, LLC

Library of Congress Cataloging-in-Publication Data

Manolopoulos, Yannis, 1957-Advanced database indexing I Yannis Manolopoulos, Yannis Theodoridis, Vassilis J. Tsotras.

p. cm. -- (The Kluwer international series on advances in database systems ; 17) Includes bibliographical references and index. ISBN 978-1-4613-4641-8 ISBN 978-1-4419-8590-3 (eBook) DOI 10.1007/978-1-4419-8590-3 1. Database management. 2. Indexing. 1. Theodoridis, Yannis, 1967- II. Tsotras,

Vassilis J., 1961- III. Title. IV. Series.

QA76.9.D3 M3375 1999 005.74' l--dc21

Copyright ® 2000 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 2000 Softcover reprint of the hardcover 1 st edition 2000

99-048329

AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo­copying, record ing, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC

Printed on acid-free paper.

To Paulina, Vassiliki and Helga

for their love and patience

Contents

List of Figures Xl

List of Tables xv

Contributors xvii

Preface XIX

Chapter 1: STORAGE SYSTEMS l. Introduction 1 2. Primary Storage Devices 2 3. Secondary Storage Devices 3 4. Tertiary Storage Devices 8 5. Connecting Storage Together 10 6. Important Issues of Storage Systems 12 7. Alternative Storage Systems 13

8. Future 14 9. Further Reading 15

References 15

Chapter 2: EXTERNAL SORTING 17 l. Introduction 18 2. Run Formation Algorithms 19 3. Merging Algorithms 23 4. Memory Adaptive External Sorting 29 5. Further Reading 34

References 34

V111 ADVANCED DATABASE INDEXING

Chapter 3: FUNDAMENTAL ACCESS METHODS 1. Introduction 2. Basic Indices 3. External Dynamic Hashing 4. Multiattribute Access Methods 5. Document Searching 6. Further Reading

References

Chapter 4: ACCESS METHODS FOR INTERVALS 1. Introduction 2. External Memory Structures for Intervals 3. Further Reading

References

Chapter 5: TEMPORAL ACCESS METHODS 1. Introduction 2. Transaction-time Indexing 3. Bitemporal Indexing 4. Further Reading

References

Chapter 6: SPATIAL ACCESS METHODS 1. Introduction 2. Spatial Indexing Methods 3. Extensions 4. Further Reading

References

Chapter 7: SPATIOTEMPORAL ACCESS METHODS 1. Introduction 2. The Discrete Spatiotemporal Environment 3. The Continuous Spatiotemporal Environment 4. Further Reading

References

Chapter 8: IMAGE AND MULTIMEDIA INDEXING 1. Introduction 2. Spatial Similarity Retrieval 3. Visual Similarity Retrieval 4. Extensions 5. Further Reading

References

Chapter 9: EXTERNAL PERFECT HASHING 1. Introduction

37 37 40 47 53 56 57 57

61 61 69 79 80

83 83 90

109 113 113

117 117 122 134 136 137

141 141 146 152 162 162

167 167 169 174 182 183 184

187 187

Contents

2. Framework and Definitions 3. Perfect Hashing and Performance Characteristics 4. Dynamic External Perfect Hashing 5. Static External Perfect Hashing 6. Performance Comparison 7. Further Reading

References

IX

188 190 192 196 205 205 207

Chapter 10: PARALLEL EXTERNAL SORTING 209 1. Introduction 209 2. Merge-based Parallel Sorting 212 3. Partition-based Parallel Sorting 214 4. Further Reading 216

References 217

Chapter 11: PARALLEL INDEX STRUCTURES 219 1. Introduction 219 2. Declustering Techniques 221 3. Multi-Disk B-trees 224 4. Parallel Linear Quadtrees 226 5. Parallel R-trees 228 6. Parallel S-trees 230 7. Further Reading 232

References 233

Chapter 12: CONCURRENCY ISSUES IN ACCESS METHODS 235 1. Introduction 235 2. Concurrency Control for B+-trees 237 3. Concurrency Control for R-trees 245 4. Concurrency Control for Hash Files 250 5. Further Reading 254

References 255

Chapter 13: LATEST DEVELOPMENTS 259 1. Data Warehouses 259 2. Semistructured Data over the Web 263 3. Main-memory Databases 264 4. Further Reading 266

References 267

Author Index 271

Term Index 279

List of Abbreviations 285

List of Figures

Figure 1.1. Levels of memory hierarchy. 1 Figure 1.2. Organization of a magnetic disk. 3 Figure 1.3. (a) linear, (b) serpentine, (c) helical scan and (d) transverse

tape types. 9 Figure 2.1. 2-way merging of four initial runs. 18 Figure 2.2. Application of first-fit technique. 22 Figure 2.3. Example of 4 runs and 7 buffers. 26 Figure 2.4. Splitting the merging phase. 31 Figure 2.5. Combining merging steps. 31 Figure 3.1. An Indexed Sequential File. 41 Figure 3.2. A B+-tree with q = 5. 44 Figure 3.3. An Extendible Hashing Scheme. 49 Figure 3.4. A Linear Hashing Scheme. 51 Figure 3.5. A Grid File. 54 Figure 4.1. An Interval Tree with U= {1, ... , 8} and n = 5 intervals. 64 Figure 4.2. A Segment Tree with U= {I, ... , 5} and n = 5 intervals. 65 Figure 4.3. An interval I is translated into a point in a 2-dimensional

space. 67 Figure 4.4. A Priority Search Tree. 68 Figure 4.5. An SR-tree. 75 Figure 5.1. An example ofa transaction-time evolution. 86 Figure 5.2. Two valid-time databases. 87 Figure 5.3. A bitemporal database. 89 Figure 5.4. The access forest for a given collection of usefulness

intervals. 95 Figure 5.5. Each page is storing data from a time-key range. 97

xii ADVANCED DATABASE INDEXING

Figure 5.6. An example of the TSB-tree. 99 Figure 5. 7. An example of a plain key-split. 100 Figure 5.8. The Overlapping B-tree. 105 Figure 5.9: Evolution of a set and its linear hashing scheme. 108 Figure 5.10. The bounding-rectangle approach for bitemporal objects. 110 Figure 5.11. The two R-tree methodology for bitemporal data. 110 Figure 6.1. The fundamental spatial data types (points, lines, regions). 118 Figure 6.2. Examples of spatial operators. 119 Figure 6.3. Two-step spatial query processing. 120 Figure 6.4. MBR approximations of objects in Figure 6.2. 121 Figure 6.5. Boundaries and capital locations in Europe. 122 Figure 6.6. The LSD-tree. 124 Figure 6. 7. MBR approximations introduce the notions of dead space

and overlapping. 126 Figure 6.8. The R-tree. 128 Figure 6.9. Access ofR-tree nodes for (a) point and (b) nearest-neigh-

bor queries. 130 Figure 6.10. The Quadtree. 132 Figure 6.11. Object o's MBR meets q's MBR while the covering node

rectangle N does not. 135 Figure 7.1. Moving objects in a 2-dimensional reference space; the

third dimension is time. 145 Figure 7.2. A conceptual view of a discrete spatiotemporal evolution. 146 Figure 7.3. Treating time as another dimension. 148 Figure 7.4. Alive objects are stored as slices at the time they are

inserted. 149 Figure 7.5. The overlapping approach. 151 Figure 7.6. Query in the primal plane. 154 Figure 7. 7. Query in the (a) dual Hough-X and (b) dual Hough-Y

plane. 155 Figure 7.8. Data regions for R- and k-d trees. 157 Figure 8.1. A symbolic image and its 2-D string representation. 169 Figure 8.2. 2-D string indexing. 170 Figure 8.3. Example image and its corresponding ARG [Petrakis and

Faloutsos, 1997]. 171 Figure 8.4. Using R-trees for spatial similarity retrieval. 172 Figure 8.5. Two solutions of a query involving spatial configuration

of three image objects. 173 Figure 8.6. Mapping of images to points in feature space. 174 Figure 8. 7. The X-tree. 176 Figure 8.8. Searching the M-tree. 180 Figure 8.9. Sub-pattern matching. 183

List of Figures X111

Figure 9.1. (a) hashing function with collisions (b) perfect hashing function and (C) minimal perfect hashing function. 189

Figure 9.2. A dynamic external perfect hashing scheme. 193 Figure 9.3. Dependency graph of the set of six words {chef, clean,

sigma, take, taken, tea}. 197 Figure 9.4. Searching step for the key set of interest. 199 Figure 9.5. A 2- dimensional array-based trie of the word set of the

example. 203 Figure 9.6. The Packed Trie array for the key set of the example. 204 Figure 10.1. The three architectures proposed to support a parallel

database system. 210 Figure 10.2.(a) backend sorting and (b) distributed sorting. 212 Figure 1O.3.Example ofa parallel merge-sort. 213 Figure 10.4. Calculation of the exact splitting vector and generation

of fragments. 215 Figure 10.5. Example ofload balance and load imbalance for a sy-

stem with two processors. 216 Figure 11.1.Using multiple disks to store different files. 220 Figure 11.2.A simple tree-based index. 221 Figure 11.3 .Record distribution approach for a three-disk system. 222 Figure 11.4 . Example of a super page, partitioned into four disks. 223 Figure 11.5.A B-tree with four nodes. 224 Figure 11.6. Distribution of a B-tree to 3 disks. The horizontal links

are omitted for clarity. 226 Figure 11.7. A binary image (left) and the corresponding Region

Quadtree (right). 227 Figure 11.8. The shaded region represents a range query, which is

the area of interest. 229 Figure 11.9. Page P4 has been split to P4a and P4b. The proximity index

of the MBR ofP4b with R1, R2, R3 and R4 is calculated. 230 Figure 1l.1O.An S-tree example. 231 Figure 12.1.Search structure before and after a split. 238 Figure 12.2.Node layout ofa Blink_tree. 240 Figure 12.3. Two stages of split in a Blink -tree. 241 Figure 12.4.Node layout for operation specific locking. 243 Figure 12.5.Node of an Rlink_tree. 246 Figure 13.1. The architecture of a data warehouse 260 Figure 13.2.A data cube 261 Figure 13.3.A data integration system. 264 Figure 13.4.(a) A T-tree, (b) a T-tree node. 265

List of Tables

Table 9.1. Set of six words with their random ho, hI and h2 values. 198 Table 9.2. Levels in the Ordering step of the example of the six word

strings. 198 Table 9.3. (a) vertices with their g values, (b) word strings (keys) with

their computed hash addresses 201 Table 9.4. Performance characteristics of hashing techniques that

produce external perfect hashing functions. 206 Table 12.1. Compatibility table for lock modes. 236 Table 12.2. Alternative compatibility table for lock modes. 236 Table 12.3. Performance results for R-trees 249

Contributors

Yannis Manolopoulos was born in Thessaloniki, Greece in 1957. He re­ceived a B.Eng. (1981) in Electrical Eng. and a Ph.D. (1986) in Computer Eng., both from the Aristotle University of Thessaloniki. He has been with the Department of Computer Science of the University of Toronto, the De­partment of Computer Science of the University of Maryland at College Park and the Department of Electrical and Computer Eng. of the Aristotle Univer­sity. Currently, he is Associate Professor at the Department of Informatics of the latter University. He has published over 90 papers in refereed scientific journals and conference proceedings. He is author of two textbooks on data/file structures, which are recommended in the vast majority of the com­puter science/engineering departments in Greece. He has served as PC mem­ber in a number of conferences such as SIGMOD, EDBT, SSD, ADBIS, and ACM-GIS, whereas currently he is member of the Editorial Board of The Computer Journal. His research interests include spatiotemporal databases, databases for web, data mining, data/file structures and algorithms, and per­formance evaluation of secondary and tertiary storage systems.

Yannis Theodoridis was born in Athens, Greece in 1967. He received a B.Eng. (1990) in Electrical Eng. and a Ph.D. (1996) in Electrical and Com­puter Eng., both from the National Technical University of Athens. Cur­rently, he is a Senior Researcher at the Computer Technology Institute (CTI) in Patras, Greece. He has published over 20 papers in refereed scientific journals and conference proceedings, including Algorithmica, IEEE TKDE, ACM Multimedia Journal, ACM SIGMODIPODS Conference. He has served on the PC of the SSD'99 Symposium and the STDBM'99 Workshop. His research interests include spatial and spatiotemporal databases, multime­dia systems, access methods, query optimization, and benchmarking.

XVlll ADVANCED DATABASE INDEXING

Vassilis J. Tsotras was born in Athens, Greece in 1961. He received his B.Eng. in Electrical Eng. from the National Technical University of Athens, Greece (1985), and the M.S., M.Phil. and Ph.D. degrees from Columbia University (1986, 1988, and 1991 respectively). Currently he is an Associate Professor at the Department of Computer Science and Engineering of the University of California, Riverside (UCR). Before joining UCR he was an Associate Professor of Computer and Information Science at Polytechnic University, Brooklyn, NY. His research interests include temporal and spa­tiotemporal databases, access methods, wireless data dissemination and par­allel database systems. He has published over 45 papers in refereed scientific journals and conference proceedings. Dr. Tsotras has served as PC member in various database conferences and workshops, including ICDE'98, SIGMOD' 99, ICDE'OO and EDBT'OO. He was the Program Co-Chair of the 5th International Workshop on Multimedia Information Systems (MIS'99). His research has been funded through grants from the National Science Foundation, DARPA, the Department of Defence, etc. In 1991 he received Natiorial Science Foundation's Research Initiation Award.

Preface

Due to the development of Data Base Management Systems (DBMSs), there is no longer need for application programmers to build their own file sys­tems, but emphasis has moved to design and tuning issues. Thus, physical design (e.g. access methods, query optimization techniques, transaction processing, etc.) might seem to be a stabilized area. For example, the stan­dard "ubiquitous" B-tree is the still dominating structure in virtually all DBMSs, although many elegant variations as well as other robust structures have appeared in the literature. The reason is that industry is not willing to change certain modules, since this costly operation will improve system per­formance only marginally. This way, the price/performance ratio will in­crease instead of decreasing.

However, in modem applications, new data types are required (such as images, text, video, voice, intervals, etc.) except the standard ones (e.g. inte­ger, real, date and the like) which are met in all commercial systems. During the recent years, there is ongoing research in the following areas:

indexing methods for special purpose databases, such as multi­dimensional, spatial, temporal, multimedia, text, object-oriented, or www databases, or databases used for data mining or on-line analytical processmg,

in connection with database architectures, e.g. in client/server, distributed and parallel data­bases, which means that access methods must be distributed or parallel as well.

It is only recently that a "new" data structure, i.e. R-trees, has been integra­ted in certain systems, such as Oracle, InformixlIllustra, 02, etc, almost 10-15 years after its appearance in the literature. Thus, the belief that physical

xx ADVANCED DATABASE INDEXING

level issues have come to a steady state does not seem to be correct any more.

Although transparent for the user of a DBMS, access methods playa key role in database performance. Thus, careful tuning or selection of the appro­priate access method is important in order to develop efficient systems in the present transition era towards object-relational and other special purpose systems. Also, understanding of the state-of-art is essential in order to pro­pose more efficient indexing techniques.

This book may serve as a textbook for graduates specializing in database systems, or database professionals, which are keen to be acquainted with the recent developments. Emphasis has been given on structure description, im­plementation techniques used, and operations performed. Most books in re­lated topics are based on COBOL, PLll, Pascal, C or a pseudo-language. This book uses a simple algorithmic pseudo-language (for some of the ac­cess methods), whereas the interested reader is encouraged to implement some of them. Note, also, that code for certain structures could be found on the Internet.

The book is divided in two parts. The fIrst part consisting of 3 chapters contains some fundamentals, which more-or-Iess may be found in most of the books about fIle structures, physical database design, or computer archi­tecture. It serves as the background knowledge for the second part, which consists of 10 chapters dealing with more advanced material on access methods. Every book chapter ends with references for further reading.

Chapter 1 briefly discusses issues related to storage media, such as mag­netic disks, optical disks and tertiary storage, parallel disks and RAID sys­tems. The next chapter describes external sorting methods, introducing no­tions useful for a chapter in the second part of the book, dedicated to parallel external sorting. The third chapter is about the most important fIle structures, which are currently used in any DBMS, such as B+-trees, Hashing with Chaining, Linear Hashing and Inverted Files, as well as other popular struc­tures such as Grid Files and k-d trees.

The first chapter of the second part, Chapter 4, explains some structures, which are used not to store integers but ranges of integers, e.g. intervals. These structures are Segment Indices, Interval B-trees and External Segment trees. Chapter 5 concerns structures for temporal databases, such as the Snapshot index, Time-Split B-tree, Multiversion B-trees and Overlapping B­trees. In Chapter 6 we examine structures used in spatial databases and GISs. The most well known methods, Quadtrees, R -trees and variants, are exam­ined along with other interesting structures such as LSD-trees, etc. Chapter 7 deals with spatiotemporal data, or in other words, spatial data that evolve over time. Certain indexing techniques based on overlapping and partial per­sistence are described. Chapter 8 examines representations and access meth-

Preface XXI

ods for image and multimedia databases, such as 2-D strings, X-trees, M­trees and R-tree based methods.

Chapter 9 contains material based on hashing. For a number of years, perfect hashing was considered as a method useful exclusively for main memory applications, e.g. for tables with a small number of keywords. Here we will describe two methods, which can apply perfect hashing for very large numbers of records. Chapter 10 considers again the issue of external sorting, which has been examined in the second chapter. However, due to the advent of new architectures and fast disks, a parallel environment is as­sumed, and thus the approach is quite different. Chapter 11 introduces the notion of declustering, i.e. techniques used to distribute a single file structure in several disks. In this context, B-trees, R-trees, Quadtrees and Linear Hashing will be examined. In Chapter 12 we deal with an important issue re­lated to the performance of indices, i.e. concurrency control, and we examine particular techniques such as the Blink-tree and Rlink-tree methods. Finally, the book ends with a chapter dedicated to the newest development in indexing, such as indexing for on-line analytical processing, data warehouses and semistructured data and main-memory databases.

Thanks are due to many friends and colleagues for their help during the various stages of authoring this book. In particular, we would like to thank (in alphabetical order) Robert Alcock, Alex Biliris, Alex Nanopoulos, Nikos Karayannidis, George Kollios, Dimitris Papadias, Apostolos Papadopoulos, Evi Pitoura, Timos Sellis, Eleni Tousidou, Michael Vassilakopoulos, and in particular Theodoros Tzouramanis. Also, Scott Delman and Melissa Fearon of Kluwer Academic Publishers provided invaluable support.

Yannis Manolopoulos

Yannis Theodoridis

Vassilis J. Tsotras