Pro Oracle Database 11 g RAC

on Linux

•$-'-> W>ri% fiiSM

M$ Ills is

Steve Shaw

Martin Bach

Apress8

Contents

M About the Authors x*i

si About the Technical Reviewers xx"

: Acknowledgments xxiii

ii Chapter 1: Introduction 1

Introducing Oracle Real Application Clusters 1

Examining the RAC Architecture 3

Deploying RAC 4

Maintaining High Availability, 5

Defining Scalability 6

Scaling Vertically vs. Horizontally 7

Increasing Manageability 8

Assessing the Cost of Ownership 10

Clustering with Oracle on Linux 13

Running Linux on Oracle 16

Understanding the Role of Unix 16

Liberating Software 17

Developing Linux 18

Expanding the Concept of Free with Open Source 19

Combining Oracle, Open Source, and Linux 20

Drilling Down on Unbreakable Linux 21

Creating and Growing Red Hat Enterprise Linux 22

Extending Red Hat with Oracle Enterprise Linux 23

Drilling Down on SuSE Linux Enterprise Server 24

v

« CONTENTS

Taking Linux to Asia 25

Summary 25

li Chapter 2: RAC Concepts 27

Clustering Concepts 27

Configuring Active/active Clusters 27

Implementing Active/passive Clusters 28

Configuring a Shared-All Architecture 28

Configuring a Shared-Nothing Architecture 29

Exploring the Main RAC Concepts 29

Working with Cluster Nodes 29

Leveraging the Interconnect 30

Clusterware/Grid Infrastructure 31

Leveraging Automatic Storage Management 39

Installing Real Application Clusters 44

Using the Global Resource Directory (GRD) 49

Transferring Data Between Instances with Cache Fusion 51

Achieving Read Consistency 52

Synchronizing System Change Numbers 52

Exploring the New Features of 11 g Release 2 52

Leveraging Grid Plug and Play 53

Modeling Resources with Server Pools 55

Ensuring POSIX Compliance with ACFS 56

Using Oracle Restart Instead of RAC 57

Simplifying Clusterd Database Access with SCAN Listener 59

Summary 60

li Chapter 3: RAC Architecture 63

Availability Considerations 63

Deciding the Number of Nodes 65

v

CONTENTS

Online Maintenance and Patching 67

Instance Recovery in RAC 72

Failover Considerations 74

Transparent Application Failover 75

Fast Connection Failover and Fast Application Notification 76

Scalability Considerations 77

Scalability Enhancers 78

Scalability Inhibitors 79

Standby Databases 81

Introduction to Oracle Standby Databases 82

Types of Standby Database 83

Active Data Guard 85

Extended Distance Clusters 90

Oracle Streams 91

Streams Processing 92

Oracle Streams Prerequisites.., 93

Cluster Topologies 94

Summary 95

a Chapter 4: Hardware 97

Oracle Availability 98

Server Processor Architecture 99

x86 Processor Fundamentals 99

Multicore Processors and Hyper-Threading 103

CPU Cache 106

CPU Power Management 109

Virtualization 111

Memory 112

Virtual Memory 11 2

<a CONTENTS

Physical Memory 113

NUMA 116

Memory Reliability 125

Additional Platform Features 125

Onboard RAID Storage 126

Machine Check Architectures 126

Remote Server Management and IPMI 127

Network Interconnect Technologies 127

Server I/O 128

Private Interconnect 131

Storage Technologies 136

RAC I/O Characteristics 137

Hard Disk and Solid State Disk Drive Performance 143

RAID 147

Storage Protocols for Linux 153

Summary 164

Chapter 5: Virtualization 165

Virilization Definition and Benefits 165

Oracle VM 168

Oracle VM Server Architecture 168

Oracle VM Design 174

Oracle VM Server Installation 178

Oracle VM Manager Installation 183

Oracle VM CU Installation and Configuration 186

Configuring Oracle VM 187

Network Configuration 187

Server Pool Configuration 192

Installing and Configuring Guests 208

vii

* CONTENTS

Importing a Template 209

Creating a Guest from a Template 210

Accessing a Guest 212

Configuring a Guest for RAC 214

Managing Domains 216

Oracle VM Agent 216

Oracle VM Manager 218

Oracle VM Manager CLI 220

The xm Command-Line Interface 222

Summary 230

if Chapter 6: Linux Installation and Configuration 231

Selecting the Right Linux Software 231

Reviewing the Hardware Requirements 232

Drilling Down on Networking Requirements 233

Configuring a GNS or a Manual IP 233

Configuring DNS and DHCP 236

Downloading the Linux Software 243

Preparing for a Network Install 243

Installing Oracle Enterprise Linux 5 247

Starting the Installation 247

Installation Media Check 247

Anaconda Installation 247

Install or Upgrade 248

Disk Partitioning 248

Configuring the Boot Loader and Network 259

Selecting a Time Zone 260

Configuring the Root Password 261

Reviewing the Package Installation Defaults 261

viii

a CONTENTS

Selecting a Package Group... 261

Installing Packages 263

Setting the Final Configuration 263

Accepting the License Agreement 263

Configuring the Firewall 263

Configuring SELinux 263

Enabling kdump 264

Setting the Date and Time 264

Creating a User 265

Installing Additional CDs 265

Configuring Oracle Enterprise Linux 5 265

Configuring a Server with the Oracle Validated RPM 266

Verifying the Oracle Validated RPM Actions 270

Post Oracle Validated RPM Configuration 282

Completing the Linux Configuration for RAC 292

Configuring Shared Storage 298

Discovering and Configuring SAN Disk 299

Network Channel Bonding 313

I/O Fencing with IPMI 317

Summary 322

ii Chapter 7: Grid Infrastructure Installation 323

Getting Ready for Installation 323

Obtain Software Distribution 323

Configure X Environment 324

Determining Configuration Type 327

Advanced Installation - Manual Configuration 327

Network Configuration 328

DNS Configuration 329

Choosing an Installation Option 330

ix

a CONTENTS

Selecting an Advanced or Typical Installation Type 332

Selecting a Language 333

Configuring the Grid Plug and Play 334

Configuring the Cluster Node Information Page 336

Configuring the Network Interface Usage Page 337

Configuring the Storage Option Information Page 338

Creating an ASM Disk Group 340

Specifying an ASM Password 341

Specifying a Username and Password for IPMI 342

Configuring Privileged Operating System Groups 342

Setting the Installation Location,

344

Specify the Central Inventory's Location 345

Performing Prerequisite Checks 345

Reviewing the Summary Page 351

Setup Page 352

Reviewing Execute Configuration Scripts 352

Monitoring Configuration Assistants 359

Implementing an Advanced Installation for Automatic Configuration 360

Configuring a Network Configuration 360

Configuring DNS 362

Configuring DHCP 363

Setting up the Grid Plug and Play information Page 364

Configuring the Cluster Node Information Page 365

The Summary Page 366

Typical Installation 367

Choosing the Installation Type 367

Specifying the Cluster Configuration Page 368

Install Locations Page 369

Reviewing the Summary Page for a Typical Installation 370

x

CONTENTS

Installing a Standalone Server 371

Selecting an Installation Option 372

Creating an ASM Disk Group Page 373

Reviewing the Summary Page for a Standalone Installation 373

Configuring the Execute Configuration Scripts 375

Deinstalling the Grid Infrastructure Software 376

Summary 377

n Chapter 8: Clusterware .........379

Introducing Clusterware 379

Examining the Hardware and Software Requirements ,380

Using Shared Storage with Oracle Clusterware 381

Storing Cluster Information with the Oracle Cluster Registry 381

Storing Information in the Oracle Local Registry 382

Fencing with the Voting Disk 382

Recording Information with the Grid Plug and Play Profile 383

Using Background Processes 384

Grid Infrastructure Software Stacks 384

Drilling Down on the High Availability Stack 385

Drilling Down on the Cluster Ready Services Stack 386

Using Grid Infrastructure Agents 388

Initiating the Startup Sequence 389

Managing Oracle Clusterware 391

Using the Enterprise Manager 392

Using the Clusterware Control Utility 392

Managing Resources with srvctl 395

Verifying the Cluster with the CVU 396

Configuring Network Interfaces with oifcfg 400

Administering the OCR and OLR with ocrconfig 400

xi

• CONTENTS

Checking the State of the OCR and its Mirrors with ocrcheck 400

Dumping Contents of the OCR with ocrdump 400

Defining Server-Side Callouts 401

Protecting Applications with Clusterware 403

Managing Resource Profiles 403

Configuring Active/Passive Clustering for Oracle Database 404

Configuring Active/Passive Clustering for Apache Tomcat 409

Using Oracle Restart 413

Troubleshooting 415

Resolving Startup Issues 415

Resolving Problems with Java Utilities 422

Patching Grid Infrastructure 422

Adding and Deleting Nodes 426

Adding Nodes 426

Deleting Nodes 433

Exploring More Advanced Topics 438

Selecting non-Default Listener Ports 439

Selecting a non-Default SCAN Listener Endpoint 442

Changing the SCAN After Installation 443

Maintaining Voting Disks 444

Maintaining Local and Cluster Registry 448

Summary 453

..- Chapter 9: Automatic Storage Management 455

Introducing ASM 455

ASM Terminology 456

Supported File Types 457

ASM Management 458

ASM and RDBMS Support 458

xii

a CONTENTS

ASM Installation 459

ASM Components and Concepts 459

ASM Instances 459

Failure Groups 464

ASM Files 465

Redundancy 468

Striping 468

Mirroring 469

Intelligent Data Placement 469

Access Control 470

Maintaining ASM 475

Creating an ASM Disk Group 475

Extending an ASM Disk Group 478

Dropping Disks from an ASM Disk Group 479

Enabling Disk Discovery 480

Understanding the ASM Header 480

Installing the Grid Infrastructure, 481

Re-creating the ASM Disks 482

ASM Cluster File System 482

Creating and Mounting an ACFS Using ASMCA 484

Creating and Mounting an ACFS Using the Command Line 491

Maintaining the ACFS 494

Using ACFS with Oracle Restart 496

Administering ASM 496

Using SQLTIus to Administer ASM 497

ASM Administration Using SRVCTL 499

Accessing Files in ASM 500

Using Files Instead of Devices 501

Virilization and Shared Disks 502

xiii

a CONTENTS

Summary 503

B Chapter 10: RDBMS Installation and Configuration 505

Installing the RAC Software 505

Start the Installer 505

Configuring the Security Updates Page 506

Configuring the Installation Options Page 506

Configuring the Node Selection Page 507

Configuring the Product Language Selection Page 508

Configuring the Database Editions Page 509

Configuring the Installation Locations Page 510

Configuring the Privileged Operating Systems Group Page 511

Configuring the Prerequisites Check Page 512

Reviewing the Summary Page 512

Executing Configuration Scripts 513

Using the Database Configuration Assistant (DBCA) 514

Starting the DBCA and Choosing an Operation 514

Creating a Database 516

Reviewing the Summary Page 535

Configuring the Database Options 536

Deleting a Database 538

Managing Templates 539

Building Database Creation Scripts 539

Setting up Admin-Managed Database Scripts 540

Building Policy-Managed Database Scripts , 552

Deinstalling the RDBMS Software 555

Summary 557

i Chapter 11: Workload Management.... 559

Introducing Services 559

xiv

CONTENTS

Creating an Administrator Managed Database vs. Policy-Managed Database 560

Managing Services with the Database Scheduler,

561

Using Services with Shared Server 563

Managing Services,

564

Managing Services with SRVCTL 564

Managing Services with Enterprise Manager 569

Managing Services with DBMS_SERVICE 572

Balancing the Workload 572

Configuring Client-Side Load Balancing 573

Configuring Server-Side Load Balancing 574

Exploring the Load Advisory Framework 576

Using Transparent Application Failover 577

Implementing Fast Connection Failover 584

Using the Resource Manager 597

Caging an Instance 600

Database Resident Connection Pool 601

Summary 604

Chapter 12: Oracle Performance Monitoring ...607

Enterprise Manager Database Control 608

The Cluster Tab 609

The Database Tab 611

The Performance Tab 611

AWR Reports 613

Interpreting the RAC Statistics of an AWR Report 617

Top 5 Timed Foreground Events 618

Global Cache Load Profile 619

Global Cache Efficiency Percentages 619

Global Cache and Enqueue Services - Workload Characteristics 619

xv

CONTENTS

Global Cache and Enqueue Services - Messaging Statistics 620

Cluster Interconnect 620

Foreground Wait Class 621

Wait Event Histogram 621

"SQL Statement" Sections 622

RAC-Related Segment Statistics 622

Dictionary Cache Stats (RAC) 623

Library Cache Activity (RAC) 623

Global Messaging Statistics 624

Global CR Served Statistics 624

Global Current Served Statistics 624

Global Cache Transfer Statistics 625

Interconnect Statistics 626

Dynamic Remastering Statistics 626

Active Session History 627

Automatic Database Diagnostic Monitor 629

Executing an ADDM Report 629

Controlling ADDM 629

The Report Format 631

AWR SQL Report 631

Performance Monitoring Using SQL*Plus 632

GV$ Views 633

System Statistics 633

Segment Statistics 633

Global Caches Services: Consistent and Current Reads 635

Global Cache Services: Current Block Activity 637

Global Enqueue Service 640

Library Cache 641

Dictionary Cache 642

xvi

s CONTENTS

Lock Conversions 642

Automatic Diagnostic Repository 644

Summary 652

Chapter 13: Linux Performance Monitoring ....653

The uptime and last Commands 653

The ps Command 654

free, ipcs, pmap, and Isof 655

The free Command 655

The /proc File System 656

The /sys/devices/system/node File System 657

The ipcs Command 658

The pmap Command 658

The Isof Command 660

top 660

vmstat-

662

strace 663

netstat, ss, and tcpdump 664

Looking at Interface Statistics 664

Summary Statistics 665

Listening Socket Statistics 665

Looking up Weil-Known Ports 666

Reporting on Socket Statistics Using ss 666

Capturing and Displaying Network Packets 667

iostat 668

mpstat 669

sar and kSar 670

Configuring sar 670

Invoking sar Directly 671

xvii

CONTENTS

Graphing the Results 672

Oracle Cluster Health Monitor 674

Installing the Oracle Cluster Health Monitor 674

Starting and Stopping the Oracle Cluster Health Monitor 677

Understanding the Architecture 678

Installing the Client-Side GUI 678

Viewing Current and Captured Activity 679

OSWatcher 680

Installing OSWatcher 680

Starting OSWatcher 681

Stopping OSWatcher 681

Viewing Results Graphically 682

nmon...,,

683

Summary 685

Chapter 14: Parallel Execution 687

Parallel Execution Concepts 688

Serial Execution 688

Parallel Execution 689

Producers and Consumers 691

Bloom Filters 696

Partitioning 698

Parallel Execution Configuration 700

clusterjnterconnects 700

db_block_size, db_cache_size, and db_file_multiblock_read_count 700

instance_groups and parallel_instance_group 701

large_pool_size, parallel_execution_message_size, and shared_pool_size 702

parallel_adaptive_multijjser 702

parallel_automatic_tuning 703

xviii

i CONTENTS

parallel_degree_limit 703

parallel_degree_policy, paralleLminJimeJhreshold, and parallel_servers_target 704

paralleljorcejocal 707

paralleljo_cap_enabled 707

paralleLmax_servers, parallel_min_servers, parallel_mreads_per_cpu, and processes 708

parallel_min_percent 708

pga_aggregate_target 709

Parallel Execution Performance... 709

AWR Reports 709

SQL*Plus 713

Trace Files 714

Summary 715

ft Chapter 15: Upgrading to Oracle 11^ Release 2 , ..717

Upgrading Grid Infrastructure Components 717

Installing the Prerequisites 718

Running the Installer 719

Specifying Options 720

Running the Upgrade 725

Upgrading RAC Software 729

Running the Installer 730

Running Configuration Scripts 732

Preparing for the Database Upgrade 734

Identifying the Upgrade Path 734

Determine Upgrade Method 735

Testing the Upgrade Process 735

Running the pre-Upgrade Tool 736

Performing Other Checks 741

Saving Current Database Parameters 741

xix

a CONTENTS

Backing up the Database 742

Configuring the Listener Process 743

Upgrading Automatically with DBUA 743

Upgrading a Database Manually 752

Preparing the Parameter Files 754

Preparing Password Files 755

Modifying the Initialization Parameters 755

Restarting the Database in UPGRADE Mode 755

Running the Catalog Upgrade Script 755

Configuring SPFILE 756

Running the post-Upgrade Status Tool 757

Running post-Upgrade Scripts 758

Recompiling Invalid Packages 760

Updating /etc/oratab 762

Updating Environment Variables 762

Updating the Oracle Cluster Registry 762

Setting the Initialization Parameters for the New Release 763

Performing the Necessary post-Upgrade Steps 764

Completing Mandatory post-Upgrade Tasks 764

Performing the Recommended Tasks 765

Resolving Problems in Mixed-Database Environments 767

Using a Swing Kit, 768

Summary 769

3 Index 771

XX