Oracle Exadata Recipes

A Problem-Solution Approach

John Clarke

Apress-

Contents

About the Author xxxiii

About the Technical Reviewer xxxv

Acknowledgments xxxvii

Introduction xxxix

Part 1: Exadata Architecture 1

Chapter 1: Exadata Hardware 3

1-1. Identifying Exadata Database Machine Components 3

Problem 3

Solution 3

How It Works 7

1-2. Displaying Storage Server Architecture Details 10

Problem 10

Solution 10

How It Works 12

1-3. Displaying Compute Server Architecture Details 14

Problem 14

Solution 14

How It Works 16

1-4. Listing Disk Storage Details on the Exadata Storage Servers 17

Problem 17

Solution 17

How It Works 21

vii

CONTENTS

1-5. Listing Disk Storage Details on the Compute Servers 23

Problem 23

Solution 23

How It Works 27

1-6. Listing Flash Storage on the Exadata Storage Servers 27

Problem 27

Solution 27

How It Works 29

1-7. Gathering Configuration Information for the InfiniBandSwitches 29

Problem 29

Solution 29

How It Works 32

Chapter 2: Exadata Software 33

2-1. Understanding the Role of Exadata Storage Server Software 33

Problem 33

Solution 33

How It Works 35

2-2. Validating Oracle 11gR2 Databases on Exadata 35

Problem 35

Solution 35

How It Works 38

2-3. Validating Oracle 11gR2 Grid Infrastructure on Exadata 38

Problem 38

Solution 38

How It Works 41

2-4. Locating the Oracle Cluster Registry and Voting Disks on Exadata 42

Problem 42

Solution 42

How It Works 43

viii

CONTENTS

2-5. Validating Oracle 11gR2 Real Application Clusters Installation and Database

Storage on Exadata 44

Problem 44

Solution 44

How It Works 47

2-6. Validating Oracle 11gR2 Real Application Clusters Networking on Exadata 47

Problem 47

Solution 47

How It Works 52

a Chapter 3: How Oracle Works on Exadata 53

3-1. Mapping Physical Disks, LUNs, and Cell Disks on the Storage Servers 53

Problem 53

Solution 53

How It Works 56

3-2. Mapping ASM Disks, Grid Disks, and Cell Disks 58

Problem 58

Solution 58

How It Works 60

3-3. Mapping Flash Disks to Smart Flash Storage 61

Problem 61

Solution 61

How It Works 62

3-4. Identifying Cell Server Software Processes 62

Problem 62

Solution 63

How It Works 63

3-5. Tracing Oracle I/O Requests on Exadata Compute Nodes 65

Problem 65

Solution 65

How It Works 65

ix

CONTENTS

3-6. Validating That Your Oracle RAC Interconnect Is Using InfiniBand 68

Problem 68

Solution 68

How It Works 70

3-7. Tracing cellsrv on the Storage Servers 71

Problem 71

Solution 71

How It Works 73

Part 2: Preparing for Exadata 75

Chapter 4: Workload Qualification 77

4-1. Quantifying I/O Characteristics of Your Current Database 77

Problem 77

Solution 77

How It Works 80

4-2. Conducting a Smart Scan Fit Analysis Using AWR 81

Problem 81

Solution 81

How It Works 85

4-3. Conducting a Smart Scan Fit Analysis Using Exadata Simulation 87

Problem 87

Solution 87

How It Works 91

4-4. Performing a Hybrid Columnar Compression Fit Assessment 93

Problem 93

Solution 93

How It Works 95

Chapter 5: Sizing Exadata 97

5-1. Determining CPU Requirements 98

Problem 98

Solution 98

How It Works 107

x

CONTENTS

5-2. Determining lOPs Requirements 108

Problem 108

Solution 108

How It Works 112

5-3. Determining I/O Bandwidth Requirements 113

Problem 113

Solution 113

How It Works 118

5-4. Determining ASM Redundancy Requirements 119

Problem 119

Solution 119

How It Works 119

5-5. Forecasting Storage Capacity 122

Problem 122

Solution 122

How It Works 124

5-6. Planning for Database Growth 125

Problem 125

Solution 125

How It Works 133

5-7. Planning for Disaster Recovery 133

Problem 133

Solution 133

How It Works 133

5-8. Planning for Backups 134

Problem 134

Solution 134

How It Works 135

5-9. Determining Your Fast Recovery Area and RECO Disk Group Size Requirements 136

Problem 136

Solution 136

How It Works 138

xi

CONTENTS

Chapter 6: Preparing for Exadata 139

6-1. Planning and Understanding Exadata Networking 139

Problem 139

Solution 139

How It Works 141

6-2. Configuring DNS 143

Problem 143

Solution 143

How It Works 144

6-3. Running checkip.sh 144

Problem 144

Solution 144

How It Works 147

6-4. Customizing Your InfiniBand Network Configuration 147

Problem 147

Solution 147

How It Works 147

6-5. Determining Your DATA and RECO Storage Requirements 148

Problem 148

Solution 148

How It Works 149

6-6. Planning for ASM Disk Group Redundancy 149

Problem 149

Solution 149

How It Works 152

6-7. Planning Database and ASM Extent Sizes 152

Problem 152

Solution 152

How It Works 152

xii

CONTENTS

6-8. Completing the Pre-Delivery Survey 153

Problem 153

Solution 153

How It Works 154

6-9. Completing the Configuration Worksheet 154

Problem 154

Solution 154

How It Works 156

Part 3: Exadata Administration 157

Chapter 7: Administration and Diagnostics Utilities 159

7-1. Logging in to the Exadata Compute and Storage Cells Using SSH 159

Problem 159

Solution 159

How It Works 160

7-2. Configuring SSH Equivalency 161

Problem 161

Solution 161

How It Works 162

7-3. Locating Key Configuration Files and Directories on the Cell Servers 162

Problem 162

Solution 162

How It Works 163

7-4. Locating Key Configuration Files and Directories on the Compute Nodes 164

Problem 164

Solution 164

How It Works 164

7-5. Starting and Stopping Cell Server Processes 165

Problem 165

Solution 165

How It Works 166

xiii

CONTENTS

7-6. Administering Storage Cells Using CellCLI 167

Problem 167

Solution 167

How It Works 168

7-7. Administering Storage Cells Using deli 168

Problem 168

Solution 168

How It Works 170

7-8. Generating Diagnostics from the ILOM Interface 171

Problem 171

Solution 171

How It Works 173

7-9. Performing an Exadata Health Check Using exachk 173

Problem 173

Solution 173

How It Works 176

7-10. Collecting Compute and Cell Server Diagnostics Using the sundiag.sh Utility 176

Problem 176

Solution 176

How It Works 177

7-11. Collecting RAID Storage Information Using the MegaCLI utility 178

Problem 178

Solution 178

How It Works 179

7-12. Administering the Storage Cell Network Using ipconf 179

Problem 179

Solution 179

How It Works 180

7-13. Validating Your InfiniBand Switches with the CheckSWProfile.sh Utility 181

Problem 181

Solution 181

How It Works 182

xiv

CONTENTS

7-14. Verifying Your InfiniBand Network Topology 182

Problem 182

Solution 182

How It Works 183

7-15. Diagnosing Your InfiniBand Network 183

Problem 183

Solution 183

How It Works 187

7-16. Connecting to Your Cisco Catalyst 4948 Switch and Changing Switch Configuration 187

Problem 187

Solution 187

How It Works 188

Chapter 8: Backup and Recovery 189

8-1. Backing Up the Storage Servers 189

Problem 189

Solution 189

How It Works 190

8-2. Displaying the Contents of Your CELLBOOT USB Flash Drive 191

Problem 191

Solution 191

How It Works 192

8-3. Creating a Cell Boot Image on an External USB Drive 192

Problem 192

Solution 193

How It Works 194

8-4. Backing Up Your Compute Nodes Using Your Enterprise Backup Software 195

Problem 195

Solution 195

How It Works 196

XV

CONTENTS

8-5. Backing Up the Compute Servers Using LVM Snapshots 196

Problem 196

Solution 197

How It Works 200

8-6. Backing Up Your Oracle Databases with RMAN 201

Problem 201

Solution 201

How It Works 201

8-7. Backing Up the InfiniBand Switches 202

Problem 202

Solution 203

How It Works 204

8-8. Recovering Storage Cells from Loss of a Single Disk 205

Problem 205

Solution 205

How It Works 208

8-9. Recovering Storage Cells from Loss of a System Volume Using CELLBOOT Rescue 210

Problem 210

Solution 210

How It Works 221

8-10. Recovering from a Failed Storage Server Patch 222

Problem 222

Solution 222

How It Works 223

8-11. Recovering Compute Server Using LVM Snapshots 223

Problem 223

Solution 223

How It Works 228

8-12. Reimaging a Compute Node 229

Problem 229

Solution 229

How It Works 231

xvi

CONTENTS

8-13. Recovering Your InfiniBand Switch Configuration 232

Problem 232

Solution 232

How It Works 234

8-14. Recovering from Loss of Your Oracle Cluster Registry and Voting Disks 235

Problem 235

Solution 235

How It Works 237

Chapter 9: Storage Administration 239

9-1. Building ASM Disk Groups on Exadata 239

Problem 239

Solution 239

How It Works 245

9-2. Properly Configuring ASM Disk Group Attributes on Exadata 246

Problem 246

Solution 246

How It Works 247

9-3. Identifying Unassigned Grid Disks 248

Problem 248

Solution 248

How It Works 248

9-4. Configuring ASM Redundancy on Exadata 248

Problem 248

Solution 248

How It Works 251

9-5. Displaying ASM Partner Disk Relationships on Exadata 253

Problem 253

Solution 253

How It Works 254

xvii

CONTENTS

9-6. Measuring ASM Extent Balance on Exadata 254

Problem 254

Solution 254

How It Works 258

9-7. Rebuilding Cell Disks 258

Problem 258

Solution 258

How It Works 273

9-8. Creating Interleaved Cell Disks and Grid Disks 274

Problem 274

Solution 274

How It Works 280

9-9. Rebuilding Grid Disks 282

Problem 282

Solution 282

How It Works 284

9-10. Setting smart_scan_capable on ASM Disk Groups 286

Problem 286

Solution 286

How It Works 291

9-11. Creating Flash Grid Disks for Permanent Storage 292

Problem 292

Solution 292

How It Works 296

Chapter 10: Network Administration 297

10-1. Configuring the Management Network on the Compute Nodes 297

Problem 297

Solution 297

How It Works 299

xviii

CONTENTS

10-2. Configuring the Client Access Network 299

Problem 299

Solution 299

How It Works 301

10-3. Configuring the Private Interconnect on the Compute Nodes 301

Problem 301

Solution 302

How It Works 303

10-4. Configuring the SCAN Listener 304

Problem 304

Solution 304

How It Works 305

10-5. Managing Grid Infrastructure Network Resources 306

Problem 306

Solution 306

How It Works 309

10-6. Configuring the Storage Server Ethernet Network 309

Problem 309

Solution 309

How It Works 311

10-7. Changing IP Addresses on Your Exadata Database Machine 312

Problem 312

Solution 312

How It Works 330

Chapter 11: Patching and Upgrades 331

11-1. Understanding Exadata Patching Definitions, Alternatives, and Strategies 331

Problem 331

Solution 331

How It Works 333

xix

CONTENTS

11 -2. Preparing to Apply Exadata Patches 333

Problem 333

Solution 333

How It Works 335

11-3. Patching Your Exadata Storage Servers 335

Problem 335

Solution 335

How It Works 341

11-4. Patching Your Exadata Compute Nodes and Databases 342

Problem 342

Solution 342

How It Works 349

11-5. Patching the InfiniBand Switches 349

Problem 349

Solution 349

How It Works 352

11-6. Patching Your Enterprise Manager Systems Management Software 352

Problem 352

Solution 352

How It Works 353

HChapter 12: Security 355

12- 1. Configuring Multiple Oracle Software Owners on Exadata Compute Nodes 355

Problem 355

Solution 355

How It Works 358

12-2. Installing Multiple Oracle Homes on Your Exadata Compute Nodes 359

Problem 359

Solution 359

How It Works 361

XX

CONTENTS

12-3. Configuring ASM-Scoped Security 362

Problem 362

Solution 362

How It Works 364

12-4. Configuring Database-Scoped Security 364

Problem 364

Solution 364

How It Works 367

HPart 4: Monitoring Exadata 369

HChapter 13: Monitoring Exadata Storage Cells 371

13-1. Monitoring Storage Cell Alerts 371

Problem 371

Solution 371

How It Works 375

13-2. Monitoring Cells with Active Requests 376

Problem 376

Solution 376

How It Works 380

13-3. Monitoring Cells with Metrics 381

Problem 381

Solution 381

How It Works 386

13-4. Configuring Thresholds for Cell Metrics 389

Problem 389

Solution 389

How It Works 390

13-5. Using deli with Special Characters 391

Problem 391

Solution 391

How It Works 393

xxi

CONTENTS

13-6. Reporting and Summarizing metrichistory Using R 393

Problem 393

Solution 393

How It Works 398

13-7. Reporting and Summarizing metrichistory Using Oracle and SQL 399

Problem 399

Solution 399

How It Works 400

13-8. Detecting Cell Disk I/O Bottlenecks 400

Problem 400

Solution 400

How It Works 402

13-9. Measuring Small I/O vs. Large I/O Requests 402

Problem 402

Solution 402

How It Works 404

13-10. Detecting Grid Disk I/O Bottlenecks 404

Problem 404

Solution 404

How It Works 405

13-11 Detecting Host Interconnect Bottlenecks 406

Problem 406

Solution 406

How It Works 407

13-12. Measuring I/O Load and Waits per Database, Resource Consumer Group,

and Resource Category 407

Problem 407

Solution 407

How It Works 409

xxii

CONTENTS

Chapter 14: Host and Database Performance Monitoring 411

14-1. Collecting Historical Compute Node and Storage Cell Host Performance Statistics 411

Problem 411

Solution 411

How It Works 413

14-2. Displaying Real-Time Compute Node and Storage Cell Performance Statistics 414

Problem 414

Solution 414

How It Works 417

14-3. Monitoring Exadata with Enterprise Manager 417

Problem 417

Solution 417

How It Works 421

14-4. Monitoring Performance with SQL Monitoring 421

Problem 421

Solution 421

How It Works 424

14-5. Monitoring Performance by Database Time 425

Problem 425

Solution 425

How It Works 431

14-6. Monitoring Smart Scans by Database Time and AAS 432

Problem 432

Solution 432

How It Works 434

14-7. Monitoring Exadata with Wait Events 435

Problem 435

Solution 435

How It Works 439

xxiii

CONTENTS

14-8. Monitoring Exadata with Statistics and Counters 440

Problem 440

Solution 440

How It Works 441

14-9. Measuring Cell I/O Statistics for a SQL Statement 442

Problem 442

Solution 442

How It Works 444

Part 5: Exadata Software 445

Chapter 15: Smart Scan and Cell Offload 447

15-1. Identifying Cell Offload in Execution Plans 447

Problem 447

Solution 447

How It Works 448

15-2. Controlling Cell Offload Behavior 450

Problem 450

Solution 450

How It Works 451

15-3. Measuring Smart Scan with Statistics 451

Problem 451

Solution 451

How It Works 455

15-4. Measuring Offload Statistics for Individual SQL Cursors 455

Problem 455

Solution 455

How It Works 457

15-5. Measuring Offload Efficiency 457

Problem 457

Solution 457

How It Works 460

xxiv

CONTENTS

15-6. Identifying Smart Scan from 10046 Trace Files 461

Problem 461

Solution 461

How It Works 462

15-7. Qualifying for Direct Path Reads 463

Problem 463

Solution 463

How It Works 468

15-8. Influencing Exadata's Decision to Use Smart Scans 469

Problem 469

Solution 469

How It Works 473

15-9. Identifying Partial Cell Offload 473

Problem 473

Solution 473

How It Works 474

15-10. Dealing with Fast Object Checkpoints 475

Problem 475

Solution 475

How It Works 475

Chapter 16: Hybrid Columnar Compression 477

16-1. Estimating Disk Space Savings for HCC 477

Problem 477

Solution 477

How It Works 479

16-2. Building HCC Tables and Partitions 479

Problem 479

Solution 479

How It Works 482

XXV

CONTENTS

16-3. Contrasting Oracle Compression Types 482

Problem 482

Solution 482

How It Works 483

16-4. Determining the Compression Type of a Segment 487

Problem 487

Solution 487

How It Works 488

16-5. Measuring the Performance Impact of HCC for Queries 488

Problem 488

Solution 489

How It Works 491

16-6. Direct Path Inserts into HCC Segments 492

Problem 492

Solution 492

How It Works 493

16-7. Conventional Inserts to HCC Segments 493

Problem 493

Solution 494

How It Works 495

16-8. OMLand HCC 495

Problem 495

Solution 495

How It Works 501

16-9. Decompression and the Performance Impact 502

Problem 502

Solution 502

How It Works 504

xxvi

CONTENTS

Chapter 17: I/O Resource Management and Instance Caging 505

17-1. Prioritizing I/O Utilization by Database 505

Problem 505

Solution 505

How It Works 507

17-2. Limiting I/O Utilization for Your Databases 507

Problem 507

Solution 507

How It Works 508

17-3. Managing Resources within a Database 508

Problem 508

Solution 508

How It Works 510

17-4. Prioritizing I/O Utilization by Category of Resource Consumers 511

Problem 511

Solution 511

How It Works 512

17-5. Prioritizing I/O Utilization by Categories of Resource Consumers and Databases 513

Problem 513

Solution 513

How It Works 514

17-6. Monitoring Performance When IORM Is Enabled 517

Problem 517

Solution 517

How It Works 520

17-7. Obtaining IORM Plan Information 521

Problem 521

Solution 521

How It Works 525

xxvii

CONTENTS

17-8. Controlling Smart Flash Cache and Smart Flash Logging with IORM 526

Problem 526

Solution 526

How It Works 526

17-9. Limiting CPU Resources with Instance Caging 527

Problem 527

Solution 527

How It Works 527

(Chapter 18: Smart Flash Cache and Smart Flash Logging 529

18-1. Managing Smart Flash Cache and Smart Flash Logging 529

Problem 529

Solution 530

How It Works 531

18-2. Determining Which Database Objects Are Cached 532

Problem 532

Solution 532

How It Works 533

18-3. Determining What's Consuming Your Flash Cache Storage 534

Problem 534

Solution 534

How It Works 536

18-4. Determining What Happens When Querying Uncached Data 536

Problem 536

Solution 536

How It Works 540

18-5. Measuring Smart Flash Cache Performance 541

Problem 541

Solution 541

How It Works 546

xxviii

CONTENTS

18-6. Pinning Specific Objects in Smart Flash Cache 546

Problem 546

Solution 546

How It Works 548

18-7. Quantifying Benefits of Smart Flash Logging 549

Problem 549

Solution 549

How It Works 550

S Chapter 19: Storage Indexes 553

19-1. Measuring Performance Impact of Storage Indexes 553

Problem,

553

Solution 553

How It Works 555

19-2. Measuring Storage Index Performance with Not-So-Well-Ordered Data 557

Problem 557

Solution 557

How It Works 559

19-3. Testing Storage Index Behavior with Different Query Predicate Conditions 559

Problem 559

Solution 560

How It Works 564

19-4. Tracing Storage Index Behavior 564

Problem 564

Solution 565

How It Works 568

19-5. Tracing Storage Indexes When More than Eight Columns Are Referenced 568

Problem 568

Solution 568

How It Works 572

xxix

CONTENTS

19-6. Tracing Storage Indexes when DML Is Issued against Tables 573

Problem 573

Solution 573

How It Works 574

19-7. Disabling Storage Indexes 575

Problem 575

Solution 575

How It Works 576

19-8. Troubleshooting Storage Indexes 576

Problem 576

Solution 576

How It Works 576

Part 6: Post Implementation Tasks 577

Chapter 20: Post-Installation Monitoring Tasks 579

20-1. Installing Enterprise Manager 12c Cloud Control Agents for Exadata 579

Problem 579

Solution 579

How It Works 591

20-2. Configuring Enterprise Manager 12c Cloud Control Plug-ins for Exadata 592

Problem 592

Solution 592

How It Works 592

20-3. Configuring Automated Service Requests 592

Problem 592

Solution 592

How It Works 597

Chapter 21: Post-Install Database Tasks 599

21-1. Creating a New Oracle RAC Database on Exadata 599

Problem 599

Solution 599

How It Works 601

XXX

CONTENTS

21-2. Setting Up a DBFS File System on Exadata 601

Problem 601

Solution 601

How It Works 605

21-3. Configuring HugePages on Exadata 606

Problem 606

Solution 606

How It Works 609

21-4. Configuring Automatic Degree of Parallelism 610

Problem 610

Solution 610

How It Works 611

21-5. Setting I/O Calibration on Exadata 612

Problem 612

Solution 612

How It Works 614

21-6. Measuring Impact of Auto DOP and Parallel Statement Queuing 614

Problem 614

Solution 614

How It Works 619

21-7. Measuring Auto DOP and In-Memory Parallel Execution 620

Problem 620

Solution 621

How It Works 625

21-8. Gathering Optimizer Statistics on Exadata 625

Problem 625

Solution 626

How It Works 630

Index 633

xxxi