Oracle Exadata Recipes
A Problem-Solution Approach
John Clarke
Apress-
Contents
About the Author xxxiii
About the Technical Reviewer xxxv
Acknowledgments xxxvii
Introduction xxxix
Part 1: Exadata Architecture 1
Chapter 1: Exadata Hardware 3
1-1. Identifying Exadata Database Machine Components 3
Problem 3
Solution 3
How It Works 7
1-2. Displaying Storage Server Architecture Details 10
Problem 10
Solution 10
How It Works 12
1-3. Displaying Compute Server Architecture Details 14
Problem 14
Solution 14
How It Works 16
1-4. Listing Disk Storage Details on the Exadata Storage Servers 17
Problem 17
Solution 17
How It Works 21
vii
CONTENTS
1-5. Listing Disk Storage Details on the Compute Servers 23
Problem 23
Solution 23
How It Works 27
1-6. Listing Flash Storage on the Exadata Storage Servers 27
Problem 27
Solution 27
How It Works 29
1-7. Gathering Configuration Information for the InfiniBandSwitches 29
Problem 29
Solution 29
How It Works 32
Chapter 2: Exadata Software 33
2-1. Understanding the Role of Exadata Storage Server Software 33
Problem 33
Solution 33
How It Works 35
2-2. Validating Oracle 11gR2 Databases on Exadata 35
Problem 35
Solution 35
How It Works 38
2-3. Validating Oracle 11gR2 Grid Infrastructure on Exadata 38
Problem 38
Solution 38
How It Works 41
2-4. Locating the Oracle Cluster Registry and Voting Disks on Exadata 42
Problem 42
Solution 42
How It Works 43
viii
CONTENTS
2-5. Validating Oracle 11gR2 Real Application Clusters Installation and Database
Storage on Exadata 44
Problem 44
Solution 44
How It Works 47
2-6. Validating Oracle 11gR2 Real Application Clusters Networking on Exadata 47
Problem 47
Solution 47
How It Works 52
a Chapter 3: How Oracle Works on Exadata 53
3-1. Mapping Physical Disks, LUNs, and Cell Disks on the Storage Servers 53
Problem 53
Solution 53
How It Works 56
3-2. Mapping ASM Disks, Grid Disks, and Cell Disks 58
Problem 58
Solution 58
How It Works 60
3-3. Mapping Flash Disks to Smart Flash Storage 61
Problem 61
Solution 61
How It Works 62
3-4. Identifying Cell Server Software Processes 62
Problem 62
Solution 63
How It Works 63
3-5. Tracing Oracle I/O Requests on Exadata Compute Nodes 65
Problem 65
Solution 65
How It Works 65
ix
CONTENTS
3-6. Validating That Your Oracle RAC Interconnect Is Using InfiniBand 68
Problem 68
Solution 68
How It Works 70
3-7. Tracing cellsrv on the Storage Servers 71
Problem 71
Solution 71
How It Works 73
Part 2: Preparing for Exadata 75
Chapter 4: Workload Qualification 77
4-1. Quantifying I/O Characteristics of Your Current Database 77
Problem 77
Solution 77
How It Works 80
4-2. Conducting a Smart Scan Fit Analysis Using AWR 81
Problem 81
Solution 81
How It Works 85
4-3. Conducting a Smart Scan Fit Analysis Using Exadata Simulation 87
Problem 87
Solution 87
How It Works 91
4-4. Performing a Hybrid Columnar Compression Fit Assessment 93
Problem 93
Solution 93
How It Works 95
Chapter 5: Sizing Exadata 97
5-1. Determining CPU Requirements 98
Problem 98
Solution 98
How It Works 107
x
CONTENTS
5-2. Determining lOPs Requirements 108
Problem 108
Solution 108
How It Works 112
5-3. Determining I/O Bandwidth Requirements 113
Problem 113
Solution 113
How It Works 118
5-4. Determining ASM Redundancy Requirements 119
Problem 119
Solution 119
How It Works 119
5-5. Forecasting Storage Capacity 122
Problem 122
Solution 122
How It Works 124
5-6. Planning for Database Growth 125
Problem 125
Solution 125
How It Works 133
5-7. Planning for Disaster Recovery 133
Problem 133
Solution 133
How It Works 133
5-8. Planning for Backups 134
Problem 134
Solution 134
How It Works 135
5-9. Determining Your Fast Recovery Area and RECO Disk Group Size Requirements 136
Problem 136
Solution 136
How It Works 138
xi
CONTENTS
Chapter 6: Preparing for Exadata 139
6-1. Planning and Understanding Exadata Networking 139
Problem 139
Solution 139
How It Works 141
6-2. Configuring DNS 143
Problem 143
Solution 143
How It Works 144
6-3. Running checkip.sh 144
Problem 144
Solution 144
How It Works 147
6-4. Customizing Your InfiniBand Network Configuration 147
Problem 147
Solution 147
How It Works 147
6-5. Determining Your DATA and RECO Storage Requirements 148
Problem 148
Solution 148
How It Works 149
6-6. Planning for ASM Disk Group Redundancy 149
Problem 149
Solution 149
How It Works 152
6-7. Planning Database and ASM Extent Sizes 152
Problem 152
Solution 152
How It Works 152
xii
CONTENTS
6-8. Completing the Pre-Delivery Survey 153
Problem 153
Solution 153
How It Works 154
6-9. Completing the Configuration Worksheet 154
Problem 154
Solution 154
How It Works 156
Part 3: Exadata Administration 157
Chapter 7: Administration and Diagnostics Utilities 159
7-1. Logging in to the Exadata Compute and Storage Cells Using SSH 159
Problem 159
Solution 159
How It Works 160
7-2. Configuring SSH Equivalency 161
Problem 161
Solution 161
How It Works 162
7-3. Locating Key Configuration Files and Directories on the Cell Servers 162
Problem 162
Solution 162
How It Works 163
7-4. Locating Key Configuration Files and Directories on the Compute Nodes 164
Problem 164
Solution 164
How It Works 164
7-5. Starting and Stopping Cell Server Processes 165
Problem 165
Solution 165
How It Works 166
xiii
CONTENTS
7-6. Administering Storage Cells Using CellCLI 167
Problem 167
Solution 167
How It Works 168
7-7. Administering Storage Cells Using deli 168
Problem 168
Solution 168
How It Works 170
7-8. Generating Diagnostics from the ILOM Interface 171
Problem 171
Solution 171
How It Works 173
7-9. Performing an Exadata Health Check Using exachk 173
Problem 173
Solution 173
How It Works 176
7-10. Collecting Compute and Cell Server Diagnostics Using the sundiag.sh Utility 176
Problem 176
Solution 176
How It Works 177
7-11. Collecting RAID Storage Information Using the MegaCLI utility 178
Problem 178
Solution 178
How It Works 179
7-12. Administering the Storage Cell Network Using ipconf 179
Problem 179
Solution 179
How It Works 180
7-13. Validating Your InfiniBand Switches with the CheckSWProfile.sh Utility 181
Problem 181
Solution 181
How It Works 182
xiv
CONTENTS
7-14. Verifying Your InfiniBand Network Topology 182
Problem 182
Solution 182
How It Works 183
7-15. Diagnosing Your InfiniBand Network 183
Problem 183
Solution 183
How It Works 187
7-16. Connecting to Your Cisco Catalyst 4948 Switch and Changing Switch Configuration 187
Problem 187
Solution 187
How It Works 188
Chapter 8: Backup and Recovery 189
8-1. Backing Up the Storage Servers 189
Problem 189
Solution 189
How It Works 190
8-2. Displaying the Contents of Your CELLBOOT USB Flash Drive 191
Problem 191
Solution 191
How It Works 192
8-3. Creating a Cell Boot Image on an External USB Drive 192
Problem 192
Solution 193
How It Works 194
8-4. Backing Up Your Compute Nodes Using Your Enterprise Backup Software 195
Problem 195
Solution 195
How It Works 196
XV
CONTENTS
8-5. Backing Up the Compute Servers Using LVM Snapshots 196
Problem 196
Solution 197
How It Works 200
8-6. Backing Up Your Oracle Databases with RMAN 201
Problem 201
Solution 201
How It Works 201
8-7. Backing Up the InfiniBand Switches 202
Problem 202
Solution 203
How It Works 204
8-8. Recovering Storage Cells from Loss of a Single Disk 205
Problem 205
Solution 205
How It Works 208
8-9. Recovering Storage Cells from Loss of a System Volume Using CELLBOOT Rescue 210
Problem 210
Solution 210
How It Works 221
8-10. Recovering from a Failed Storage Server Patch 222
Problem 222
Solution 222
How It Works 223
8-11. Recovering Compute Server Using LVM Snapshots 223
Problem 223
Solution 223
How It Works 228
8-12. Reimaging a Compute Node 229
Problem 229
Solution 229
How It Works 231
xvi
CONTENTS
8-13. Recovering Your InfiniBand Switch Configuration 232
Problem 232
Solution 232
How It Works 234
8-14. Recovering from Loss of Your Oracle Cluster Registry and Voting Disks 235
Problem 235
Solution 235
How It Works 237
Chapter 9: Storage Administration 239
9-1. Building ASM Disk Groups on Exadata 239
Problem 239
Solution 239
How It Works 245
9-2. Properly Configuring ASM Disk Group Attributes on Exadata 246
Problem 246
Solution 246
How It Works 247
9-3. Identifying Unassigned Grid Disks 248
Problem 248
Solution 248
How It Works 248
9-4. Configuring ASM Redundancy on Exadata 248
Problem 248
Solution 248
How It Works 251
9-5. Displaying ASM Partner Disk Relationships on Exadata 253
Problem 253
Solution 253
How It Works 254
xvii
CONTENTS
9-6. Measuring ASM Extent Balance on Exadata 254
Problem 254
Solution 254
How It Works 258
9-7. Rebuilding Cell Disks 258
Problem 258
Solution 258
How It Works 273
9-8. Creating Interleaved Cell Disks and Grid Disks 274
Problem 274
Solution 274
How It Works 280
9-9. Rebuilding Grid Disks 282
Problem 282
Solution 282
How It Works 284
9-10. Setting smart_scan_capable on ASM Disk Groups 286
Problem 286
Solution 286
How It Works 291
9-11. Creating Flash Grid Disks for Permanent Storage 292
Problem 292
Solution 292
How It Works 296
Chapter 10: Network Administration 297
10-1. Configuring the Management Network on the Compute Nodes 297
Problem 297
Solution 297
How It Works 299
xviii
CONTENTS
10-2. Configuring the Client Access Network 299
Problem 299
Solution 299
How It Works 301
10-3. Configuring the Private Interconnect on the Compute Nodes 301
Problem 301
Solution 302
How It Works 303
10-4. Configuring the SCAN Listener 304
Problem 304
Solution 304
How It Works 305
10-5. Managing Grid Infrastructure Network Resources 306
Problem 306
Solution 306
How It Works 309
10-6. Configuring the Storage Server Ethernet Network 309
Problem 309
Solution 309
How It Works 311
10-7. Changing IP Addresses on Your Exadata Database Machine 312
Problem 312
Solution 312
How It Works 330
Chapter 11: Patching and Upgrades 331
11-1. Understanding Exadata Patching Definitions, Alternatives, and Strategies 331
Problem 331
Solution 331
How It Works 333
xix
CONTENTS
11 -2. Preparing to Apply Exadata Patches 333
Problem 333
Solution 333
How It Works 335
11-3. Patching Your Exadata Storage Servers 335
Problem 335
Solution 335
How It Works 341
11-4. Patching Your Exadata Compute Nodes and Databases 342
Problem 342
Solution 342
How It Works 349
11-5. Patching the InfiniBand Switches 349
Problem 349
Solution 349
How It Works 352
11-6. Patching Your Enterprise Manager Systems Management Software 352
Problem 352
Solution 352
How It Works 353
HChapter 12: Security 355
12- 1. Configuring Multiple Oracle Software Owners on Exadata Compute Nodes 355
Problem 355
Solution 355
How It Works 358
12-2. Installing Multiple Oracle Homes on Your Exadata Compute Nodes 359
Problem 359
Solution 359
How It Works 361
XX
CONTENTS
12-3. Configuring ASM-Scoped Security 362
Problem 362
Solution 362
How It Works 364
12-4. Configuring Database-Scoped Security 364
Problem 364
Solution 364
How It Works 367
HPart 4: Monitoring Exadata 369
HChapter 13: Monitoring Exadata Storage Cells 371
13-1. Monitoring Storage Cell Alerts 371
Problem 371
Solution 371
How It Works 375
13-2. Monitoring Cells with Active Requests 376
Problem 376
Solution 376
How It Works 380
13-3. Monitoring Cells with Metrics 381
Problem 381
Solution 381
How It Works 386
13-4. Configuring Thresholds for Cell Metrics 389
Problem 389
Solution 389
How It Works 390
13-5. Using deli with Special Characters 391
Problem 391
Solution 391
How It Works 393
xxi
CONTENTS
13-6. Reporting and Summarizing metrichistory Using R 393
Problem 393
Solution 393
How It Works 398
13-7. Reporting and Summarizing metrichistory Using Oracle and SQL 399
Problem 399
Solution 399
How It Works 400
13-8. Detecting Cell Disk I/O Bottlenecks 400
Problem 400
Solution 400
How It Works 402
13-9. Measuring Small I/O vs. Large I/O Requests 402
Problem 402
Solution 402
How It Works 404
13-10. Detecting Grid Disk I/O Bottlenecks 404
Problem 404
Solution 404
How It Works 405
13-11 Detecting Host Interconnect Bottlenecks 406
Problem 406
Solution 406
How It Works 407
13-12. Measuring I/O Load and Waits per Database, Resource Consumer Group,
and Resource Category 407
Problem 407
Solution 407
How It Works 409
xxii
CONTENTS
Chapter 14: Host and Database Performance Monitoring 411
14-1. Collecting Historical Compute Node and Storage Cell Host Performance Statistics 411
Problem 411
Solution 411
How It Works 413
14-2. Displaying Real-Time Compute Node and Storage Cell Performance Statistics 414
Problem 414
Solution 414
How It Works 417
14-3. Monitoring Exadata with Enterprise Manager 417
Problem 417
Solution 417
How It Works 421
14-4. Monitoring Performance with SQL Monitoring 421
Problem 421
Solution 421
How It Works 424
14-5. Monitoring Performance by Database Time 425
Problem 425
Solution 425
How It Works 431
14-6. Monitoring Smart Scans by Database Time and AAS 432
Problem 432
Solution 432
How It Works 434
14-7. Monitoring Exadata with Wait Events 435
Problem 435
Solution 435
How It Works 439
xxiii
CONTENTS
14-8. Monitoring Exadata with Statistics and Counters 440
Problem 440
Solution 440
How It Works 441
14-9. Measuring Cell I/O Statistics for a SQL Statement 442
Problem 442
Solution 442
How It Works 444
Part 5: Exadata Software 445
Chapter 15: Smart Scan and Cell Offload 447
15-1. Identifying Cell Offload in Execution Plans 447
Problem 447
Solution 447
How It Works 448
15-2. Controlling Cell Offload Behavior 450
Problem 450
Solution 450
How It Works 451
15-3. Measuring Smart Scan with Statistics 451
Problem 451
Solution 451
How It Works 455
15-4. Measuring Offload Statistics for Individual SQL Cursors 455
Problem 455
Solution 455
How It Works 457
15-5. Measuring Offload Efficiency 457
Problem 457
Solution 457
How It Works 460
xxiv
CONTENTS
15-6. Identifying Smart Scan from 10046 Trace Files 461
Problem 461
Solution 461
How It Works 462
15-7. Qualifying for Direct Path Reads 463
Problem 463
Solution 463
How It Works 468
15-8. Influencing Exadata's Decision to Use Smart Scans 469
Problem 469
Solution 469
How It Works 473
15-9. Identifying Partial Cell Offload 473
Problem 473
Solution 473
How It Works 474
15-10. Dealing with Fast Object Checkpoints 475
Problem 475
Solution 475
How It Works 475
Chapter 16: Hybrid Columnar Compression 477
16-1. Estimating Disk Space Savings for HCC 477
Problem 477
Solution 477
How It Works 479
16-2. Building HCC Tables and Partitions 479
Problem 479
Solution 479
How It Works 482
XXV
CONTENTS
16-3. Contrasting Oracle Compression Types 482
Problem 482
Solution 482
How It Works 483
16-4. Determining the Compression Type of a Segment 487
Problem 487
Solution 487
How It Works 488
16-5. Measuring the Performance Impact of HCC for Queries 488
Problem 488
Solution 489
How It Works 491
16-6. Direct Path Inserts into HCC Segments 492
Problem 492
Solution 492
How It Works 493
16-7. Conventional Inserts to HCC Segments 493
Problem 493
Solution 494
How It Works 495
16-8. OMLand HCC 495
Problem 495
Solution 495
How It Works 501
16-9. Decompression and the Performance Impact 502
Problem 502
Solution 502
How It Works 504
xxvi
CONTENTS
Chapter 17: I/O Resource Management and Instance Caging 505
17-1. Prioritizing I/O Utilization by Database 505
Problem 505
Solution 505
How It Works 507
17-2. Limiting I/O Utilization for Your Databases 507
Problem 507
Solution 507
How It Works 508
17-3. Managing Resources within a Database 508
Problem 508
Solution 508
How It Works 510
17-4. Prioritizing I/O Utilization by Category of Resource Consumers 511
Problem 511
Solution 511
How It Works 512
17-5. Prioritizing I/O Utilization by Categories of Resource Consumers and Databases 513
Problem 513
Solution 513
How It Works 514
17-6. Monitoring Performance When IORM Is Enabled 517
Problem 517
Solution 517
How It Works 520
17-7. Obtaining IORM Plan Information 521
Problem 521
Solution 521
How It Works 525
xxvii
CONTENTS
17-8. Controlling Smart Flash Cache and Smart Flash Logging with IORM 526
Problem 526
Solution 526
How It Works 526
17-9. Limiting CPU Resources with Instance Caging 527
Problem 527
Solution 527
How It Works 527
(Chapter 18: Smart Flash Cache and Smart Flash Logging 529
18-1. Managing Smart Flash Cache and Smart Flash Logging 529
Problem 529
Solution 530
How It Works 531
18-2. Determining Which Database Objects Are Cached 532
Problem 532
Solution 532
How It Works 533
18-3. Determining What's Consuming Your Flash Cache Storage 534
Problem 534
Solution 534
How It Works 536
18-4. Determining What Happens When Querying Uncached Data 536
Problem 536
Solution 536
How It Works 540
18-5. Measuring Smart Flash Cache Performance 541
Problem 541
Solution 541
How It Works 546
xxviii
CONTENTS
18-6. Pinning Specific Objects in Smart Flash Cache 546
Problem 546
Solution 546
How It Works 548
18-7. Quantifying Benefits of Smart Flash Logging 549
Problem 549
Solution 549
How It Works 550
S Chapter 19: Storage Indexes 553
19-1. Measuring Performance Impact of Storage Indexes 553
Problem,
553
Solution 553
How It Works 555
19-2. Measuring Storage Index Performance with Not-So-Well-Ordered Data 557
Problem 557
Solution 557
How It Works 559
19-3. Testing Storage Index Behavior with Different Query Predicate Conditions 559
Problem 559
Solution 560
How It Works 564
19-4. Tracing Storage Index Behavior 564
Problem 564
Solution 565
How It Works 568
19-5. Tracing Storage Indexes When More than Eight Columns Are Referenced 568
Problem 568
Solution 568
How It Works 572
xxix
CONTENTS
19-6. Tracing Storage Indexes when DML Is Issued against Tables 573
Problem 573
Solution 573
How It Works 574
19-7. Disabling Storage Indexes 575
Problem 575
Solution 575
How It Works 576
19-8. Troubleshooting Storage Indexes 576
Problem 576
Solution 576
How It Works 576
Part 6: Post Implementation Tasks 577
Chapter 20: Post-Installation Monitoring Tasks 579
20-1. Installing Enterprise Manager 12c Cloud Control Agents for Exadata 579
Problem 579
Solution 579
How It Works 591
20-2. Configuring Enterprise Manager 12c Cloud Control Plug-ins for Exadata 592
Problem 592
Solution 592
How It Works 592
20-3. Configuring Automated Service Requests 592
Problem 592
Solution 592
How It Works 597
Chapter 21: Post-Install Database Tasks 599
21-1. Creating a New Oracle RAC Database on Exadata 599
Problem 599
Solution 599
How It Works 601
XXX
CONTENTS
21-2. Setting Up a DBFS File System on Exadata 601
Problem 601
Solution 601
How It Works 605
21-3. Configuring HugePages on Exadata 606
Problem 606
Solution 606
How It Works 609
21-4. Configuring Automatic Degree of Parallelism 610
Problem 610
Solution 610
How It Works 611
21-5. Setting I/O Calibration on Exadata 612
Problem 612
Solution 612
How It Works 614
21-6. Measuring Impact of Auto DOP and Parallel Statement Queuing 614
Problem 614
Solution 614
How It Works 619
21-7. Measuring Auto DOP and In-Memory Parallel Execution 620
Problem 620
Solution 621
How It Works 625
21-8. Gathering Optimizer Statistics on Exadata 625
Problem 625
Solution 626
How It Works 630
Index 633
xxxi