basic mysql troubleshooting for oracle dbas

99
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1

Upload: sveta-smirnova

Post on 27-Jan-2015

151 views

Category:

Technology


4 download

DESCRIPTION

Presentation which I showed at MySQL Connect 2013

TRANSCRIPT

Page 1: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1

Page 2: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.2

Insert Picture Here

Basic MySQL Troubleshooting for Oracle DBAsSveta SmirnovaPrincipal Technical Support Engineer

Page 3: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3

Program Agenda

Introduction Basic single-client issues Concurrency issues High availability solutions Tools

Page 4: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4

Insert Picture Here

Introduction

Page 5: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5

Insert Picture Here

Base● Installation layout● Log files

Connectors● Clients● APIs

OptimizerCaches & buffersStorage enginesManagement

Overview

MySQL architecture

Page 6: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6

Typical installation layout

Datadir● Schema

● Table files: *frm, *ibd, *MYD, *MYI, *par, etc.

● Log files

● InnoDB shared tablespace

Configurable

Page 7: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7

Log files

General query log

Slow query log

Binary log

Relay log

Error log

InnoDB logs (and, probably, created by other storage engines)

Page 8: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8

Connectors

Clients● MySQL CLI● MySQL Workbench● MySQL Enterprise Monitor (MEM)

● For administrators

APIs

● Exist for most popular programming languages

● C, C++, JDBC, PHP, Python, Net, ODBC

Page 9: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9

Plugins

Plugins● Storage engine● Full-text parsers● Daemon● INFORMATION_SCHEMA● Semisynchronous Replication● Audit● Authentication● Password-validation

Page 10: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10

Storage engines

From troubleshooting point of view

● Own data/index format

● Own locking model

● Own diagnostic

● Own log files

● CHECK TABLE

Page 11: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11

Insert Picture Here

Basic single-client issues

Page 12: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12

MySQL Access Privilege System

No roles by default, limited user limits

All records are in the mysql database (schema)

Pluggable authentication since version 5.5

Connections: TCP/IP with login-password, socket (Unix), named pipe (Windows)

SELECT user, host FROM mysql.user

SELECT USER(), CURRENT_USER()

SHOW GRANTS

Page 13: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13

Error handlingErrors vs warningsmysql> select max (f1) from t1;ERROR 1630 (42000): FUNCTION test.max does not exist. Check the 'Function Name Parsing and Resolution' section in the Reference Manual

mysql> select * from t1 where "f1"=1;Empty set, 1 warning (0.05 sec)

mysql> show warnings;+­­­­­­­­­­­+­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| Level     | Code   | Message                                            |+­­­­­­­­­­­+­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| Warning   | 1292   | Truncated incorrect DOUBLE value: 'f1'             |+­­­­­­­­­­­+­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+1 row in set (0.00 sec)

Page 14: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14

Error handlingApplication (C API)

Error information● mysql_errno● mysql_error

Warnings● mysql_info● mysql_sqlstate● mysql_warning_count

Page 15: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15

Error handlingperror[sveta@delly ~]$ perror 1630MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION): FUNCTION %s does not exist. Check the 'Function Name Parsing and Resolution' section in the Reference Manual[sveta@delly ~]$ perror 1292MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated incorrect %­.32s value: '%­.128s'[sveta@delly ~]$ perror 2OS error code   2:  No such file or directory[sveta@delly ~]$ perror 150MySQL error code 150: Foreign key constraint is incorrectly formed

Page 16: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16

Error handlingin stored routines

GET DIAGNOSTICS● GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;● GET DIAGNOSTICS CONDITION 1  code = RETURNED_SQLSTATE, 

msg = MESSAGE_TEXT;http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html

Page 17: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17

Options

Have global and session scopeCan be set

● In configuration files● Command line parameter● SET [GLOBAL] var_name = NEW_VAL● SHOW [GLOBAL] VARIABLES

Only two built-in levels of write access● SUPER for GLOBAL and limited SESSION variables● Non-privileged user can change any session option!

Page 18: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18

Options

Allocated at● Server startup● User connection creation● For certain operations

To watch status● SHOW [GLOBAL] STATUS

http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html

Page 19: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19

INFORMATION_SCHEMA

Contain metadata information● Tables● Indexes● Other

Allows to create plugins● InnoDB plugins

● We will discuss them later

Oracle analog: Data Dictionary Views

Page 20: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20

Optimizer

EXPLAIN is less powerful if compare with Oracle● It is improved in version 5.7.3● Graphic EXPLAIN in MySQL Workbench 6.0● MySQL’s EXPLAIN Command New Features [HOL9734, passed]EXPLAIN EXTENDEDINFORMATION_SCHEMA.OPTIMIZER_TRACEStatus variables 'Handler_%'

Page 21: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21

EXPLAIN in Oracle

EXPLAIN PLAN SET statement_id = 'example_plan4' FORSELECT h.order_number, l.revenue_amount, l.ordered_quantity  FROM so_headers_all h, so_lines_all l WHERE h.customer_id = :b1   AND h.date_ordered > SYSDATE­30   AND l.header_id = h.header_id ;

Plan­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­SELECT STATEMENT NESTED LOOPS  TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL   INDEX RANGE SCAN SO_HEADERS_N1  TABLE ACCESS BY INDEX ROWID SO_LINES_ALL   INDEX RANGE SCAN SO_LINES_N1

http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm

Page 22: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22

EXPLAIN in MySQL

mysql> EXPLAIN SELECT user, host FROM user\G*************************** 1. row ***************************           id: 1  select_type: SIMPLE        table: user         type: indexpossible_keys: NULL          key: PRIMARY      key_len: 228          ref: NULL         rows: 4        Extra: Using index1 row in set (0.00 sec)

Page 23: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23

EXPLAIN in MySQL

EXPLAIN EXTENDED● Follow by SHOW WARNINGSEXPLAIN PARTITIONSEXPLAIN FORMAT=JSONhttp://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsenhttp://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D79908_1879034

http://dev.mysql.com/doc/refman/5.6/en/explain-output.html

Page 24: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24

Storage engine specifics

They care about physical data, so all data information are on their level● Corruption● Index statisticsCHECK TABLE to check for errors

Page 25: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25

Storage engine specificsMyISAM

Stores data in files● *frm – table definition, common for all storage engines● *MYD – data file● *MYI – index filemyisamchk­­myisam_recovery_options to check each time you open a table

● First access – table is opened● Second, third, .. access – existent descriptor used● FLUSH TABLES – closes opened descriptors

Page 26: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26

Storage engine specificsInnoDB

Transactional storage enginePhysical layout

● *frm file – table definition● Shared tablespace● *ibd file – tablespace for individual table

● Optional: ­­innodb_file_per_table

● Recommended

● Redo log filesOPTIMIZE TABLE = ALTER + ANALYZEAutomatic startup check

Page 27: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27

Insert Picture Here

Concurrency issues

Page 28: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28

Lock types

MDL locksTable locksRow locksRead locks

● Block writesWrite locks

● Block both read and writes

Page 29: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29

Transactions and Their Relationship With Locks

Server-level● MDL locks

Engine level● Table locks● Row locks

AUTOCOMMIT● Supported

Page 30: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30

Diagnostic Tools

SHOW [FULL] PROCESSLIST● Universal tool which lists all currently running connectionsSHOW ENGINE INNODB STATUSINFORMATION SCHEMA

● PROCESSLIST● InnoDB tablesPERFORMANCE SCHEMA

● MDL locksUnlocking MySQL: Dealing with Locking in MySQL [CON4038, passed]

Page 31: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31

SHOW PROCESSLIST

mysql> show processlist;+­­­­+­­­­­­­+­­­­­­­­­­­+­­­­­­+­­­­­­­­­+­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+| Id | User  | Host      | db   | Command | Time | State                        | Info           |+­­­­+­­­­­­­+­­­­­­­­­­­+­­­­­­+­­­­­­­­­+­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+| 23 | sveta | localhost | test | Query   |   3  | User sleep                   | select *, sleep(20) from tlog        || 24 | sveta | localhost | test | Query   |   1  | Waiting for table level lock | update tlog set ln=5 where ln=4  || 28 | root  | localhost | NULL | Query   |   0  | NULL                         | show processlist                         |+­­­­+­­­­­­­+­­­­­­­­­­­+­­­­­­+­­­­­­­­­+­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+3 rows in set (0.00 sec)

Page 32: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32

INFORMATION_SCHEMA.PROCESSLISTstructure

mysql> DESCRIBE INFORMATION_SCHEMA.PROCESSLIST;+­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­+­­­­­­­+­­­­­­+­­­­­­­­­­+­­­­­­­­+| Field          | Type            | Null   | Key | Default  | Extra  |+­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­+­­­­­­­+­­­­­­+­­­­­­­­­­+­­­­­­­­+| ID             | bigint(4)       | NO    |      | 0        |        || USER           | varchar(16)     | NO    |      |          |        || HOST           | varchar(64)     | NO    |      |          |        || DB             | varchar(64)     | YES   |      | NULL     |        || COMMAND        | varchar(16)     | NO    |      |          |        || TIME           | int(7)          | NO    |      | 0        |        || STATE          | varchar(64)     | YES   |      | NULL     |        || INFO           | longtext        | YES   |      | NULL     |        |+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+­­­­­­­­+­­­­­­+­­­­­­­­­­+­­­­­­­+

Page 33: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33

SHOW PROCESSLIST and Transactions

mysql> show processlist;+­­­­+­­­­­­­+­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| Id | User  | Host      | db      | Command | Time     | State    | Info                                |+­­­­+­­­­­­­+­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| 28 | root  | localhost | NULL    | Query   |    0     | NULL     | show processlist                    || 31 | sveta | localhost | test    | Query   |    7     | Updating | update tlog set eid=2345 where ln=1 || 32 | sveta | localhost | test    | Sleep   |   18     |          | NULL                                |+­­­­+­­­­­­­+­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+3 rows in set (0.00 sec)

Not much info!

No lock information!

Page 34: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34

Storage engine specifics

Transactions and engine-level locks are done at the engine levelStorage engines have own diagnostic toolsUse them when hit an issue with such a lock

Page 35: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35

Storage engine specificsInnoDB

InnoDB Monitors● SHOW ENGINE INNODB STATUS● Lock MonitorINFORMATION_SCHEMA tables

● INNODB_TRX● INNODB_LOCKS● INNODB_LOCK_WAITS

Page 36: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36

Diagnostics: P_S

Monitors internal operations● Events● Waits● Mutexes● Statements● Stages

Similar to Oracle wait interfaceImproving Performance with MySQL Performance Schema [HOL9733, Sunday, 1:00 PM]Making the Performance Schema Easier to Use [CON5282, Sunday, 10:00 AM] Performance Schema and ps_helper [CON4077, passed]

Page 37: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37

MDL Locks

Waiting thread onlySHOW PROCESSLIST, INFORMATION_SCHEMA.PROCESSLIST

● Waiting for table metadata lockPerformance Schema

● MUTEX_INSTANCES● EVENTS_WAITS_CURRENT● THREADS

Page 38: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38

Performance Schema MDL queries

mysql> SELECT * FROM mutex_instances WHERE LOCKED_BY_THREAD_ID IS NOT NULL\G*********************** 1. row ***********************                 NAME: wait/synch/mutex/sql/MDL_wait::LOCK_wait_statusOBJECT_INSTANCE_BEGIN: 37104744  LOCKED_BY_THREAD_ID: 181 row in set (0.01 sec)

Page 39: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39

Performance Schema MDL queries

mysql> SELECT THREAD_ID, EVENT_ID, EVENT_NAME, SOURCE, TIMER_START, OBJECT_INSTANCE_BEGIN, OPERATION FROM events_waits_current WHERE THREAD_ID IN(SELECT LOCKED_BY_THREAD_ID FROM mutex_instances WHERE LOCKED_BY_THREAD_ID IS NOT NULL)\G*************************** 1. row ***************************            THREAD_ID: 18             EVENT_ID: 461           EVENT_NAME: wait/synch/cond/sql/MDL_context::COND_wait_status               SOURCE: mdl.cc:1210          TIMER_START: 97623253523306OBJECT_INSTANCE_BEGIN: 0            OPERATION: timed_wait1 row in set (0.00 sec)

Page 40: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40

Insert Picture Here

High availability solutions

Page 41: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41

MySQL Cluster

Special storage engine: NDBStores data on two or more physical machinesTwo or more copiesGeneral troubleshooting techniques

● ApplicableSpecific NDB storage engine techniques

Page 42: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42

ReplicationOverview

Always available, but you must setup it before useAsynchronous master-slaveMaster

● Keeps all updates in separate file: binary logSlave

● IO thread read updates from master and stores in relay log file● SQL thread executes updates

Page 43: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.43

ReplicationTroubleshooting tools

Error log fileSlave

● SHOW SLAVE STATUSMaster

● SHOW MASTER STATUSmysqlbinlog

Page 44: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.44

Insert Picture Here

Tools

Page 45: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45

For developers

MySQL Workbench● SQL editor● Database designer● Introduction to MySQL Database Development with MySQL Workbench

[CON3967, Sunday, 5:30 PM]MySQL Command line client

● Command-line interface for SQL● SELECT, INSERT, UPDATE, DELETE, etc.

Page 46: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46

For DBA

MySQL Enterprise Monitor● Real-time MySQL performance and availability monitoring● Error information● Advisors● Forcasts

Page 47: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47

For DBA

MySQL Workbench● Server administration● Backups● Using the New MySQL Workbench Tools for Performance Tuning

[HOL9786, passed] MySQL command line client

● Command-line interface for SQL● GRANT, CREATE, OPTIMIZE, etc.mysqladmin

Page 48: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.48

Command line

  They are notSame!

Set of tools to do various administrative job● MySQL Workbench Utilities

● http://dev.mysql.com/downloads/tools/utilities/

● Development of Fault-Tolerant Failover Tools with MySQL Utilities [CON4276, Sunday, 2:30 PM]

● Percona Toolkit● http://www.percona.com/software/percona-toolkit

Page 49: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.49

Command line

MySQL Sandbox● Sandbox for your tests● https://launchpad.net/mysql-sandboxPS_HELPER view

● www.markleith.co.uk/ps_helper/● Performance Schema and ps_helper [CON4077, passed]

Page 50: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.50

References

MySQL User Reference Manual● http://dev.mysql.com/doc/refman/5.6/en/index.html

Knowledge Management DatabaseForums

● http://forums.mysql.comBug trackers

● http://bugs.mysql.com● Oracle Internal Bugs database

My Oracle Support● https://support.oracle.com

Page 51: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.51

References

MySQL Troubleshooting book● http://shop.oreilly.com/product/0636920021964.do

Marc Alff's Performance Schema blog● http://marcalff.blogspot.ru/

Planet MySQL● http://planet.mysql.com/● http://dev.mysql.com/support/blogs/● https://blogs.oracle.com/mysqlinnodb/

Page 52: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.52

Insert Picture Here

?

Page 53: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.53

Insert Picture Here

THANK YOU!

https://twitter.com/#!/svetsmirnovahttps://blogs.oracle.com/svetasmirnova/

Page 54: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.54

Graphic Section Divider

Page 55: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.55

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract.It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Page 56: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.56

Insert Picture Here

MySQL Troubleshooting for Oracle DBAsPart II: detailsSveta SmirnovaPrincipal Technical Support Engineer

Page 57: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.57

Insert Picture Here

Replication

Page 58: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.58

Replication IO thread:communication issues

Access error● Check slave error log● SHOW SLAVE STATUS● Try to connect using normal MySQL client using slave credentials

● SHOW GRANTS

● Fix privileges on master● Restart slave

Page 59: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.59

Replication SQL thread:typical issues

Simple master-slave● Data is different on master and slave

● Replication event can not be applied

● Different errors on master and slave● Slave lags far behind the master

Circular replication or other writes in addition to slave SQL thread● Data is different on master and slave

Page 60: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.60

Replication SQL thread:Data is different on master and slave

Was the table modified besides the SQL thread?● How?● Can it affect content of the table in the wrong way?

Are the table definitions same on master and slave?Is it possible that master events was applied in wrong order?

● Use mysqlbinlog to find queries which caused the issue● Check master's application to find what is wrong

Page 61: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.61

Replication SQL thread: Events from master were applied in wrong order

Lock issues● InnoDB

Triggers● SET GLOBAL slave_skip_counter● Synchronize tables!

Different options● Start slave with master's options, then check

Page 62: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.62

Replication SQL thread:Slave lags far behind the master

Threads● Master runs in multiple update threads● Slave uses single

Seconds_behind_master is growingTune slave performance

● Buffers● Indexes (for statement)

Page 63: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.63

Insert Picture Here

Optimizer

Page 64: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.64

OptimizerHandler_% status variablesmysql> flush status;Query OK, 0 rows affected (0.00 sec)mysql> SHOW STATUS LIKE 'Handler_%';+­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­+| Variable_name              | Value |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­+| Handler_commit             | 0     || Handler_delete             | 0     || Handler_discover           | 0     || Handler_prepare            | 0     || Handler_read_first         | 0     || Handler_read_key           | 0     || Handler_read_last          | 0     || Handler_read_next          | 0     || Handler_read_prev          | 0     || Handler_read_rnd           | 0     |

Page 65: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.65

OptimizerHandler_% status variables| Handler_read_rnd_next      | 0     || Handler_rollback           | 0     || Handler_savepoint          | 0     || Handler_savepoint_rollback | 0     || Handler_update             | 0     || Handler_write              | 0     |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­+16 rows in set (0.00 sec)

mysql> select count(*) from employees join titles using(emp_no) where title='Senior Engineer';+­­­­­­­­­­+| count(*) |+­­­­­­­­­­+|    97750 |+­­­­­­­­­­+1 row in set (3.24 sec)

Page 66: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.66

OptimizerHandler_% status variablesmysql> SHOW STATUS LIKE 'Handler_%';+­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­+| Variable_name              | Value  |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­+| Handler_commit             | 1      || Handler_delete             | 0      || Handler_discover           | 0      || Handler_prepare            | 0      || Handler_read_first         | 1      || Handler_read_key           | 300027 || Handler_read_last          | 0      || Handler_read_next          | 397774 || Handler_read_prev          | 0      || Handler_read_rnd           | 0      || Handler_read_rnd_next      | 0      |...

Page 67: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.67

OptimizerINFORMATION_SCHEMA.OPTIMIZER_TRACE

Variables:● optimizer_trace=”enabled=on|off,one_line=off|on”● optimizer_trace_features=”greedy_search=on,range_optimi

zer=on,dynamic_range=on,repeated_subselect=on”● optimizer_trace_limit=1● optimizer_trace_max_mem_size=16384● optimizer_trace_offset=­1● end_markers_in_json=0|1

Page 68: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.68

OptimizerINFORMATION_SCHEMA.OPTIMIZER_TRACE

Turn tracing on (it's off by default):SET optimizer_trace="enabled=on";SELECT <your query here>; SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;More queries...When done with tracing, disable it:SET optimizer_trace="enabled=off";http://dev.mysql.com/doc/internals/en/optimizer-tracing.html

Page 69: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.69

OptimizerINFORMATION_SCHEMA.OPTIMIZER_TRACEmysql> SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE\G*************************** 1. row ***************************                        QUERY: select emp_no, min(from_date) from titles group by emp_no limit 10                        TRACE: {  "steps": [    {      "join_preparation": {        "select#": 1,        "steps": [          {            "expanded_query": "/* select#1 */ select `titles`.`emp_no` AS `emp_no`,min(`titles`.`from_date`) AS `min(from_date)` from `titles` group by `titles`.`emp_no` limit 10"

Page 70: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.70

OptimizerINFORMATION_SCHEMA.TRACE

join_preparationjoin_optimization

● table_dependencies● rows_estimation● considered_execution_plans● attaching_conditions_to_tables● clause_processing● refine_plan● reconsidering_access_paths_for_index_ordering

join_execution

Page 71: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.71

Insert Picture Here

Options

Page 72: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.72

OptionsScope

Global● Control parameters, necessary for all server processes

● Location of server files: datadir etc.

● Shared buffers

● More

Session● Control connection-specific parameters

http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html

Page 73: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.73

OptionsHow to set

SET [GLOBAL] var_name = NEW_VALCommand line optionConfiguration file

● In default location● Specified by option ­­defaults­file

Page 74: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.74

OptionsWho can change

Global options and few session options● A user with privilege SUPER

Session options● Anybody

There is no limits!

Page 75: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.75

OptionsWhen allocated

Those which control behavior of whole server● Once at server startup● Can start with low values, then grow to specified

Connection options● For every connection when connection opens

Operation-specific● For every operation when needed

Page 76: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.76

OptionsHow to control

SHOW [GLOBAL] STATUSGLOBAL

● Since server startSESSION

● For operations in current session● Can be reset

● FLUSH STATUS

Page 77: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.77

OptionsGlobal example

mysql> show global status like 'Handler_read_rnd_next';+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­+| Variable_name                  | Value      |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­+| Handler_read_rnd_next          | 3821812    |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­+1 row in set (0.00 sec)

Page 78: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.78

OptionsSession example

mysql> show status like 'Handler_read_rnd_next';+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­+| Variable_name                  | Value  |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­+| Handler_read_rnd_next          | 8      |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­+1 row in set (0.00 sec)

Page 79: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.79

OptionsTroubleshooting best practices

Record currently used variables● SHOW [GLOBAL] VARIABLES

Make change dynamically if possible● SET [GLOBAL] var_name=NEW_VAL

Test in one session firstThen change global variableChange configuration file after you are happy with results

Page 80: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.80

OptionsWhen affecting option is not known

Record currently used variables● SHOW [GLOBAL] VARIABLES

Start mysqld with option –no­defaults● This option must be first one!

Check if problem is solvedChange variable values one-by-one until you find one which leads to the problem

Page 81: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.81

Insert Picture Here

InnoDB Monitors and Information Schema tables

Page 82: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.82

SHOW ENGINE INNODB STATUS

­­­­­­­­­­­­TRANSACTIONS­­­­­­­­­­­­Trx id counter 1B1CPurge done for trx's n:o < 1B19 undo n:o < 0History list length 189LIST OF TRANSACTIONS FOR EACH SESSION:­­­TRANSACTION 0, not startedMySQL thread id 28, OS thread handle 0x7fa9301a0700, query id 8653 localhost rootshow engine innodb status

Page 83: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.83

SHOW ENGINE INNODB STATUS

­­­TRANSACTION 1B1B, ACTIVE 6 sec starting index readmysql tables in use 1, locked 1LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)MySQL thread id 31, OS thread handle 0x7fa9301e1700, query id 8652 localhost sveta Updatingupdate tbllog set `Employee ID`=2345 where `Log Number`=1­­­­­­­ TRX HAS BEEN WAITING 6 SEC FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 81967 n bits 72 index `PRIMARY` of table `test`.`tlog` trx id 1B1B lock_mode X locks rec but not gap waitingRecord lock, heap no 6 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000001; asc     ;; 1: len 6; hex 000000001b19; asc       ;; 2: len 7; hex 14000140340110; asc    @4  ;; 3: len 4; hex 80003039; asc   09;; 4: SQL NULL;

Page 84: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.84

SHOW ENGINE INNODB STATUS

No lock informationFor the transactionWhich holds the lock!

­­­TRANSACTION 1B19, ACTIVE 787 sec2 lock struct(s), heap size 376, 1 row lock(s), undo log entries 1MySQL thread id 32, OS thread handle 0x7fa930222700, query id 8647 localhost sveta

Page 85: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.85

InnoDB Monitors

Pseudo-tablesTurn periodical logging into error log

● CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;Lock monitor

● Changes output format of InnoDB Monitor● CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;

Page 86: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.86

InnoDB Lock Monitor

­­­TRANSACTION 1B19, ACTIVE 1935 sec2 lock struct(s), heap size 376, 1 row lock(s), undo log entries 1MySQL thread id 32, OS thread handle 0x7fa930222700, query id 8647 localhost svetaTABLE LOCK table `test`.`tlog` trx id 1B19 lock mode IXRECORD LOCKS space id 0 page no 81967 n bits 72 index `PRIMARY` of table `test`.`tlog` trx id 1B19 lock_mode X locks rec but not gapRecord lock, heap no 6 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000001; asc     ;; 1: len 6; hex 000000001b19; asc       ;; 2: len 7; hex 14000140340110; asc    @4  ;; 3: len 4; hex 80003039; asc   09;; 4: SQL NULL;

Page 87: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.87

InnoDB Information Schema tables

INNODB_TRXINNODB_LOCKSINNODB_LOCK_WAITS

Page 88: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.88

InnoDB Information Schema tables

mysql> SELECT * FROM INNODB_LOCK_WAITS;+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+|requesting_trx_id|requested_lock_id|blocking_trx_id|blocking_lock_id|+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+| 1B1F            | 1B1F:0:81967:6  | 1B19          | 1B19:0:81967:6 |+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­+­­­­­­­­­­­­­­­­+1 row in set (0.00 sec)

Page 89: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.89

InnoDB Information Schema tables

mysql> select trx_mysql_thread_id, trx_id from innodb_trx;+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­+| trx_mysql_thread_id | trx_id   |+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­+|                  31 | 1B1F     ||                  32 | 1B19     |+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­­+2 rows in set (0.00 sec)

Page 90: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.90

Insert Picture Here

MySQL Access Privilege System

Page 91: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.91

MySQL Access Privilege SystemOverview

No roles by default, limited user limits

All records are in the mysql database (schema)

Pluggable authentication since version 5.5

Connections● TCP/IP with login-password

● Socket (Unix)

● Named pipe (Windows)

Page 92: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.92

MySQL Access Privilege SystemHow to handle connection issues

SELECT user, host FROM mysql.user

Most descriptive host first, then wildcard

Socket connection by default on Unix

Page 93: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.93

MySQL Access Privilege SystemSort ordermysql> select user, host from mysql.user order by user desc, host desc;+­­­­­­+­­­­­­­­­­­­+| user | host       |+­­­­­­+­­­­­­­­­­­­+| root | localhost  || root | delly      || root | ::1        || root | 127.0.0.1  || foo  | %          ||      | localhost  |+­­­­­­+­­­­­­­­­­­­+6 rows in set (0.00 sec)

Page 94: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.94

MySQL Access Privilege SystemWrong access

SHOW GRANTS [FOR user@host]Grant tables

● mysql.db

● mysql.tables_priv

● mysql.columns_priv

● mysql.procs_priv

● mysql.proxies_priv

SELECT USER(), CURRENT_USER() 

Page 95: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.95

MySQL Access Privilege SystemGrants

mysql> show grants;+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| Grants for root@localhost                                                |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION      || GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION             |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+2 rows in set (0.02 sec)

Page 96: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.96

MySQL Access Privilege SystemGrants

mysql> show grants for foo@'%';;+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| Grants for foo@%                                      |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+| GRANT USAGE ON *.* TO 'foo'@'%'                       || GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%'         |+­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­+2 rows in set (0.00 sec)

Page 97: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.97

Insert Picture Here

THANK YOU!

https://twitter.com/#!/svetsmirnovahttps://blogs.oracle.com/svetasmirnova/

Page 98: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.98

Graphic Section Divider

Page 99: Basic MySQL Troubleshooting for Oracle DBAs

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.99

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract.It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.