mastering innodb diagnostics

1. Mastering InnoDB Diagnostics Harrison Fisk Senior Principal Technical Support Engineer

2. Overview

Lots of information

Hard to understand

3. Easy to get overwhelmed

Hard to figure out what is useful

4. Two methods for troubleshooting

Symptom based

What we will be doing today

Data based

5. Information Sources

6. Sources

MySQL info

SHOW ENGINE INNODB STATUS

7. InnoDB monitors

8. SHOW GLOBAL STATUS

9. INFORMATION_SCHEMAtables

SHOW PROCESSLIST

10. SHOW MUTEX STATUS

OS info

vmstat / iostator Performance Monitor

11. Gdb ( Poor mans profiler (PMP))

12. OS specific tools

13. SHOW INNODB STATUS

Be careful of truncation

innodb_status_file

14. Table monitors

Ensure enough time has passed

15. Exact format varies

InnoDB built-in

16. InnoDB plugin

17. XtraDB

18. Data Dictionary

19. Problems

InnoDB: Error: table test/parent already exists in InnoDB internal InnoDB: data dictionary. Have you deleted the .frm file InnoDB: and not used DROP TABLE? InnoDB: Cannot find table test/child2 from the internal data dictionary InnoDB: of InnoDB though the .frm file for the table exists. Maybe you InnoDB: have deleted and recreated InnoDB data files but have forgotten InnoDB: to delete the corresponding .frm files of InnoDB tables?

Various errors about not being in sync

20. Can't drop and/or create tables or databases

21. Example errors:

22. Diagnostics

InnoDB table monitor

Dumps InnoDB data dictionary

23. Output to error log

24. CREATE TABLE innodb_table_monitor (id int) ENGINE=innodb;

25. Interpreting

Ignore system tables

NamedSYS_

Ignore hidden fields from InnoDB

DB_ROW_ID

26. DB_TRX_ID

27. DB_ROLL_PTR

Ignore hidden indexes

GEN_CLUST_INDEX

28. Solutions

Ensure.frmmatches InnoDB

Create.frmas needed

Create another table with same structure

29. Copy.frmto the name of the table

Drop orphaned.frmfiles

If not in InnoDB

30. Remove them

31. Crashing

32. Problem

mysqld Crash

Signal 11

Segfault

Signal 6

Abort

33. InnoDB loves this one

May not actually notice immediately

mysqld_safe restarts

34. Diagnostics

Error log

Hopefully it has info in it

Core file

Not enabled by default

35. OS often restricts setuid cores

/proc/sys/fs/suid_dumpable

36. coreadm

General log

When all else fails

37. Interpreting Error log

Error log

Look for signal type

Signal 11

38. Signal 6

Backtrace

May or may not be present

OS

39. Version

In 5.0 and before you need to resolve it

resolve_stack_dump

Search bug system for function names

40. Interpreting Cores

Must be enabled

41. Memory image produced

42. Requires exactly the same binary and libraries

43. With rpm ensure debuginfo is installed

44. Non-stripped binaries

45. Load with gdb

thread apply all bt

46. Use PMP to summarize if needed

Solaris:pstack

47. Windows: Minidump/DrWatson/Windbg

48. Solutions

Find cause of the crash

InnoDB doing on purpose

49. Real crash

Find query causing problem

Stop doing it

50. Report bug to us

Upgrade

Lots of bugs fixed each release

51. Locking

52. Problems

Deadlocks

Real deadlocks

ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Lock wait timeouts

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

General locking slowness

53. Diagnostics

SHOW INNODB STATUS

TRANSACTIONSsection

innodb_lock_monitortable

Changes output ofSHOW INNODB STATUS

INFORMATION_SCHEMA

INNODB_LOCKS

54. INNODB_LOCK_WAITS

SHOW GLOBAL STATUS LIKE 'innodb_row_lock%'

55. Interpreting

Find what lock is needed

Who is holding conflicting lock

Look for type of lock

X (exclusive)

56. S (shared)

Relatively rare

Others

Look for what the lock is on

Table (primary key)

57. Index

58. Solutions

Fix queries to reduce locks

59. Shared locks

Find source

60. Eliminate

Upgrade to 5.1 run in read committed

61. Remove source of lock

AUTO-INC

Upgrade to 5.1

innodb_autoinc_lock =2?

Long running transactions

62. Performance

63. Overview

Lots of potential performance issues

Disk I/O

Log files

64. Table space

CPU

Too many rows being read

65. Thread thrashing

Gather lots of information up front

Big picture, not details first

Ensure it is during problem time

66. Disk: Log files

Many small serialized synchronous writes

67. InnoDB redo log writes bottleneck

fdatasync () is expensive in many setups

innodb_flush_log_at_trx_commit

1 Safest, sync every commit

68. 2 Write every commit

69. 0 Write only every second

70. Diagnostics

SHOW INNODB STATUS

State of transactions

COMMITTED IN MEMORY

71. PREPARED

FILE I/O

Pending flushes (fsync) log

72. fsyncs/s

iostat

Number of write operations ( w/s )

awaitand/orsvctm

73. Solutions

InnoDB plugin group commit

74. Changeinnodb_flush_log_at_trx_commit

75. Better disks

BBU write cache

76. Faster writes

77. Disk: Tablespace

Reading and writing your actual tables and indexes

78. More common with very large tables

Compare to buffer pool size

Sometimes caused by read ahead in InnoDB

79. Diagnostics

SHOW INNODB STATUS

Buffer pool hit rate

Watch absolute values too

FILE I/O

Reads/s

Avg bytes/read shows reada head

Pending reads

Read ahead

Pages read ahead/evicted without access

iostat

avgqu-sz

80. await

81. Solutions

Optimize queries

82. Increaseinnodb_buffer_pool_size

83. New LRU algorithm in InnoDB plugin

84. Tune/disable readahead

In plugin, can be tuned

85. In non-plugin

Use gdb hack

86. set srv_startup_is_before_trx_rollback_phase=1

Buy better disks

87. CPU: Lots of Rows

Bad queries can read lots of rows

Or lots and lots of smaller queries

Data fully cached

Lots of buffer pool reading

88. Often smaller dataset

Eventually will bottleneck InnoDB on CPU

89. Diagnostics

SHOW INNODB STATUS

ROW OPERATIONS

Reads/s

SHOW GLOBAL STATUS LIKE 'Handler%'

Type of row operations going on

Slow query log

log-queries-not-using-indexes

90. Solutions

Fix your queries

Ensure proper index usage

Do less queries

External caching

Scale reads

Replication

91. CPU: Thread Thrashing

InnoDB can have issues at high concurrency

92. Later releases help reduce this dramatically

InnoDB plugin

93. Further versions

XtraDB

94. MySQL 5.5

95. Diagnostics

SHOW INNODB STATUS

Lots of queued threads

TRANSACTIONS

96. Queries in queue

ROW OPERATIONS

reads/s is low

vmstat

Very high CPU usage

97. Lots of cs context switches

SHOW MUTEX STATUS

98. Solutions

innodb_thread_concurrency

Very high or 0 works best until it doesn't

Sometimes very low works best

99. A bit of a black art in tuning

innodb_concurrency_tickets

Size based on number of rows being touched

Use newer version

Lots of work to help this in InnoDB Plugin

100. Questions?

101. The presentation is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

mastering innodb diagnostics

Technology