living with a cephalopod: daily care & feeding of ceph storage
TRANSCRIPT
SYSADMIN’S TOOLBOX TOOLS FOR RUNNING CEPH IN
PRODUCTION
Paul Evansprincipal architect
daystrom technology groupPaul at Daystrom dot com
san francisco
ceph day
March 12, 2015
WHAT’S IN THIS TALK
• Tools to help Understand Ceph Status
• Tips for Troubleshooting Faster
• Won’t Cover it All
• Maybe some Fun
WHAT COULD BE SIMPLER THAN THE….CLI ?
Ceph’s CLI is Great, but…
REALITY: many Operations Teams juggle too many technologies already…
Do they need to learn another CLI?
GUI TOOL OPT IONS
Calamari VSM InkScope ceph-dash
Backing From Red Hat Intel Orange Labs Christian Eichelmann
Lastest Version 1.2.3 2014.12-0.9.1 1.1 1.0
Release Date Sep 2014 Dec 2014 Jan 2015 Feb 2015
Capabilities Monitor + Light Config
Monitor + Config
Monitor + Light Config Monitor Only
Compatability Wide Limited Wide Wide
MONITORINGCalamari VSM InkScope ceph-dash
Mon Status Y Y Y YOSD Status Y Y Y Y
OSD-Host Mapping Y Y Y YPG Status Y Y Y Y
PG-OSD Mapping N N Y NMDS Status N Y Y NHost Status Y Y Y Y
Capacity Utilization Y via Groups Y YThroughput (Cluster) N Y Y Y
IOPS (Cluster) Y Y Y YErrors/Warnings Y Y Y Y
View Logs Y N N NSend Alerts (email) N N N via nagios plug-in
Charts/Graphs Y N N via nagios plug-in
MANAGEMENTCalamari VSM InkScope ceph-dash
Deploy a Cluster N Y N NDeploy Hosts (add/remove) N Y N N
Deploy Storage Groups (create)
N Y N NCluster Services (daemons) OSD only Y N(?) NCluster Settings (ops flags) Y N Y N
Cluster Settings (parameters) Y N View NCluster Settings (CRUSH
map/rrules)N Partial View N
Cluster Settings (EC Profiles) N Y Y NOSD (start/stop/in/out) Partial Y Y N
Pools (Replicated) Y (limited) Y Y NPools (EC & Tiering) N Y Partial N
RBDs N Partial N NS3/Swift Users/Buckets N N Y N
Link to OpenStack Nova N Y N N
TH
E ‘
NO
GU
I’ O
PT
ION
ceph@m01:~$ ceph osd tree# id weight type nameup/down reweight-1 213.6 root default
-2 43.44 host s019 3.62 osd.9 down 010 3.62 osd.10 down 00 3.62 osd.0 down 05 3.62 osd.5 down 01 3.62 osd.1 down 07 3.62 osd.7 down 03 3.62 osd.3 down 04 3.62 osd.4 down 02 3.62 osd.2 down 011 3.62 osd.11 down 0
-4 43.44 host s0324 3.62 osd.24 up 125 3.62 osd.25 up 126 3.62 osd.26 up 127 3.62 osd.27 up 128 3.62 osd.28 up 129 3.62 osd.29 up 130 3.62 osd.30 up 131 3.62 osd.31 up 132 3.62 osd.32 up 133 3.62 osd.33 up 1
ceph osd treeceph health detailHEALTH_ERR 7 pgs degraded; 12 pgs down; 12 pgs peering; 1 pgs recovering; 6 pgs stuck unclean; 114/3300 degraded (3.455%); 1/3 in osds are down...pg 0.5 is down+peeringpg 1.4 is down+peering...osd.1 is down since epoch 69, last address 192.168.106.220:6801/865
ceph health
ceph pg dump_stuck staleceph pg dump_stuck inactiveceph pg dump_stuck unclean
ceph pg dump_stuck
TH
E ‘
NO
GU
I’ O
PT
ION
pg_stat objects bytes status up up_pri 10.f8 3042 25474961408 active+remapped [58,2147483647,24,20,55,59,2147483647,27] 58 10.7fa 3029 25375584256 active+remapped [51,20,60,28,2147483647,61,2147483647,11] 51 10.716 2990 25052532736 inactive [9,44,10,55,24,2147483647,47,2147483647] 9
ceph pg dump_stuck
VS
M:
TR
OU
BL
ES
HO
OT
ING
Repeated'auto*out'or'inability'to'restart'auto*out'OSD'
suggests'failed'or'failing'disk'
VSM'periodically'probes'drive'path'–'missing'drive'path'missing'indicates'complete'disk'failure'
A'set'of'auto*out'OSDs'that'share'the'same'
journal'SSD'suggests'failed'or'failing'journal'SSD'
VSM'periodically'probes'drive'path'–'missing'drive'path'indicates'complete'disk'(or'
controller)'failure'
VS
M:
TR
OU
BL
ES
HO
OT
ING
Restore'OSDs Managing&Storage&Devices&
Restore&OSDs&
Wait&
Select&
Sort&
Confirm&
Verify&(may&need&to&sort&again)&
When i t ’s t ime to go deep…
/var/log/ceph/ceph.log/var/log/ceph/ceph-mon.[host].log/var/log/ceph/ceph-osd.[xx].log
ceph tell osd.[xx] injectargs --debug-osd 0/5
REMINDER ABOUT CLUSTERSClusters rarely do things instantly.
Clusters can be like a Flock of Sheep - it starts to move in the right directly slowly and then picks up speed
(don’t run it off a cliff)
VS
M:
FAS
T D
EP
LO
YCreate&Cluster Ge#ng&Started&
Create&new&Ceph&cluster&
All&servers&present&
Correct&subnets&and&IP&addresses&
Correct&number&of&disks&iden>fied&
At&least&three&monitors&&&odd&number&of&
monitors&
Servers&located&in&correct&zone&
Servers&responsive&
One&Zone&with&serverDlevel&replica>on&