sto1315bu vsan troubleshooting deep dive or … · vsan troubleshooting deep dive vmworld 2017...

Francis Daly & Javier Menendez

STO1315BU

#VMworld #STO1315BU

vSAN Troubleshooting Deep Dive

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#STO1315BU CONFIDENTIAL 2



bution

Agenda

1 Using common sense

2 vSAN Overview

3 vSAN Tools

4 Health

5 Use cases

6 Questions




bution

Plan Your Build




bution

Ensure You Have Backups (That Have Been Tested)




bution

vSAN Cluster – The World is Your Oyster

Disk

Group

1

Disk

Group

2

Disk

Group

5…

SSD

1

SSD

or

HDD

1

SSD

or

HDD

2

SSD

or

HDD

7…

Cache

Capacity

Host 1Host 2 Host 3 Host 64…

10GbE

• Cluster: 2-64 physical hosts

• Host: 1-5 disk groups

• Disk Group:

– 1 flash device for cache

– 1-7 flash or HDD devices for capacity




bution

Important Considerations

• Ensure all hardware is supported to ensure no performance degradation

• Disks/Controllers/Firmware/Drivers

• Objects:

• VM Home, VM Swap, VMDK

• Delta Disk, Memory Delta

vSphere vSAN

vSAN Datastore




bution

What if I Choose Incorrect Policy?

• Applied at per VM level, or VMDK level

• Define protection level & performance




bution

vSAN Objects and Components

• The vSAN datastore is an object store

• Each object made up of one or more components

• Policy will determine how components are distributed across cluster

C1

RAID-0 RAID-0

C2 W

RAID-0

200GB

RAID-1

FTT=1

Keeper_VM




bution

vSAN Objects and Components


• Object store allows you to meet granular availability and performance requirements


• Data (components) is distributed across cluster based on VM storage policy

C1

RAID-0 RAID-0

C2 W

RAID-0

200GB

RAID-1

FTT=1

RAID-1 requires 2n+1 hostsNeed >50% of components for object

to remain active (quorum)




bution

Network Partition


• Object store allows you to meet granular availability and performance requirements


• Data (components) is distributed across cluster based on VM storage policy

C1

RAID-0 RAID-0

C2 W

RAID-0

200GB

RAID-1

FTT=1

Witness or votes are used as tie

breaker if 2N+1 not satisfied#STO1315BU CONFIDENTIAL 11



bution

vSAN Object Component States

C1

RAID-0

C2 C3 C1

RAID-0

C2 C3 W

RAID-0

RAID-1

FTT=1

• Active. Component accessible

• Absent. Inaccessible, but no explicit error codes sensed

– Ex: Host outage, or EMM with “ensure accessibility”

– Rebuild begins after 60 minute timeout window

• Degraded. Inaccessible, with error codes sensed

– Ex: Device failure

– Rebuild begins immediately.

• Can check state in UI,CLI,RVC and more




bution

Tools



bution

Useful Tools for Trouble-shooting

Health Check

RVCESXCLIvRealize Ops

vSAN Observer




bution

ESXCLI



bution

VC Unavailable – Now What?




bution

Health Cluster List




bution

Investigating Yellow State

Threshold is

30Disk A = 25% Disk B = 60% Delta = 35%

Rebalance

required

Consider performance impact




bution

Where are the Most Serious Issues in my Cluster?

What is an object? vmdk,

vswp etc




bution

So What Happened?




bution

Diving Deeper




bution

Fundamentals of vSAN ResyncsResync is like DNA Replication!




bution

Resync Summary

Where else can I see this

vSphere Web Client

RVC: Resync Dashboard

Log Insight

vSAN Observer




bution

Resync Summary

Do not

Reboot

hosts/Maintenance mode

Remove hosts/disks

Do

Configure hosts to reboot

after PSOD

Increase default clomd

repair time




bution

vSAN Debug ListAre my disks keeping up?

Network

VM IO workload exceeds

bandwidth

Resync

Throttling will prioritize

VM IO’s




bution

Performance BottlenecksFast disks but still seeing large queues and failed IO’s

Manage expectations Controller queue depth

SSD gradeNetwork setup




bution

RVC



bution

vsan.check_state

• What state is my cluster in?

• I want to quickly know if there is anything serious happening in my cluster




bution

vsan.disks_stats 0

KB 2145267 – Understand vSAN on-disk format




bution

vsan.cluster_info




bution

Health UI



bution

Cluster Level Quick Peek

CEIP will need to be enabled




bution

Issues at Cluster Level




bution

vSAN Disk Balance




bution

What to Do if Rebalance is Required?




bution

vSAN Disk Balance




bution

What Happens During a Rebalance?




bution

Rebalance in Action




bution

Post Rebalance




bution

Health ServiceCLI



bution

vCenter vSAN Health Status Script

• Provides a text output for all the Health Tests

• Might be useful to run on the vCenter as the log bundles are being collected

• Manually copy off the text file

python /usr/lib/vmware-vpx/vsan-health/vsan-vc-health-status.py > /tmp/vsan-vc-health-status.txt




bution

vCenter vSAN Health Status Script

Runs the following RVC commands and collects their output

• vsan.cluster_info on each Cluster

• vsan.host_info for each host in every cluster

• vsan.vm_object_info on a host-per-host basis

• vsan.disks_info for each host on a per-cluster basis

• vsan.disks_stats

• vsan.check_limits

• vsan.check_state

• vsan.lldpnetmap

• vsan.obj_status_report

• vsan.resync_dashboard

• vsan.disk_object_info

What else does it do for me?

How much time would that save you on a call?




bution

ESXi vSAN Health Status Script

• Command-line tool to display the vSAN health for a particular node

What is the vSAN Health Status script?

• On each host individually, not vCenter• /usr/lib/vmware/vsan/bin/vsan-health-status.pyc

Where do I find it?

• [root@esxi] python /usr/lib/vmware/vsan/bin/vsan-health-

status.pyc

How do I run it?

See KB 2107705




bution

When to Use it

• The Health Service on vCenter is not available

• You can check the Health of individual nodes and their components

• Output is quite different to vCenter version

When vCenter is down




bution

ESXi vSAN Health Status script – What Will I Find in the Output

vSAN HCL related hardware info

• Displays key information about controllers to check against HCL

Limits summary

• How many components vs limit on the host

• How much space is consumed by components

Network summary

• Subnet used by vSAN

• Multicast addresses

Physical VSAN disk summary

• Info about physical disks including cmmds UUIDs




bution

Missing Disks



bution

I Added Hosts to my Cluster, Not Seeing Space




bution

Which Tools/Logs Do I Need

Track down missing MD’s

What is vobd.log?

What is boot.log?

What happens in cmmds?

esxcli vsan storage list – check if in cmmds




bution

esxcli vsan Storage List




bution

What to Do Next

Take a disk naa ID that is not “In CMMDS”

cd /var/run/log

less vobd.log

/”Disk UUID 12345678”

Disk not found




bution

What to Do Next

cd var/log

cat boot.log |less

/”Disk UUID 12345678”

LVM: 8355:Device naa.”Disk UUID 12345678” detected to be a snapshot:




bution

What Happened and Resolution

The disk cannot be added to the cluster because there is a file system on it

Disk was given a UUID, It was in a cluster, and used at some point

Verified with customer we could delete data

Used PartedUtil to kill partitions

Delete disk groups and create new disk groups




bution

Poor VM performanceSome VM’s are performing poorly and these VM’s are sitting on the vSAN datastore, this has resulted in effective data unavailability. Performance improved after some VSAN disks entered a failed state



bution

What Happened to the Disk?

less vmkernel.log | grep failed

Check the sense codes – scsi decoder




bution

What Else Happened? Controllers

Host ID




bution

Sense Code Output




bution

What Happened?

Three disks failed in short space of time

Controllers were aborting

What would you do?




bution

VMware vSAN Training

Training

VMware vSAN: Deploy and Manage [V6.6]

• Classroom

• Live Online

• Onsite

More Options

On Demand Courses

• Self-paced learning

• Meets certification requirements

VMware Learning Zone

• vSAN Troubleshooting

• Cloud-based learning

• Supplements traditional training

View complete options and descriptions:www.vmware.com/education #STO1315BU CONFIDENTIAL 58



bution

http://mylearn.vmware.com/mgrReg/plan.cfm?plan=39210&ui=www_edu

https://mylearn.vmware.com/mgrReg/plan.cfm?plan=51506&ui=www_edu

http://www.vmware.com/education

Global Support Services

Learn more about how VMware is radically transforming Customer Support through VMware Skyline™ technology.

• See a demo in the VMware booth in the Solutions Exchange

• Sign up for a Meet the Experts roundtable in the Schedule Builder on the VMworld mobile app or visit the Meet the Experts, Level 2

• Visit www.vmware.com/support/service/skyline




bution

http://www.vmware.com/support/service/skyline

Additional Education Resources

At VMworld 2017

• Education & Certification Lounge: VM Village

• Certification Exam Center: Jasmine EFG, Level 3

Online

• VMware Training: www.vmware.com/education

• VMware Certification: www.vmware.com/certification

Save 50% off VCP & VCAP

exams at

VMworld 2017




bution

http://www.vmware.com/education

http://www.vmware.com/certification

Prove Your Expertise with VMware Digital Badges

What are VMware Digital Badges?

• Single source that combines your credential and

a complete overview of your skills

• Way for you to easily share your accomplishments

in social media

• Provides employers with easy, valid verification of

VMware credentials

Digital Badges Online

• vmware.com/go/vSAN2017badge

• vmware.com/go/VxRail2017badge

VMware vSAN 2017 Specialist

Dell EMC VMwareCo-Skilled - VxRail 2017

NEW!VMware Digital Badges

Prove your expertiseon vSAN and

vSAN HCI environments




bution

http://www.vmware.com/go/vSAN2017badge

http://www.vmware.com/go/VxRail2017badge



bution

sto1315bu vsan troubleshooting deep dive or … · vsan troubleshooting deep dive vmworld 2017...

Documents