STO1315BU vSAN Troubleshooting Deep Dive or ?· vSAN Troubleshooting Deep Dive VMworld 2017 Content:…

Download STO1315BU vSAN Troubleshooting Deep Dive or ?· vSAN Troubleshooting Deep Dive VMworld 2017 Content:…

Post on 29-Aug-2018

213 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>Francis Daly &amp; Javier Menendez</p><p>STO1315BU</p><p>#VMworld #STO1315BU</p><p>vSAN Troubleshooting Deep Dive</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p> This presentation may contain product features that are currently under development.</p><p> This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.</p><p> Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.</p><p> Technical feasibility and market demand will affect final delivery.</p><p> Pricing and packaging for any new technologies or features discussed or presented have not been determined.</p><p>Disclaimer</p><p>#STO1315BU CONFIDENTIAL 2</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Agenda </p><p>1 Using common sense</p><p>2 vSAN Overview</p><p>3 vSAN Tools</p><p>4 Health</p><p>5 Use cases</p><p>6 Questions</p><p>#STO1315BU CONFIDENTIAL 3</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Plan Your Build</p><p>#STO1315BU CONFIDENTIAL 4</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Ensure You Have Backups (That Have Been Tested)</p><p>#STO1315BU CONFIDENTIAL 5</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Cluster The World is Your Oyster</p><p>Disk</p><p>Group</p><p>1</p><p>Disk</p><p>Group</p><p>2</p><p>Disk</p><p>Group</p><p>5</p><p>SSD</p><p>1</p><p>SSD </p><p>or </p><p>HDD</p><p>1</p><p>SSD </p><p>or </p><p>HDD</p><p>2</p><p>SSD </p><p>or </p><p>HDD</p><p>7</p><p>Cache</p><p>Capacity</p><p>Host 1Host 2 Host 3 Host 64</p><p>10GbE</p><p> Cluster: 2-64 physical hosts</p><p> Host: 1-5 disk groups</p><p> Disk Group:</p><p> 1 flash device for cache</p><p> 1-7 flash or HDD devices for capacity</p><p>#STO1315BU CONFIDENTIAL 6</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Important Considerations</p><p> Ensure all hardware is supported to ensure no performance degradation</p><p> Disks/Controllers/Firmware/Drivers</p><p> Objects: </p><p> VM Home, VM Swap, VMDK</p><p> Delta Disk, Memory Delta</p><p>vSphere vSAN</p><p>vSAN Datastore</p><p>#STO1315BU CONFIDENTIAL 7</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What if I Choose Incorrect Policy?</p><p> Applied at per VM level, or VMDK level</p><p> Define protection level &amp; performance</p><p>#STO1315BU CONFIDENTIAL 8</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Objects and Components</p><p> The vSAN datastore is an object store</p><p> Each object made up of one or more components</p><p> Policy will determine how components are distributed across cluster</p><p>C1</p><p>RAID-0 RAID-0</p><p>C2 W</p><p>RAID-0</p><p>200GB</p><p>RAID-1</p><p>FTT=1</p><p>Keeper_VM</p><p>#STO1315BU CONFIDENTIAL 9</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Objects and Components</p><p> The vSAN datastore is an object store</p><p> Object store allows you to meet granular availability and performance requirements</p><p> Each object made up of one or more components</p><p> Data (components) is distributed across cluster based on VM storage policy</p><p>C1</p><p>RAID-0 RAID-0</p><p>C2 W</p><p>RAID-0</p><p>200GB</p><p>RAID-1</p><p>FTT=1</p><p>RAID-1 requires 2n+1 hostsNeed &gt;50% of components for object </p><p>to remain active (quorum)</p><p>#STO1315BU CONFIDENTIAL 10</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Network Partition</p><p> The vSAN datastore is an object store</p><p> Object store allows you to meet granular availability and performance requirements</p><p> Each object made up of one or more components</p><p> Data (components) is distributed across cluster based on VM storage policy</p><p>C1</p><p>RAID-0 RAID-0</p><p>C2 W</p><p>RAID-0</p><p>200GB</p><p>RAID-1</p><p>FTT=1</p><p>Witness or votes are used as tie </p><p>breaker if 2N+1 not satisfied#STO1315BU CONFIDENTIAL 11</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Object Component States</p><p>C1</p><p>RAID-0</p><p>C2 C3 C1</p><p>RAID-0</p><p>C2 C3 W</p><p>RAID-0</p><p>RAID-1</p><p>FTT=1</p><p> Active. Component accessible</p><p> Absent. Inaccessible, but no explicit error codes sensed</p><p> Ex: Host outage, or EMM with ensure accessibility</p><p> Rebuild begins after 60 minute timeout window</p><p> Degraded. Inaccessible, with error codes sensed</p><p> Ex: Device failure</p><p> Rebuild begins immediately.</p><p> Can check state in UI,CLI,RVC and more</p><p>#STO1315BU CONFIDENTIAL 12</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Tools</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Useful Tools for Trouble-shooting</p><p>Health Check</p><p>RVCESXCLIvRealize Ops</p><p>vSAN Observer</p><p>#STO1315BU CONFIDENTIAL 14</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>ESXCLI</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>VC Unavailable Now What?</p><p>#STO1315BU CONFIDENTIAL 16</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Health Cluster List</p><p>#STO1315BU CONFIDENTIAL 17</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Investigating Yellow State</p><p>Threshold is </p><p>30Disk A = 25% Disk B = 60% Delta = 35%</p><p>Rebalance </p><p>required</p><p>Consider performance impact</p><p>#STO1315BU CONFIDENTIAL 18</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Where are the Most Serious Issues in my Cluster?</p><p>What is an object? vmdk, </p><p>vswp etc</p><p>#STO1315BU CONFIDENTIAL 19</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>So What Happened?</p><p>#STO1315BU CONFIDENTIAL 20</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Diving Deeper</p><p>#STO1315BU CONFIDENTIAL 21</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Fundamentals of vSAN ResyncsResync is like DNA Replication!</p><p>#STO1315BU CONFIDENTIAL 22</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Resync Summary</p><p>Where else can I see this</p><p>vSphere Web Client</p><p>RVC: Resync Dashboard</p><p>Log Insight</p><p>vSAN Observer</p><p>#STO1315BU CONFIDENTIAL 23</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Resync Summary</p><p>Do not</p><p>Reboot </p><p>hosts/Maintenance mode</p><p>Remove hosts/disks</p><p>Do</p><p>Configure hosts to reboot </p><p>after PSOD</p><p>Increase default clomd</p><p>repair time</p><p>#STO1315BU CONFIDENTIAL 24</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Debug ListAre my disks keeping up?</p><p>Network</p><p>VM IO workload exceeds </p><p>bandwidth</p><p>Resync</p><p>Throttling will prioritize </p><p>VM IOs</p><p>#STO1315BU CONFIDENTIAL 25</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Performance BottlenecksFast disks but still seeing large queues and failed IOs</p><p>Manage expectations Controller queue depth</p><p>SSD gradeNetwork setup</p><p>#STO1315BU CONFIDENTIAL 26</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>RVC</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vsan.check_state</p><p> What state is my cluster in?</p><p> I want to quickly know if there is anything serious happening in my cluster</p><p>#STO1315BU CONFIDENTIAL 28</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vsan.disks_stats 0</p><p>KB 2145267 Understand vSAN on-disk format</p><p>#STO1315BU CONFIDENTIAL 29</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vsan.cluster_info</p><p>#STO1315BU CONFIDENTIAL 30</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Health UI</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Cluster Level Quick Peek</p><p>CEIP will need to be enabled</p><p>#STO1315BU CONFIDENTIAL 32</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Issues at Cluster Level</p><p>#STO1315BU CONFIDENTIAL 33</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Disk Balance</p><p>#STO1315BU CONFIDENTIAL 34</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What to Do if Rebalance is Required?</p><p>#STO1315BU CONFIDENTIAL 35</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vSAN Disk Balance</p><p>#STO1315BU CONFIDENTIAL 36</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What Happens During a Rebalance?</p><p>#STO1315BU CONFIDENTIAL 37</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Rebalance in Action</p><p>#STO1315BU CONFIDENTIAL 38</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Post Rebalance</p><p>#STO1315BU CONFIDENTIAL 39</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Health ServiceCLI</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vCenter vSAN Health Status Script</p><p> Provides a text output for all the Health Tests</p><p> Might be useful to run on the vCenter as the log bundles are being collected</p><p> Manually copy off the text file</p><p>python /usr/lib/vmware-vpx/vsan-health/vsan-vc-health-status.py &gt; /tmp/vsan-vc-health-status.txt</p><p>#STO1315BU CONFIDENTIAL 41</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>vCenter vSAN Health Status Script</p><p>Runs the following RVC commands and collects their output</p><p> vsan.cluster_info on each Cluster</p><p> vsan.host_info for each host in every cluster</p><p> vsan.vm_object_info on a host-per-host basis</p><p> vsan.disks_info for each host on a per-cluster basis</p><p> vsan.disks_stats</p><p> vsan.check_limits</p><p> vsan.check_state</p><p> vsan.lldpnetmap</p><p> vsan.obj_status_report</p><p> vsan.resync_dashboard</p><p> vsan.disk_object_info</p><p>What else does it do for me?</p><p>How much time would that save you on a call?</p><p>#STO1315BU CONFIDENTIAL 42</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>ESXi vSAN Health Status Script</p><p> Command-line tool to display the vSAN health for a particular node</p><p>What is the vSAN Health Status script?</p><p> On each host individually, not vCenter /usr/lib/vmware/vsan/bin/vsan-health-status.pyc</p><p>Where do I find it?</p><p> [root@esxi] python /usr/lib/vmware/vsan/bin/vsan-health-</p><p>status.pyc</p><p>How do I run it?</p><p>See KB 2107705</p><p>#STO1315BU CONFIDENTIAL 43</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>When to Use it</p><p> The Health Service on vCenter is not available</p><p> You can check the Health of individual nodes and their components</p><p> Output is quite different to vCenter version</p><p>When vCenter is down</p><p>#STO1315BU CONFIDENTIAL 44</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>ESXi vSAN Health Status script What Will I Find in the Output</p><p>vSAN HCL related hardware info</p><p> Displays key information about controllers to check against HCL </p><p>Limits summary</p><p> How many components vs limit on the host</p><p> How much space is consumed by components</p><p>Network summary</p><p> Subnet used by vSAN</p><p> Multicast addresses</p><p>Physical VSAN disk summary</p><p> Info about physical disks including cmmds UUIDs</p><p>#STO1315BU CONFIDENTIAL 45</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Missing Disks</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>I Added Hosts to my Cluster, Not Seeing Space</p><p>#STO1315BU CONFIDENTIAL 47</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Which Tools/Logs Do I Need</p><p>Track down missing MDs </p><p>What is vobd.log? </p><p>What is boot.log? </p><p>What happens in cmmds? </p><p>esxcli vsan storage list check if in cmmds</p><p>#STO1315BU CONFIDENTIAL 48</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>esxcli vsan Storage List</p><p>#STO1315BU CONFIDENTIAL 49</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What to Do Next</p><p>Take a disk naa ID that is not In CMMDS </p><p>cd /var/run/log</p><p>less vobd.log</p><p>/Disk UUID 12345678</p><p>Disk not found</p><p>#STO1315BU CONFIDENTIAL 50</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What to Do Next</p><p>cd var/log </p><p>cat boot.log |less </p><p>/Disk UUID 12345678</p><p>LVM: 8355:Device naa.Disk UUID 12345678 detected to be a snapshot:</p><p>#STO1315BU CONFIDENTIAL 51</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What Happened and Resolution</p><p>The disk cannot be added to the cluster because there is a file system on it</p><p>Disk was given a UUID, It was in a cluster, and used at some point </p><p>Verified with customer we could delete data</p><p>Used PartedUtil to kill partitions</p><p>Delete disk groups and create new disk groups</p><p>#STO1315BU CONFIDENTIAL 52</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Poor VM performanceSome VMs are performing poorly and these VMs are sitting on the vSAN datastore, this has resulted in effective data unavailability. Performance improved after some VSAN disks entered a failed state</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What Happened to the Disk?</p><p>less vmkernel.log | grep failed</p><p>Check the sense codes scsi decoder</p><p>#STO1315BU CONFIDENTIAL 54</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What Else Happened? Controllers</p><p>Host ID</p><p>#STO1315BU CONFIDENTIAL 55</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>Sense Code Output</p><p>#STO1315BU CONFIDENTIAL 56</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>What Happened?</p><p>Three disks failed in short space of time</p><p>Controllers were aborting</p><p>What would you do?</p><p>#STO1315BU CONFIDENTIAL 57</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>VMware vSAN Training</p><p>Training</p><p>VMware vSAN: Deploy and Manage [V6.6]</p><p> Classroom</p><p> Live Online</p><p> Onsite</p><p>More Options</p><p>On Demand Courses</p><p> Self-paced learning </p><p> Meets certification requirements</p><p>VMware Learning Zone</p><p> vSAN Troubleshooting</p><p> Cloud-based learning </p><p> Supplements traditional training </p><p>View complete options and descriptions:www.vmware.com/education #STO1315BU CONFIDENTIAL 58</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p><p>http://mylearn.vmware.com/mgrReg/plan.cfm?plan=39210&amp;ui=www_eduhttps://mylearn.vmware.com/mgrReg/plan.cfm?plan=51506&amp;ui=www_eduhttp://www.vmware.com/education</p></li><li><p>Global Support Services</p><p>Learn more about how VMware is radically transforming Customer Support through VMware Skyline technology.</p><p> See a demo in the VMware booth in the Solutions Exchange</p><p> Sign up for a Meet the Experts roundtable in the Schedule Builder on the VMworld mobile app or visit the Meet the Experts, Level 2</p><p> Visit www.vmware.com/support/service/skyline</p><p>#STO1315BU CONFIDENTIAL 59</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p><p>http://www.vmware.com/support/service/skyline</p></li><li><p>Additional Education Resources</p><p>At VMworld 2017</p><p> Education &amp; Certification Lounge: VM Village</p><p> Certification Exam Center: Jasmine EFG, Level 3</p><p>Online</p><p> VMware Training: www.vmware.com/education</p><p> VMware Certification: www.vmware.com/certification</p><p>Save 50% off VCP &amp; VCAP</p><p>exams at </p><p>VMworld 2017</p><p>#STO1315BU CONFIDENTIAL 60</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p><p>http://www.vmware.com/educationhttp://www.vmware.com/certification</p></li><li><p>Prove Your Expertise with VMware Digital Badges</p><p>What are VMware Digital Badges?</p><p> Single source that combines your credential and </p><p>a complete overview of your skills </p><p> Way for you to easily share your accomplishments </p><p>in social media</p><p> Provides employers with easy, valid verification of</p><p>VMware credentials</p><p>Digital Badges Online</p><p> vmware.com/go/vSAN2017badge</p><p> vmware.com/go/VxRail2017badge</p><p>VMware vSAN 2017 Specialist</p><p>Dell EMC VMwareCo-Skilled - VxRail 2017 </p><p>NEW!VMware Digital Badges</p><p>Prove your expertiseon vSAN and</p><p>vSAN HCI environments</p><p>#STO1315BU CONFIDENTIAL 61</p><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p><p>http://www.vmware.com/go/vSAN2017badgehttp://www.vmware.com/go/VxRail2017badge</p></li><li><p>VMworld</p><p> 2017 C</p><p>ontent: N</p><p>ot for p</p><p>ublicatio</p><p>n or dis</p><p>tribution</p></li><li><p>VMworld</p><p> 2017 C</p><p>onte