________________________________________________________________________

IBM Flex System Education

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) &

Options

May 2012 Study guide

GX40650 Release 1.10

This course is owned and published by the IBM MTS

Global Support Skills and Knowledge Enablement Team and created by Alex Badia.

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Preface

May 2012 2 GX40650r110.pdf

© International Business Machines Corporation, 2012 All rights reserved. IBM MTS Global Support Skills and Knowledge Enablement IBM Systems, Department EYGA. Building 203, Post Office Box 12195, Research Triangle Park, North Carolina 27709-2195 IBM reserves the right to change specifications or other product information without notice. This publication could include technical inaccuracies or typographical errors. References herein to IBM products and services do not imply that IBM intends to make them available in other countries. IBM provides this publication as is, without warranty of any kind —either expressed or implied—including the implied warranties of merchantability or fitness for a particular purpose. Some jurisdictions do not allow disclaimer of expressed or implied warranties. Therefore, this disclaimer may not apply to you. Data on competitive products is obtained from publicly obtained information and is subject to change without notice. Please contact the manufacturer for the most recent information. The following terms are trademarks or registered trademarks of IBM Corporation in the United States, other countries or both: Active Memory, Active PCI, AT, BladeCenter, the e-business logo, EasyServ, Enterprise X-Architecture, EtherJet, HelpCenter, HelpWare, IBM RXE-100 Remote Expansion Enclosure, IBM XA-32, IBM XA-64, IntelliStation, LANClient Control Manager, Memory ProteXion, NetBAY3, Netfinity, Netfinity Manager, Predictive Failure Analysis, RXE Expansion Port, SecureWay, ServeRAID, ServerProven, ServicePac, SMART Reaction, SMP Expansion Module, SMP Expansion Port, UM Services, Universal Manageability, Update Connector, Wake on LAN, XceL4 Server Accelerator Cache, XpandOnDemand scalability. IBM Corporation Subsidiaries: Lotus, Lotus Notes, Domino, and SmartSuite are trademarks of Lotus Development Corporation. Tivoli and Planet Tivoli are trademarks of Tivoli Systems, Inc. LLC, Adobe, and PostScript are trademarks of Adobe Systems, Inc. Intel Celeron, LANDesk®, MMX, Pentium II, Pentium III, Pentium 4, SpeedStep, and Xeon are trademarks or registered trademarks of Intel Corporation. Linux is a trademark of Linus Torvalds. Microsoft Windows® and Windows NT® are trademarks or registered trademarks of Microsoft Corporation. Other company, product, and service names may be trademarks or service marks of others. For more information, visit: http://www.ibm.com/legal/us/en/copytrade.shtml.

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Preface

May 2012 3 GX40650r110.pdf

Preface Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options This document may not be copied or sold, either in part or in whole, to non-IBM personnel. Please write your name and address below to personalize your copy.

Issued to: Address:

Current release date: May 2012 Current release level: 1.10

The information contained within this publication is current as of the date of the latest revision and is subject to change at any time without notice. Please forward all comments and suggestions regarding the course material format and content to your local IBM System x Service and Support Education country coordinator or contact.

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Table of Contents

May 2012 4 GX40650r110.pdf

Table of Contents Preface ............................................................................................................................ 3  

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options........................................................................................................................ 3  

Table of Contents ........................................................................................................... 4  Prerequisites ............................................................................................................... 5  Objectives ................................................................................................................... 6  

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options ........................................................................................................................ 7  

Overview ..................................................................................................................... 7  Flex System Enterprise Chassis & Options ........................................................... 7  

IBM Flex System Enterprise Chassis & Options – Lightpath ...................................... 8  Chassis Front Panel Card LEDs ........................................................................... 8  Power Supply LEDs ............................................................................................ 11  Compute Node LEDs .......................................................................................... 12  

Systems Management .............................................................................................. 17  Service Data Capture Instructions from the CMM CLI ........................................ 17  FSM Basic Chassis Management ....................................................................... 18  

Summary ....................................................................................................................... 21  

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Prerequisites and Objectives

May 2012 5 GX40650r110.pdf

Prerequisites A curriculum is a combination of specific roadmaps that are attempted in a logical sequence. Each roadmap is a set of course modules that are also attempted in a logical sequence. A brand curriculum consists of the roadmap types listed below. Attempting the courses in each roadmap in the sequence presented affords the best change of successful completion. Access the complete IBM Flex System curriculum using the following Global Support Enablement University (GSEU) link:

https://w3-connections.ibm.com/wikis/home?lang=en#/wiki/Global%20Support%20Enablement%20University/page/Flex%20MTS%20RSC%20and%20SSR%20Combined%20Curriculum

• Prerequisites: Prerequisite courses provide basic concepts in technologies, service tasks, and products. All prerequisites should be completed prior to attempting subsequent roadmaps in the curriculum.

• Fundamentals: Fundamental courses describe in general the architectures, technologies, capabilities, and limitations of the specific product brand. Fundamental courses should be attempted prior to problem determination and product specific course modules.

• Point of Reference Training: Point of Reference Training (PORT) is the use of the brand external product documentation to learn about the products as do our customers. This training is designed to provide service personnel the customer's point of reference for the products they have installed.

• New Technologies: New technologies courses introduce new major concepts not previously seen within the brand. Over time, new technologies courses are integrated into the fundamentals courses.

• Access and Interface Training: Access and Interface (A and I) training modules detail the ability to access service data and the use of typical application interfaces, such as system management, service management, and configuration management.

• Problem Determination Training: Problem Determination (PD) training details the processes and tasks used to isolate failures and generate a resolution plan for the products covered within the curriculum.

• Product and Options modules: Product and Options course modules provide specific information for each product covered within the curriculum. These course modules specifically relate to machine types and option part numbers, and provide the details of a particular product.

• Service Task Training: Service task training describes the processes and practices used by service personnel to perform problem determination, problem isolation, and action plans to resolve customer issues. Service task training is

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Prerequisites and Objectives

May 2012 6 GX40650r110.pdf

typically hands on classroom or virtual exercises and practice sessions using brand products.

• Compute Environment Training: Compute environment training details the typical customer use of the products within the curriculum, as well as external factors that can cause non obvious issues with these products.

• Education Assist Elements: Education Assist (EA) elements include installation and removal videos, simulators, trifold summaries, and self-running product or application demonstrations. EA elements are designed to be used as daily work aids, not as stand-alone course modules.

• Curriculum Assessments: Assessments for each course module within the curriculum are provided at the completion of the course module. Curriculum assessments are taken when a large majority of the curriculum has been completed.

The prerequisites for this course include the roadmaps and courses listed below:

http://xsnsftp.raleigh.ibm.com/docs/university3/prereq_sum_bycrs.asp?crscode=GX40650

Objectives Upon completion of this course, you will be able to:

1. Provide an overview of basic problem determination for the IBM Flex System Enterprise chassis and options.

2. Describe the IBM Flex System and options lightpath. 3. Identify the IBM Flex System locations. 4. Provide systems management information regarding the Flex System Manager

node and chassis management module.

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Overview

May 2012 7 GX40650r110.pdf

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options Overview Flex System Enterprise Chassis & Options Flex System chassis problem determination methodology is similar to the BladeCenter environment. Obtain the service data log information from either the chassis management module (CMM) or Flex System Manager (FSM) node. The most efficient way to populate the chassis with nodes is from the bottom up. For hardware, ensure that all connectors are intact and the correct parts are in the proper locations according to installation and configurations guides. Lightpath is used to alert the end user of a condition that requires attention. Not all notifications are fault based. Some instances that are not fault based include, but are not limited to information notifications, power good, and locator LEDs. Understand the IBM Flex System Enterprise chassis and option locations. Approach each problem logically by following the hardware communication path or be sure to search the appropriate logs to isolate the error.

Figure 1 – IBM Flex System Enterprise Chassis

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Lightpath

May 2012 8 GX40650r110.pdf

IBM Flex System Enterprise Chassis & Options – Lightpath Chassis Front Panel Card LEDs There are four LEDs on the front panel, all located at the bottom left corner of the chassis. The front customer interface panel card has chassis information LEDs and the fault LED light-up button. When the combination fault LED button is in deferred maintenance mode, the fault LED of a failed component (e.g., compute node, switch, power supply, fan, etc.) does not light. Pushing the button illuminates the fault LED of all failed components, which are visible from the front, or rear of the chassis. Internal fault LEDs (i.e., board LEDs next to DIMMs) does not light until activated by internal lightpath switch (today’s blue marked button on the system board).

Figure 2 – IBM Flex System Enterprise Chassis Information LEDs

There is a personality card in addition to the front panel card. The personality card provides two EEPROMs that contain chassis manufacturing vital product data (VPD) and the remaining space is for CMM control and usage. This card is also the control point for the front and rear service indicators. The state of the front chassis fault switch

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Lightpath

May 2012 9 GX40650r110.pdf

is via the personality card. Two temperature sensors on the personality card provide inlet temperature information from the switches to the CMMs. The personality card interfaces via a cable to the front panel card which provides a light up logo (used to indicate power on), a blue location LED, and a combination fault LED / switch. The personality card logic is power by 3.3V received from the CMMs. If there is no CMM in the system, the personality card forces the fault LED to turn on.

Figure 3 – Personality Card

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Lightpath

May 2012 10 GX40650r110.pdf

Figure 4 – IBM Flex System Enterprise Chassis Block Diagram

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Lightpath

May 2012 11 GX40650r110.pdf

Power Supply LEDs Each power supply has an AC good LED, DC good LED, and a fault LED. The AC/DC good LEDs illuminate green and the fault LED is amber.

Figure 5 – Power Supply

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Lightpath

May 2012 12 GX40650r110.pdf

Compute Node LEDs Figure 6 below displays the hard disk drive (HDD) activity and status LEDs as well as the power, identify, check log, and fault LEDs.

Figure 6 – Compute Node Front Bezel Locations

In addition, Figure 7 displays a LED panel located on the top right hand side of the compute node. This is a lightpath diagnostics panel considered to be located inside the node because it is not visible when plugged into the chassis. It is necessary to use the lightpath diagnostics panel when the fault LED is illuminated.

Figure 7 – Compute Node Locations Indicating Top LED panel

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Lightpath

May 2012 13 GX40650r110.pdf

Use the following power on sequence and observe the LEDs to confirm a successful boot operation. Wait until the LED on the power button is flashing slowly, before pressing the power button. Button is not responsive until the LED is solid. Flashing LED indicates node is starting the integrated management module (IMM) and doing low-level boot and diagnostics. Power button LED is on solid indicates initial IMM boot completed and the node is continuing to start imbedded OS and management functions. Monitor Check Log LED and Fault LED for error indications. Check Log LED on indicates a boot error. In the event this happens the servicer should first update the firmware. Second, reseat the node. If the first and second steps do not resolve the boot issue, then replace the system board. View the lightpath diagnostics panel inside the node when the fault LED is illuminated. This will indicate the following types of errors…battery, power, DIMM, IO expansion, memory, CPU, system board, drive, and temperature. Figure 8 displays the lightpath diagnostics panel.

Figure 8 – Lightpath Diagnostics Panel

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Locations

May 2012 14 GX40650r110.pdf

IBM Flex System Enterprise Chassis & Option Locations The following figures, 6 and 7, display front and rear chassis locations for the IBM Flex System. Bay 1 starts at the bottom left of the chassis and ends with bay 14 in the top right front side corner. At the rear, it displays locations for four switches, two CMMs, six power supplies, and ten fans.

Figure 9 – IBM Flex System Enterprise Chassis Bay Locations

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Locations

May 2012 15 GX40650r110.pdf

Figure 10 – IBM Flex System Enterprise Chassis Rear Power Supply, Fan, and Switch Bay

Locations

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Locations

May 2012 16 GX40650r110.pdf

The IBM Flex System Enterprise chassis has an outer skeleton. Inside there are compute node bay shelves, midplane, fan distribution cards, personality card, rear LED card, fan logic module, fillers (node, power, switch, CMM, fan), fans, and power supplies. Some of these are labeled in Figure 11 below.

Figure 11 – IBM Flex System Enterprise Chassis Components

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Systems Management

May 2012 17 GX40650r110.pdf

Systems Management The FSM and CMM are used for systems management. Management at the compute node level is available through the IMM. The IMM logs are passed up to both the CMM and FSM. Upon failure, the servicer should access the chassis map when operating with a CMM and FSM. Depending on the type of error, there will be different views to navigate. These include, but are not limited to problems view, status view, and CMM log. A servicer should consider the following when operating with a CMM only. Collect a CMM or OS log depending on the nature of the issue. A CMM log is necessary for platform events (memory, memory controllers, clocks, CPU, etc.). If no detail is available on the event start with an OS log for SAN, fibre, or disk issues. Then check the chassis switch logs.

Service Data Capture Instructions from the CMM CLI 1. To save the full service data logs type the command: displaysd -save .tgz -i

xxx.xxx.xxx.xxx -T mm[x] Where: = the name of the .tgz file to direct the output to. xxx.xxx.xxx.xxx = the IP address of the tftp server you are using. X = 1 or 2 depending on the CMM you are working on.

2. To copy the file down to your local system, use the following command prompt

command on your workstation or laptop: tftp -i xxx.xxx.xxx.xxx get .tgz c:\temp\.tgz Where xxx.xxx.xxx.xxx = the IP address of the TFTP server is the name of the tgz file you created to hold the service data.

3. When you use the CMM as the TFTP server, you can remove the file from the

CMM storage by entering this command: files -d tftproot/.tgz -T mm[x] Where = the name of the tgz file you collected from the CMM TFTP server. Where x = 1 or 2 depending on the CMM you are working with.

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Systems Management

May 2012 18 GX40650r110.pdf

FSM Basic Chassis Management The FSM has a chassis manager that allows the user to view chassis lightpath, compute nodes, switches, fans, and power supplies.

Figure 12 – FSM Chassis Manager Screen

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Systems Management

May 2012 19 GX40650r110.pdf

There are overlays that include status, access, LEDs, asset tags and energy/thermal. To display more statuses in a view, the user must select one of the filter settings for Status View. The user can select different overlays. For example, hardware access states, hardware LEDs, and component names. Figure 13 displays an overlay.

Figure 13 – FSM Overlay Screen

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Systems Management

May 2012 20 GX40650r110.pdf

Active Energy Manager, an advanced manager plug-in, has also been integrated into the chassis manager. This gives the user a quick sense of where thermal/energy problems may exist.

Figure 14 – Active Energy Manager Screen

Basic Problem Determination for the IBM Flex System Enterprise Chassis (8721) & Options – Summary

May 2012 21 GX40650r110.pdf

Summary This course has enabled you to:

1. Provide an overview of basic problem determination for the IBM Flex System Enterprise chassis and options.

2. Describe the IBM Flex System and options lightpath. 3. Identify the IBM Flex System locations. 4. Provide systems management information regarding the FSM and CMM.