fas32xx service v2

Doc Rev -017b

README FIRST This AP has been updated to include commands for systems running "Cluster-Mode" (C-Mode) ONTAP.

First Fixed Release7.3.x 8.0.x 8.1.x Action

N/A 8.0.2 P3 8.1P1

Part Replacement Process Changes:

N/A N/A N/A

N/A N/A N/A

N/A N/A N/A

Link to Statement of Volatility by Platform is: http://support.netapp.com/info/web/ECMP1132988.html

If the 3200, SA320 system is running any version of ONTAP 7.3.5 or 7.3.6, the "memory" test in diag version A.00.02 can panic the partner node that is up and running. The diag section has been modified to check for this situation.

No "Failed" Disks can exist in the target node in a HA config or the disk reassign will not execute. The AP covers this.

AP doc rev is at top of page - If using hard-copy for secure site, be sure to print all the linked documents in this AP.

If min version not met, patch ONTAP prior to part replace or a system outage is required.

643008 Min FW required for "SSM, ASM, BSM" disks in N/A 8.0.4 P1

8.1.2 P1LinkedFW Table

DS4243 Shelves - Click FW link for drive list.

InternalTSB-1110-04

HA Configurations: Additional TO/GB process required to sync the system-IDs across the HA pair.

This Action Plan includes the additional Takeover/Giveback process.The TSB link is NetApp internal only.

IF IOM FW is down rev, apply FW rollup prior to part replacement2.

1 Output of: [run local] storage show disk -x will list the disk models and the FW of each. 2 Output of: [run local] sasadmin expander_map will list the shelf and IOM model with the revision of the IOM FW. 3 "Int" means this is a "NetApp Employee Internal" Link.

489060 NDMP, Qtree-SnapMirror, Vol-SnapMirror or SnapVault processes can hang TO/GB

README FIRST

If this is a V-Series system with a 3rd party storage array or has SAN attached Tape drives, confirm a storage admin is available to update the LUN Masking and/or switch zoning if the internal FC adapters are connected to the array.

Pre-ONTAP 8.2 Part Replacement Bugs and Part Replacement Process Issues that must be addressed. Skip IF ONTAP 8.2, otherwise the minimum ONTAP and FW version is defined in the table below. If ONTAP is not patched to eliminate bug(s), a "system outage" is required for part replacement. Questions? Call Support.

Do NOT run the fcache diags, the diag section provides steps to disable the fcache diags.

This AP covers the steps for configuring a replacement Chassis (ha-config issue). A STOP message is added to the "Verify FC Adapter Configuration / Chassis Check" Section with a link to the additional steps to be performed.

Bug Description509898

Master-Slave System-ID bug

Apply Disk FW rollup prior to part replace if don't have any X302_SMOOS01TSSM or X310_SMOOST50SSM disks1.

This Action Plan includes the process to stop and restart these activities.

647231 Shelf Module FW Bugs on IOM3/6 ModulesN/A 8.0.4 P1

8.1.2 P1647235 Min FW: IOM3=>152, IOM6=>142 required

590488 Disruptive MB replacements: A TO/GB from the repaired node is required

This Action Plan includes the process to re-enable the "controller-failover" and perform the additional TO/GB.

Controller Module Replacement for the V-FAS3200 Family For NetApp Authorized Service Engineers

I. Appliance/Controller Visual Checks XI. Set date and time on the RTCII. Node Pre-Checks XII. Service Processor Firmware CheckIII. Node State Check and Shutdown Procedure XIII. Run Diagnostics (5-20 min based on model)IV. ONTAP Check XIV. Verify FC Adapter Configuration / Chassis CheckV. Capture the Current System Configuration XV. Capture new System-ID on replacement ControllerVI. XVI. Disk Reassign

XVII. Boot PROM Variable ChecksVII. Move SFPs - Move PCI Cards XVIII. Boot the Operating System - 'giveback' if applicableVIII. Move the NVMEM Battery, NVMEM & Main Memory DIMMs. XIX. NetApp Storage Encryption (NSE) System? IX. Move the Boot Device XX.X. Partially Reinsert the Controller and reconnect the cables

I.Step

1 The V-FAS3200 series is available in several configuration options. Review Figures 2-4 and TERMS text box.

2 Continue with Section I on next page.

Page 1 of 27

SECTION OUTLINE for a V-FAS3200 (FAS3210/20/40/50/70) Appliance Controller Module ReplacementThis procedure will take 90-180 minutes

Open the System: Remove the cables, Cable Management Tray and extract the Controller

Complete system setup, Registration, Submit logs, Part return

V-FAS3200 Family: Appliance / Controller Visual Checks Action Description

V-FAS3200 Series System Configuration Options


V-FAS3240/70: "A" Controller and IOXM (CI)

V-FAS32x0: Dual Controllers (CC)

Single "A" Controller-Base (CB) (V-FAS3210 Only)

Fig 4

Fig 3

Fig 2

A

A

B

A

AC Power LED

V-FAS32x0 Model Number

3u

Fig 1

Controller activity LEDs: If LED flashes GREEN, that controller is online.

" ! " LED is ON when hardware failures are detected or if controller failover is disabled.

Filler Plate

The NVMEM LED on the faceplate will start flashing when power is removed from the controller if the system is "waiting for giveback", or the system was not shutdown properly (uncommitted data). Follow the steps in Section VI carefully.

STOP !!

NVMEM (TOP) LED- Flashes Green once every 2 seconds when power is off and NVMem is dirty - See STOP!! below

Fig 5

Controller Rear View

TERMS IOXM: I/O eXpansion Module (B-slot only) CB: Controller (A) - Base Configuration CC: Controller-Controller Configuration CI: Controller (A) - IOXM Configuration (B)

IOIOI (Console)

Click on the figure for port details and system s/n.

I.Step

II.Step

1

2 Adhere to anti-static precautions. (A paper ESD strap is included inside the RMA box if you don't have your own)3 Remove the replacement Controller Module from the anti-static bag and examine the housing and connector for damage.4 Go to Section III "Node State Check and Shutdown Procedure" on next page.

V-FAS3200 Family: Node Pre-Checks Action DescriptionVerify the "Order Reference 8xxxxxxxxx number on the RMA packing slip is the same as the Part Request (PREQ) number listed in your dispatch notes.

Page 2 of 27

V-FAS3200 Family: Appliance / Controller Module Visual Checks (cont.) Action Description


Fig 6

FAS3200 Controller

Module

Notes: 1. This Action Plan covers FAS and V-Series Controller running ONTAP 7-Mode or Cluster-Mode . 2. This procedure will take 90-150 minutes. 3. Note the Caution on NVMEM LEDs in Section VI. 4. This Action Plan needs to be followed in step order 5. FC port configuration, disk list and the system date are captured prior to removing the original Controller. 6. Many parts need to be moved from the Original Controller to the Replacement Controller Module. 7. System variables; date-time, disk reassignment and FC port configuration must be verified before rebooting the system. 8. If a HA configuration and ONTAP 8, the console may report you "must perform a final ' cf takeover' and 'cf giveback' from the 'partner node", the node that was repaired to complete the 'disk reassign' process. Follow the new steps in 'Disk Reassign' and 'Boot the OS' sections carefully.

III.Step

1 Always capture the nodes console output to a text file, even if using the end-user's computer. To review the Job Aid on how to connect to the console (IOIOI) port and serial emulator options, click here > Console Attach Aid

2

3 Must perform the Visual HA Check in table above, then follow the "action" based on console response in the table below:

4 Go to Section IV "ONTAP Check" on next page.

VISUAL CHECKS: Observe the green "Activity" LEDs, Fig 1, and match up sequence in the following table - Will need to examine the rear of unit to determine the chassis configuration and if it appears to be set up as a HA or MetroCluster system.

Page 3 of 27

V-FAS3200 Family: Node State Check and Shutdown Procedure Action Description

WARNING HA (Active-Active) configurations: If the failure has caused a controller failover you may have been dispatched on the surviving controller's serial number, not the failed one. Perform the checks below.


IV. V-FAS3200 Family: ONTAP CheckStep Action Description

The login name for Cluster-Mode (C-Mode) systems is "admin", not "root".

1

STOP If ONTAP update is required but it cannot be performed now, a system outage must be taken.3

a)

b) Disable the auto-giveback option if enabled from the partner node. (copy-n-paste)

4 Go to Section V, "Capture the Current System Configuration" on next page.

IF HA system and the partner took-over: Bug 509898 on the README page must be taken into consideration. Enter the appropriate command on the partner to display the ONTAP version. If the ONTAP does not meet the minimum version as described in the README, a system outage is required.

IF Cluster-Mode, continue with next step otherwise skip to step 4.Login to the partner and check if the auto-giveback option is enabled by entering the following command: You can copy-n-paste the command syntax.

Page 4 of 27

NOTE

C-Mode: Has two console command shells, clustershell and nodeshell. The default shell is clustershell. You are in the clustershell if the console prompt includes a double colon ( ) prior to the ">" sign, Ex:, netapp01 >To switch from clustershell to nodeshell, enter run local at the ::> prompt, then the double colons (::) are removed. To exit nodeshell, enter exit or Ctrl-D.From clustershell, nodeshell commands can be entered by prefacing the 7-Mode command with run local".Ex: netapp01::> run local sysconfig -v Note, all 7-Mode commands are not supported in C-Mode.

IF enabled C-Mode will show: node auto-giveback -------------- ------------- br3270cl1 true

Cluster-Mode (Run in clustershell) cluster::> sto fa show -node local -fields auto-giveback

Cluster-Mode (Run in clustershell) cluster::> sto fa modify -node local -auto-giveback false


:: ::

7-Mode

partner(takeover)> version Cluster-Mode

cluster::> run local version

VStep NOTE

1 If the console response is: "Waiting for giveback..." follow steps 1a-1c. If console response is LOADER-A|B> skip to step 2.a) At the "Waiting for giveback " prompt, Enter: Ctrl-C b) At the message: "Do you wish to halt this node rather than wait [y/n]? " Enter: yc) After the system drops to the LOADER-A|B> prompt, continue with step 2.

2

3 At the LOADER-A|B> prompt, enter: sp status to capture the Service Processor (SP) configuration. Example below.

4 Enter: printenv This command displays (and captures) all boot environmental variables.

An example of a "printenv" output is here > printenv-C-all.pdf

5 Follow Steps 5A-C if this a V-series Filer (V3200) or has SAN attached Tape Drives < ask customer. If not, go to step 6.A.B.C.

An example of "printenv" output highlighting the variables is here > printenv-C-all.pdf6 Enter the commands below to login to the Service Processor (SP) and capture the Remote Support Agent (rsa) configuration.

NOTE IF the output states the "Remote Support Agent is enabled", it needs be set on the replacement MB in last Section of the AP.7 Enter: exit To exit the Service Processor back to the LOADER prompt.8 Continue with Section V on next page.

The date and time is stored in the system PROM in Greenwich Mean Time, (GMT) also known as Universal Time Clock, (UTC). At the LOADER> prompt, enter: show date Record on paper the system's GMT time and the local time to determine the number of hours (and minutes) the local time is ahead or behind GMT.

Open up the console log.Use the "Find" function to search for variables prefixed with: "fc-no" - Do not rely on a visual search.Identify these variables from the printenv output in the console log: (i) fc-non-array-adapter-list,(ii) fc-no-mgmt-ports OR fc-no-mgmt? NOTE - Only one of these two variables may be used.

Page 5 of 27

V-FAS3200 Family: Capture the Current System Configuration Action DescriptionConfirm the "console" output is being saved to a text file. It will be needed later in this action plan.

LOADER-A> show date Current date & time is: 05/03/2011 12:42:12 LOADER-A>

Step 2): Enter: show date


Waiting for giveback...(Press Ctrl-C to abort wait) ^C This node was previously declared dead. ..... ..... The HA partner is currently operational and in takeover mode. ..... Do you wish to halt this node rather than wait [y/n]? y System halting...

LOADER-A> Step 1b): Enter: y

LOADER-A> sp status Firmware Version: 1.2.3 Ethernet Link: up, 100 Mb, full duplex, auto-neg complete Mgmt MAC Address: 00:A0:98:13:D0:B4 IPv4 Settings Using DHCP: NO IP Address: 10.128.133.231 Netmask: 255.255.255.0 Gateway: 10.128.133.1 IPv6: Disabled

Example Only

LOADER-A> printenv Enter printenv

Step 1: Hitting Enter displays this prompt Step 1a): Enter: CTRL-C

Information on Partner Status

LOADER-A> ^G (Ctrl-G) Service Processor Login: Password: SP node-name> rsa show

Enter: 1) ^G to shell to the SP login prompt 2) Then login using: 7-Mode: naroot, C-Mode: admin 3) Enter the system password 4) At SP-node-name> prompt enter: rsa show

VStep

9 From the LOADER-A|B> prompt enter autoboot to initiate a prom bootstrap. a)

b) Refer to the left and right menus given below. Enter: 5 or maint based on the ONTAP version and mode.c) If asked "Continue with boot?" Answer: y

10 From the *> prompt enter fcadmin config to log the configuration of the integrated FC host adapters. a) Check if "0c" and "0d" Adapter ports are configured as a "target".

11 Continue with Section V on next page.

Page 6 of 27

V-FAS3200 Family: Capture the Current System Configuration (cont.) Action Description

When this message appears: "Press CTRL-C for Boot Menu" , press CTRL-C (^C) to load the "Boot Menu". After about 30-40 seconds, the "Maintenance menu" will appear.

NOTE If the original MB fails to boot to the Maintenance menu due to an error, skip to Section VI.

LOADER-A> autoboot Loading X86_64/freebsd/image1/kernel:0x100000/3375736 0x538280/3221872 ..... Copyright (C) 1992-2010 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* ^CBoot Menu will be available. Please choose one of the following: (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. Selection (1-8)? 5 You have selected the maintenance boot option: ..... ..... In a High Availablity configuration, you MUST ensure that the partner node is (and remains) down, or that takeover is manually disabled on the partner node, because High Availability software is not started or fully enabled in Maintenance mode. FAILURE TO DO SO CAN RESULT IN YOUR FILESYSTEMS BEING DESTROYED NOTE: It is okay to use 'show/status' sub-commands such as 'disk show or aggr status' in Maintenance mode while the partner is up Continue with boot? yes ..... ..... *>


LOADER> autoboot Loading x86_64/freebsd/image2/kernel:....0x100000/3386664 0x53b000/3222096 0x84da50/1190096 ..... NetApp Data ONTAP 8.0.1 Cluster-Mode Copyright (C) 1992-2010 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* ^CBoot Menu will be available. How would you like to continue booting? (normal) Normally (install) Install new software first (password []) Change user password (setup) Run setup first (init) Initialize disks and create flexvol (maint) Boot into maintenance mode (syncflash) Update flash from backup config (reboot) Reboot node Please make a selection: maint ..... ..... In a High Availablity configuration, you MUST ensure that the partner node is (and remains) down, or that takeover is manually disabled on the partner node, because High Availability software is not started or fully enabled in Maintenance mode. FAILURE TO DO SO CAN RESULT IN YOUR FILESYSTEMS BEING DESTROYED NOTE: It is okay to use 'show/status' sub-commands such as 'disk show or aggr status' in Maintenance mode while the partner is up ..... *>

ONTAP 8.0.x C-Mode Only

Step 9: Enter autoboot Step 9: Enter autoboot

Step 9a): Wait for this message, then hit ^C (CTRL-C)

Step 9b): Enter 5

Step 9c): If this node has a partner node this message will be displayed. Answer: y to the "Continue with boot?" question.

maintenance mode console prompt maintenance mode console prompt

Step 9a): Wait for this message, then hit ^C (CTRL-C)

Step 9b): Enter maint

*> fcadmin config Local Adapter Type State Status --------------------------------------------------- 0c initiator CONFIGURED. online 0d target CONFIGURED. offline

Example Only Step 10: Enter fcadmin config

Step 10a): Log all the adapters listed as "target" adapters. In our example, adapters 0d is target.

ONTAP 7-Mode and C-Mode (except 8.0.x C-Mode)

VStep

12 Follow Steps 12A-B if this a V-series Filer (V3200) or has SAN attached Tape Drives < need to ask customer. If not, go to step 13.A. Enter the 'fcadmin channels' command.

See sample output here >> fcadmin channels13 Enter: storage show disk -x to capture disk models. (Examine listing, note if any "SSM", "ASM", "BSM" Disk models exist.)14 Enter: sysconfig -v This output captures System IDs and System Memory size.15 Enter storage show disk -p to capture multipathing information.16 Next, enter disk show -v to view which SAS and FC Adapter ports are driving disks- See Text Box 16 below.

For example of a V-Series 'disk show -v', click here > V-Series Disk Show17

STOP

18 At the *> prompt enter halt (after prom initialization the console will display the LOADER-A|B> prompt)

19 Go to Section VI, "Remove the cables, Cable Management Tray and extract the Controller Module" on next page.

Check to make sure no disks are assigned to a sysID =0, the HOME column shows "000000000". To determine this,enter: disk show -s 0 IF any disks are listed, do NOT proceed - contact NGS on how to reassign these disks.

Take note of all the "unique" Adapter port numbers displayed. See Text Box STEP 17. In this example: SAS Adapters 1b, 0a and FC Adapters 0c, 0d are displayed. Typically many more storage adapters and disks are displayed and switch connections in V-Series.

Page 7 of 27

V-FAS3200 Family: Capture the Current System Configuration (cont.) Action Description

NOTE

The "disk show -v" sample output below is abbreviated console output. In ONTAP 8.x, use the HOME column instead of OWNER to determine which node owns the storage. HOME displays the node's system-name (and system-ID). After the MB is replaced it is necessary to confirm each SAS/FC Adapter port is seeing its storage and to reassign the storage.

B. Note if any of the on-board (0c or 0d) FC Adapters that are configured as "Initiators" display as "N or NL Ports", these adapters are likely connected to the third party storage. Ignore any "N" or "NL" ports that are connected to "target" adapters.

*> disk show -v Local System ID: 1573753606 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 1b.02.4 fas3270cl2-ams(1573753632) Pool0 9QJ7VRRF fas3270cl2-ams(1573753632) 1b.02.3 fas3270cl2-ams(1573753632) Pool0 9QJ7WMNQ fas3270cl2-ams(1573753632) 1b.02.10 fas3270cl2-ams(1573753632) Pool0 9QJ7WF87 fas3270cl2-ams(1573753632) 1b.02.20 fas3270cl2-ams(1573753632) Pool0 9QJ7WMNE fas3270cl2-ams(1573753632) 1b.02.19 fas3270cl2-ams(1573753632) Pool0 9QJ7WWR4 fas3270cl2-ams(1573753632) 1b.02.6 fas3270cl2-ams(1573753632) Pool0 9QJ7W80G fas3270cl2-ams(1573753632) 1b.02.16 fas3270cl2-ams(1573753632) Pool0 9QJ7WSH9 fas3270cl2-ams(1573753632) 1b.02.7 fas3270cl2-ams(1573753632) Pool0 9QJ7W3KA fas3270cl2-ams(1573753632) ..... ..... 0d.41 fas3270cl2-ams(1573753632) Pool0 JLVT29GC fas3270cl2-ams(1573753632) 0d.43 fas3270cl2-ams(1573753632) Pool0 JLVT7BUC fas3270cl2-ams(1573753632) 0d.33 fas3270cl2-ams(1573753632) Pool0 JLVS4EHC fas3270cl2-ams(1573753632) ..... ..... 0c.21 fas3270cl1-ams(1573753606) Pool0 JLVT0KDC fas3270cl1-ams(1573753606) 0c.18 fas3270cl1-ams(1573753606) Pool0 JLVT2HZC fas3270cl1-ams(1573753606) 0c.28 fas3270cl1-ams(1573753606) Pool0 JLVS585C fas3270cl1-ams(1573753606) ..... 0a.01.13 fas3270cl1-ams(1573753606) Pool0 9QJ7W3XZ fas3270cl1-ams(1573753606) 0a.01.21 fas3270cl1-ams(1573753606) Pool0 9QJ7WSX8 fas3270cl1-ams(1573753606) 0a.01.16 fas3270cl1-ams(1573753606) Pool0 9QJ7W3YT fas3270cl1-ams(1573753606) 0a.01.12 fas3270cl1-ams(1573753606) Pool0 9QJ7WS0R fas3270cl1-ams(1573753606) 0a.01.14 fas3270cl1-ams(1573753606) Pool0 9QJ7W8GM fas3270cl1-ams(1573753606) 0a.01.18 fas3270cl1-ams(1573753606) Pool0 9QJ7WX7E fas3270cl1-ams(1573753606) 0a.01.22 fas3270cl1-ams(1573753606) Pool0 9QJ7WY15 fas3270cl1-ams(1573753606) 0a.01.11 fas3270cl1-ams(1573753606) Pool0 9QJ7W8N2 fas3270cl1-ams(1573753606) ..... ..... *> halt

A typical listing will display many more disks than this partial listing. Step 18: Enter halt to exit to the LOADER-A|B> prompt

Step 16: The disk show -v command prints out the System ID of the Local System (1573753606). It also prints the owner of each disk under the HOME heading which lists the node's system name. This system name is (fas3270cl1-ams) and owns disks: 0c.21, 0c.18, 0c.28, 0a.01.13, 0a.01.21, etc.

NOTE- Partner owned disks are intermixed in the output. The partner hostname is 'fas3270cl2-ams' and it's System ID is (1573753632).

Example Only

Step 17: Under the DISK heading, all SAS & FC Adapters are listed. In this example SAS adapter 1b and 0a and FC adapter ' 0d and 0c' are seen, but typically there are more. After the controller is replaced, confirm the same adapters are listed meaning there is an active SAS/FC path to the disks.


VI.Step

NOTE

1

(i)

(ii) If a non-successful takeover, the flashing LED indicates uncommitted customer data - Contact NGS

Non-HA Configuration*: If the green NVMEM Status LED is flashing, the system was not 'halted' properly: (i)

* The node configuration should have been determined by following Section III. 2 Before proceeding further the state of the NVMEM LED should be resolved if it's valid by reading caution above.3 Label each cable connector with its port number and unplug all the cables.

NOTE4 If installed, slide the Cable Management Arm out of C channel located on either sides of Controller module. Ref Fig 7, 8.5 Pull the cam handle downward and slide the controller module out of the system. Ref Fig 9.

6 Go to Section VII, "Move SFPs - Move PCI Cards" on next page.

Confirm from end-user or NGS that the partner controller had a clean takeover, or if this controller was "waiting for giveback", the flashing LED can be ignored.

Ask end-user if controller was properly "halted". If not, re-insert controller and if the system does not autoboot, enter bye at the LOADER-A|B> prompt . If the system boots to the login prompt, login and then enter halt to properly shutdown. Engage NGS if questions.

If possible keep the cables on the cable management arm to keep them in the correct position for reconnection.

Page 8 of 27

V-FAS3200 Family: Remove the cables, Cable Management Tray and extract the Controller Module Action DescriptionIf TWO controllers are installed, do NOT shut off the power supplies to remove one of the controllers. If only ONE controller is installed with or without an IOXM, do shut OFF both Power Supplies.On the node to be serviced, loosen the red thumbscrew, ref Fig 7. Pull down on the cam lever and slide the controller module towards you halfway out of the chassis.

HA (Active-Active) Configuration*: If the green NVMEM Status LED starts flashing ref Page-1, Fig 5, when the controller is extracted from the chassis:


STOP!

and

READ

this

CAUTION

Fig 7 Fig 8

"C" Channel on sides of Controller

Cam Handle Set Screw

Fig 9

Lower Cam Handle and pull to extract the tray

VII.Step

1

2

3

STOP ALL Cards must maintain their original slot #! (SAS bugs: 643008, 647231 and 647235).4 Attach and close the side panels of both the System Trays.

5 Go to Section VIII, "Move the NVMEM Battery, NVMEM & Main Memory DIMMs" on next page.

Remove each SFP/GBIC, Ref Fig 10, one at a time inserted in the original Controller Module's on-board Ethernet and FC ports and install into the same port location in the replacement Module. (Do not mix them up!)Loosen the thumbscrew on the controller module side panel and swing the side panel open until it comes off the controller module. Ref Fig 11a. Open the replacement controller module side panel also.Label each card with it's PCI slot number before removing. Move one card at a time into the new TRAY in same slot. Ref Fig 11b.

Page 9 of 27

V-FAS3200 Family: Move SFPs - Move PCI Cards Action Description


Fig 10

Fig 11a Fig 11b

NOTE: All PCI cards need to be marked with it's slot number before it is removed. Insert into the same slot on the replacement Tray

VIII.Step

1 Lift up at blue arrow to open the CPU air duct. Ref. Fig 12.2

STOP

NOTE Adhere to anti-static precautions. (A paper ESD strap is included inside the RMA box if you don't have your own)3 Disconnect the cable connector to the battery by pressing the tab on the connector. Ref Fig 13 above.4 Continue with Section VIII on next page.

The NVMEM Battery must be disconnected before removing the NVMEM DIMM. Insert the NVMEM DIMM into the replacement controller BEFORE the battery is connected. Otherwise, the system may not boot properly.

Page 10 of 27

V-FAS3200 Family: Move the NVMEM Battery, NVMEM & Main Memory DIMMs Action Description

Use Fig 13 below to locate the Main Mem DIMM, NVMEM DIMM and NVMEM Battery.

NOTE

The two NVMEM sockets have White ejection tabs. The 3210 only populates NVMEM DIMM socket 1. All other models populate both NVMEM sockets.Main Memory sockets have Black ejection tabs. Main Mem sockets 1 and 2 are always populated in all 32xx models, while sockets 3 and 4 are only populated in the 3270.


Fig 13

NVMEM Battery

I/O Panel this Side

Main Memory DIMMs 1-4

(Black locking tabs)

NVMEM DIMMs 1-2 have White locking tabs (Battery Backed )

Press the tab to release the connector to the battery

Boot Media Device (Shown with Cover closed)

Fig 12

CPU Air duct

VIII.Step

5

6

7 Move the NVMEM battery by following step 7(a) and 7(b).a) Removing the NVMEM Battery from the defective controller module. Ref fig 15.

i) Gently push the battery housing up towards the open Air Duct cover.ii)

b) Inserting the NVMEM battery into the replacement controller module. Ref fig 16.i)

ii)

8 Re-connect the battery cable on the replacement controller module and close the CPU duct.

9 Go to Section IX, "Move the Boot Device" on next page.

Firmly squeeze pack into side wall and slide pack towards I/O panel until pack latches in place with an audible click.

Page 11 of 27

V-FAS3200 Family: Move the NVMEM Battery, NVMEM & Main Memory DIMMs (cont.) Action Description

CAUTION: Handle the DIMM by the board edges- Do not touch the gold contacts. Oil from the skin contaminates the connection.Move the NVMEM DIMM(s) and Main Memory DIMMs, one at a time, from the Controller Module and install each one into the same slot in the replacement Controller Module (Do not mix them up!) . See next step for proper insertion technique.Align DIMM with key slot and insert the DIMM straight into the slot. Use both thumbs on the outer edge of DIMM, Fig 14 and press evenly but firmly on the DIMM. It should "snap" in. If not, eject it and re-insert until it snaps in place.

Gently pull the battery housing toward the center of the controller and then lift the battery out of the controller.

Position battery pack by aligning Notches to blue arrows and tilt Key Tabs into notches in the sheetmetal side wall.

Sheetmetal "Notch"

"Key Tab" Fig 16

Fig 14

Figure is not of a V-FAS3200 Series, but demonstrates the proper technique to insert Memory DIMMs

Fig 15

7a(i)

7a(ii)


IX.Step

1 The boot device is installed in a blue box in the center of the MB Ref Fig 13 on page 10.2 Open the boot device cover and gently lift the boot device out of the housing. Ref Fig 17 below.3

4 Close the boot device cover.

X.Step

1 Partially insert the controller into the slot so that the cables can be attached- DO NOT engage the backplane yet.2 Re-attach the Cable Management Arm if removed.

a) For MB: The SAS cable release is on top for ports 0a, 0b, ref Fig 19 below, but on bottom for PCI slots 1-2 (not shown).b)

3

45 Go to Section XI, "Set date and time on the RTC" on next page.

Cables: Fully insert each cable that was removed to its proper port until it clicks in.

Page 12 of 27

V-FAS3200 Family: Move the Boot Device Action Description

Open the boot device cover on the new controller module. Align the boot device with the boot device socket or connector, and then firmly push the boot device into the socket. If necessary, remove the boot device and reseat it into the socket.

V-FAS3200 Series: Partially Reinsert the Controller and reconnect the cables Action Description

NOTE If re-inserting any SAS cables in the IOXM that were removed to extract the MB: The SAS cable release is on the top when connected to SAS cards installed on the left side; slots 3-4. The SAS cable release is on the bottom when connected to SAS cards installed on the right side; slots 5-6. Ref Fig 20.

Cables: Fully insert each cable that was removed to its proper port until it clicks in.


Fig 17

Fig 18 Boot Media FRU

For SAS cards on the left side, slots 3-4, the cable release is on the top.

For SAS cards on the right side, slots 5-6, the cable release is on the bottom.

Fig 20 IOXM

Fig 19

The cable release is on top when inserted into the 3200 MB SAS adapters 0a, 0b as shown,

but on the bottom for PCI cards in MB slots 1-2.

MB SAS Ports: 0a, 0b

XI.Step

12

NOTE

3 Follow steps 3a-3d in the console output below.

4 Continue with Section XI on next page.

Page 13 of 27

V-FAS3200 Series: Set date and time on the RTC Action DescriptionRe-attach laptop to the console port and capture the display output even if using the end user's computer.Fully Insert the Controller Module into the slot and raise the cam lever and secure it with Red thumbscrew. The module starts the power on boot process as soon as it connects to the midplane. It take 75-110 seconds for the first boot message to print on the console screen.


Phoenix TrustedCore(tm) Server Copyright 1985-2006 Phoenix Technologies Ltd. ..... ..... CPU Type: Intel(R) Xeon(R) CPU E5240 @ 3.00GHz Starting AUTOBOOT press Ctrl-C to abort... Autoboot of PRIMARY image aborted by user. LOADER-A>

The boot media contains a newer firmware image (5.1.1). Warning: Automatic firmware update will start in 5 seconds. To abort, press Ctrl-C during this period.

LOADER-A> update_flash New BIOS Version: 5.1.1 New Loader Version: 3.2 Saving Primary Image to Backup flash device Programming ---------------------------- done. 2097152 bytes written Updating Primary Boot Flash Programming .+.+.+.+.+.+.+.+----------.+.+ done. 1310720 bytes written LOADER-A>

Step 3b): IMMEDIATELY press Ctrl-C (^C) key to abort the autoboot. If miss, follow step 4 below.

Step 3c): (If accidently aborted automatic MB FW update) Enter: update_flash to manually update if aborted in step 3a.

< DO NOT abort this!

Step 3a): IF this message is displayed, DO NOT abort the FW update process. If you abort by mistake, you MUST follow steps 3c & 3d below.

LOADER-A> bye ..... ..... LOADER-A>

Step 3d): (IF MB FW was automatically OR manually updated in step 3a or 3c) Enter: bye to complete the boot of the new FW.

Press Ctrl-C (^C) key when prompted to abort the autoboot.

XI.Step

5

a. Immediately Press ^C (CTRL-C) to access the "Boot menu".

(1) For Dual-Controller Config: Power cycle the controller using the SP > here

b. If a 'System ID mismatch' warning message is displayed, answer : y

NOTE

Do NOT bypass the system halt on a NVMEM battery voltage issue. Controller giveback will fail.c. Next, drop to the LOADER prompt from the Boot Menu following the linked process >> here

6

7

NOTE Detailed instructions for another method of obtaining the time in GMT and setting the date and time is here RTC Check8 To set the time issue: set time hh:mm:ss Set the time in GMT using 24 hour format - Do not set the time to local time.

NOTE If this maintenance period spans across the midnight hour in GMT time, the DATE will also need to be set.9 To change the date, issue: set date mm/dd/yyyy (mm = 2-digit month, dd = 2-digit Day, yyyy = 4-digit Year)10 If the date or time was changed, issue: show date again to verify the GMT date and time are correct.11 Go to Section XII, "Service Processor Firmware Check" on next page.

Page 14 of 27

The original motherboard's GMT time and local time should have been recorded in Section V. If you don't have it, you can obtain the GMT time from the partner node, or another NetApp appliance or any Unix Server using: date -u (The "-u" option displays the time in GMT/UTC) The new motherboard's Real Time Clock (RTC) must be set within 2 minutes of the time displayed (which is GMT time) for users to be able to re-connect to this appliance.

At the LOADER-A|B> prompt enter: show date to display the date and time in GMT on the new PCM

IF you miss the window to abort the autoboot, look for this message: "Press CTRL-C for boot menu" and complete steps 5a-5c, otherwise if at the "LOADER" prompt, skip to step 6.

If the replacement MB fails to boot to the Maintenance menu, confirm the memory DIMMS and all PCI cards are properly seated. Also was the original USB Boot Device moved from the original MB to the replacement? Engage NGS for assistance.

NOTE

(i) Some automatic FW updates at this point of time may not allow you to Ctrl-C (^C) and may cause the system to boot to Waiting for giveback...(Press Ctrl-C to abort wait) . If so, try Ctrl-C (^C) to drop to LOADER-A|B first and then autoboot the system and repeat step 3 on previous page.

STOP

If the system reports the NVMEM battery is not detected, re-check the battery cable connection. If the system reports the battery voltage is too low or a critical failure, do NOT proceed - Do NOT bypass the system stop. Engage NGS and ask if this motherboard is being replaced due to a battery issue. If so, a new battery needs to be installed before continuing.

SOLUTION: Disconnect the FC cable on adapters '0c' and/or ' 0d' until after they are configured in "Verify FC Adapter Configuration / Chassis Check" section.

WARNING

V-FAS3200 Family: Set date and time on the RTC (cont.) Action Description

(ii) If you are unable to Ctrl-C (^C) out of the "Waiting for giveback...(Press Ctrl-C to abort wait)" state, follow the below process:

(2) For all other Configs: Power cycle both power supplies - (Power Off, Wait 5 sec and then Power On) While rebooting repeat step 3 on previous page.

If one of the console prints out: "Call home for DISK NOT SPINNING " and dumps core, it will drop to the LOADER prompt. The failure is because one or more of the onboard "initiator" adapters are connected to a SAN switch.


LOADER-A> show date Current date & time is: 10/14/2010 16:36:50 LOADER-A>

Time is displayed in in 24hr mode

Daylight SavingsTime will vary the offset from GMT

.......

....... ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* ^C Boot Menu will be available. Restoring /var from /cfcard/x86/freebsd/varfs.tgz WARNING: System id mismatch. This usually occurs when replacing CF or NVRAM cards! Override system id? {y|n} [n] y

STEP 5b): Enter: y

STEP 5a): Press CTRL-C

WARNING: "Ctrl-C" may not work in case of any automatic FW updates. Read NOTE above.

XII.Step

1 At the LOADER-A|B> prompt, enter: sp status to view the configuration of SP on the replacement controller.SP Firmware must be updated if not version 1.3.1 or higher

If using Procedure B, execute these steps here.2 Go to Section XIII, "Run Diagnostics" on next page.

This method requires the new SP firmware 'tar.gz' binary to be downloaded from NetApp support site and installed on a web-server that this controller's SP can access across the network. The customer is required to perform all the steps to download the image file to the web-server and for the web-server to be able to access the controller's SP interface through the "Wrench" port.

STOPIf the 'Firmware Version' captured with: sp status in Section V is lower than 1.3.1, it will have to be updated. If the SP FW version is 1.3.1 or higher, skip to next section "Run Diagnostics".There are two different procedures to update the SP Firmware. Each uses it's own binary file (tar.gz|.zip) and install process. Any questions with the steps of either procedure, engage NGS.

Action Description

Page 15 of 27

V-FAS3200 Family: Service Processor Firmware Check

Procedure-A: (Recommended)

SP FW update from ONTAP prompt after the controller comes back online (giveback completed if HA). This procedure is the easiest to execute as it does not require a web-server, however the SP update is performed AFTER the node is back up and online.

This method requires the customer to download the SP firmware 'zip' binary from NetApp support site and puts it into the /etc/software folder on the controller and it is installed using the ONTAP "software install" command. The steps for this procedure have been added to the "Boot the Operating System" section.

Procedure-B: SP FW update prior to bringing the controller back online (Requires a SP Setup and a Web Server). This procedure updates the SP FW while the node is "offline" but requires the use of a web-server.


XIII.Step

1 Review the captured Power Up boot log messages and confirm correct amount of memory is reported by model : SA320, 3270, 3250=~20GB, 3220=~12GB, 3240=~8GB, 3210=~5GB. Sample output here

2 Enter: boot_diags to run diagnostics. (It goes into maintenance mode.) Read NOTE below.a. If a 'System ID mismatch' warning message is displayed, answer : y

(1) For Dual-Controller Config: Power cycle the controller using the SP > here

(i)

Do NOT bypass the system halt on a NVMEM battery voltage issue. Controller giveback will fail.(ii)

NOTE

3 Enter: yes to the question "Continue with boot?".

4 Continue with Section XIII on next page.

If the console prints out: "Call home for DISK NOT SPINNING " and dumps core, it will drop to the LOADER prompt. The failure is because one or more of the onboard "initiator" adapters are connected to a SAN switch. SOLUTION: Disconnect the FC cable on adapters '0c' and/or ' 0d' until after they are validated in "Verify FC Adapter Configuration / Chassis Check" section.

Page 16 of 27

V-FAS3200 Series: Run Diagnostics (5-20 minutes based on model) Action Description

If the reported memory size is not correct, remove the controller if CC, otherwise power off and reseat the memory and recheck. If still incorrect, report to NGS.

STOP

If the system reports the NVMEM battery is not detected, re-check the battery cable connection. If the system reports the battery voltage is too low or a critical failure, do NOT proceed - Do NOT bypass the system stop. Call NGS.

NOTE

Some FW updates may cause the system to boot to Waiting for giveback...(Press Ctrl-C to abort wait) and you are unable to Ctrl-C(^C) out of it. To drop the system to LOADER-A|B again follow the below process:

If the replacement MB fails to boot to the Maintenance menu, confirm the memory DIMMS and all PCI cards are properly seated. Also was the original USB Boot Device moved from the original MB to the replacement?

(2) For all other Configs: Power cycle both power supplies - (Power Off, Wait 5 sec and then Power On) While rebooting press Ctrl-C(^C) to abort the autoboot and skip to LOADER-A|B. Then repeat step 2 above.


LOADER-A> boot_diags Loading X86_64/freebsd/image1/kernel:0x100000/3386728 0x53b000/3222096 ........................ ........................ ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* ................................. ................................. ................................ In a High Availability configuration, you MUST ensure that the partner node is (and remains) down, or that takeover is manually disabled on the partner node, because High Availability software is not started or fully enabled in Maintenance mode. FAILURE TO DO SO CAN RESULT IN YOUR FILESYSTEMS BEING DESTROYED NOTE: It is okay to use 'show/status' sub-commands such as 'disk show or aggr status'in Maintenance mode while the partner is up Continue with boot? yes ......................... ......................... *>

STEP 2: Boot the Diags

STEP 3: Enter: yes

STEP 2a: If this message displays, enter y to continue.

Restoring /var from /cfcard/x86/freebsd/varfs.tgz WARNING: System id mismatch. This usually occurs when replacing CF or NVRAM cards! Override system id? {y|n} [n] y

..... Updating NVMEM firmware .... The system will automatically reboot when the update is complete ......

NOTE An automatic 'NVRAM FW update' may reboot the system to LOADER, enter: boot_diags again. Read the NOTE under step 2(a) above. If any errors reported, Contact NGS

XIII. V-FAS3200: Run Diagnostics (cont. )Step Action Description

5

STOP

a) Enter: sldiag version to list the Diag Version. The diag and ONTAP version will determine if we run the "mem" test.

b) Enter the following command:

c) Enter: sldiag device show to confirm the list of disabled devices.

d) Continue with Section XIII on next page.

Bug 648013

Specific versions of ONTAP and sldiags may cause the fcache card(s) to become inoperable if diags are run on them, requiring replacement. Disable the fcache device as detailed in text box Step 5 below.

Page 17 of 27

Disable the devices by copying and pasting the disable commands or you can also use the UP arrow to repeat the last line and just modify the device name. See text box STEP 5 below.These devices must be disabled to avoid system panic and for the diagnostic tests to complete without errors - sas, fcal, nic, acpm, cna and fcache.

STOP If the diag version, printed in step 5a above, is "A.00.02" AND the ONTAP version is less than 7.3.7 follow step (b) to also disable the "mem" test. If ONTAP version is 8.x, skip to step (c).

*> sldiag device modify -selection disable -dev sas *> sldiag device modify -selection disable -dev fcal *> sldiag device modify -selection disable -dev nic *> sldiag device modify -selection disable -dev acpm *> sldiag device modify -selection disable -dev cna *> sldiag device modify -selection disable -dev fcache ERROR: device name not found. *>

*> sldiag device modify -selection disable -dev mem

STEP 5: These devices need to be disabled: Copy and paste the top line and use the UP arrow key to repeat the last line and revise each device name to be disabled.

If the device does not exist, this error will display.

*> sldiag device show DEV NAME SELECTION ---------------- ---------------- ---------------- nvmem nvmem enabled interconnect c0b disabled interconnect c0a disabled nic e0M disabled .... acp acpview enabled acpm 0a.2.A disabled acpm 0a.2.B disabled .... fcal 0c disabled .... sas 1a disabled sas 1b disabled mem mem enabled storage 0a disabled bootmedia bootmedia enabled environmentals environmentals enabled serviceProcessor serviceProcessor enabled .... *>

Confirm these devices are disabled if present: sas, fcal, nic, acpm, cna, fcache

*> sldiag version System Level Diagnostic Version A.00.02

STEP 5(a): Enter: sldiag version


XIII. V-FAS3200: Run Diagnostics (cont. )Step Action Description

e) Enter: sldiag device run to run diagnostics on all enabled components.

f)

g)

6 Go to Section XIV, "Verify FC Adapter Configuration / Chassis Check" on next page.

Enter: sldiag device status and confirm that the status shows "PASSED" or "Completed" without errors for all tests. If not all PASSED call NGS and report failure and read next NOTE.

NOTE: If you re-run sldiags, the log file needs to be cleared first. Enter: sldiag device clearstatus

Page 18 of 27

NOTE: SLDIAG runs its tests in the background by default.Wait until the "" message is displayed. (approx 14-20 minutes)

NOTE "sldiag device status" can be entered to view test status. Testing is not complete until the message in step 5(f) above is displayed.

*> sldiag device run *> *> sldiag device status To see details use: sldiag device status -long Device TestName StartDate EndDate Status LOOP ---------- ---------------- ------------------- ------------------- ---------- -------- nvmem NVMEM details Mon Feb 28 10:13:31 Mon Feb 28 10:13:31 PASSED 1/1 acpview ACP show Mon Feb 28 10:13:31 Mon Feb 28 10:13:31 Completed 1/1 nvmem Read test Mon Feb 28 10:13:31 Mon Feb 28 10:13:34 PASSED 1/1 mem Random Address Mem Test Mon Feb 28 10:13:35 Mon Feb 28 10:19:13 PASSED 1/1 bootmedia Media Info Mon Feb 28 10:13:36 Mon Feb 28 10:13:36 PASSED 1/1 environmentals Sensor Test Mon Feb 28 10:13:36 Mon Feb 28 10:13:41 PASSED 1/1 serviceProcessor Service Processor Mon Feb 28 10:13:36 Mon Feb 28 10:13:36 PASSED 1/1 mem Random Data Mem Test Mon Feb 28 10:19:13 Mon Feb 28 10:24:09 PASSED 1/1 *>

NOTE: That all the tests show "PASSED/Completed"

STEP 5(e): Enter: sldiag device run to run the selected tests

WAIT until this message is displayed

STEP 5(g): Enter: sldiag device status to see the test result


XIV.Step

1

2

NOTE

3

4 Enter fcadmin config to confirm the changed FC Adapters are displaying as PENDING: (target) ports.

5 If any FC cables were disconnected from adapters '0c' and/or '0d' due to boot issue, firmly reconnect them now. Must click in.

depot, follow the steps in the linked doc to ensure the "ha" FRU PROM parameters are correctly set > > here.6 Follow Steps 6A-D if this a V-series Filer (V3200) or has SAN attached Tape Drives < ask customer. If not, go to step 7.

A.B.

See sample output here >> fcadmin channelsC.

WWPNs changing before and after a MB swap is >> here.D.

7 Go to Section XV, "Capture new System-ID on replacement Controller" on next page.

Wait until the end-user administrator verifies the SAN Fabric zoning is changed if necessary, and he has updated the host-group on the array with the new WWPNs and that the array can see the NetApp FC WWPNs before continuing.

Page 19 of 27

V-FAS3200 Family: Verify FC Adapter Configuration / Chassis Check Action DescriptionReview the fcadmin config output captured from the original controller. If Adapter Ports "0c" or "0d" were configured as a "target", proceed with next steps - If both were configured as "initiators", skip to step 5.

STOP For HA Systems: IF the physical chassis was replaced and/or the replacement controller was not in a sealed bag from the

At the *> prompt, enter: fcadmin config to view the FC Adpater configuration. If adapters 0c, 0d match the original configuration, skip to step 5. If one or more needs to be set, continue. If the adapter that needs to be changed to a target, is listed as " online", it must be off-lined first before it can be changed. Issue: fcadmin offline For each Adapter to be configured as a target enter: fcadmin config -t target Issue one command per adapter. This example configures Adapter port 0d as target:

Enter the command: 'fcadmin channels' to list new WWPNs of on-board ports, 0c-0d.Provide the output of fcadmin channels to the end-user administrator to remap the array's host-group (a.k.a. LUN Masking) and if WPWN zoning is used, to update the fabric zones for disk and tape - NOTE - This is only required if FC Adapters 0c or 0d are configured as "Initiators" and display as "N or NL. No changes are required to the array or the fabric if 0c or 0d adapters are configured as "target" adapters or not shown as "N" or "NL" ports.

If the Array or SAN Tape is using one or more of the on-board FC Adapters, the end-user may see the new WWPNs through the SAN switch and already making the changes. An example of "fcadmin channels" output highlighting the


*> fcadmin config Local Adapter Type State Status --------------------------------------------------- 0c initiator CONFIGURED. online 0d initiator CONFIGURED. offline *> fcadmin config -t target 0d Tue Oct 28 07:19:05 GMT [fci.config.state:info]: Fibre channel initiator adapter 0d is in the PENDING (target) state. A reboot is required for the new adapter configuration to take effect. *> fcadmin config Local Adapter Type State Status --------------------------------------------------- 0c initiator CONFIGURED. online 0d initiator PENDING (target) offline

STEP 2: Enter: fcadmin config

STEP 3: Enter: fcadmin config -t target for each port to be configured as a target

STEP 4: Enter: fcadmin config to confirm each target port is shown as PENDING

Example Only

XV.Step

1a)

b)

2

a)

3

4 Go to Section XVI, "Disk Reassign" on next page.

Scan the "original" storage show disk -x captured in Sec V for model strings that end in "SSM", "ASM" or "BSM" in the MODEL column. The disk model could start with X302, X306, X308 or X310. Ex: X306_HJUPI02TSSM. IF any of these disks were listed, some of them may disappear if any SAS cards were replaced. If any are missing, READ the IMPORTANT note below.

Page 20 of 27

V-FAS3200 Family: Capture new System-ID on replacement Controller Action Description

Enter: storage show disk -p to confirm all adapters list a PRIMARY and SECONDARY path. No? Re-check cable seating.

Enter: disk show -v This shows disk ownership by system-ID. Confirm all disks are listed as originally captured. NOTE: The primary and secondary path, to the SAS and FC Adapter under the Disk heading, can reverse.

Compare the new system ID to the old system ID. The old system-ID is on the 2nd line of the sysconfig -v output that was captured in Section IV.

IMPORTANT! IF "SSM", "ASM or "BSM" disks are missing from the storage show disk -x, do NOT proceed, contact NGS and reference Disk Bug 643008 and IOM Bugs 647231, 647235 listed in the README FIRST page.

Enter: storage show disk -x to force some disk I/O.In this listing verify the same number of disks are displayed as compared to the output captured in Sec V. IF disks are missing: Confirm all FC and SAS cables are fully seated and re-run: storage show disk -x. If unsure of the disk count, follow step 1b, otherwise skip to step 2.


*> disk show -v Local System ID: 1943753293 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ----------- ------------- 1b.02.4 fas3270cl2-ams(1573753632) Pool0 9QJ7VRRF fas3270cl2-ams(1573753632) 1b.02.3 fas3270cl2-ams(1573753632) Pool0 9QJ7WMNQ fas3270cl2-ams(1573753632) ..... 0d.41 fas3270cl2-ams(1573753632) Pool0 JLVT29GC fas3270cl2-ams(1573753632) 0d.43 fas3270cl2-ams(1573753632) Pool0 JLVT7BUC fas3270cl2-ams(1573753632) ..... 0c.21 fas3270cl1-ams(1573753606) Pool0 JLVT0KDC fas3270cl1-ams(1573753606) 0c.18 fas3270cl1-ams(1573753606) Pool0 JLVT2HZC fas3270cl1-ams(1573753606) ..... 0a.01.13 fas3270cl1-ams(1573753606) Pool0 9QJ7W3XZ fas3270cl1-ams(1573753606) 0a.01.21 fas3270cl1-ams(1573753606) Pool0 9QJ7WSX8 fas3270cl1-ams(1573753606) 0a.01.16 fas3270cl1-ams(1573753606) Pool0 9QJ7W3YT fas3270cl1-ams(1573753606) 0a.01.12 fas3270cl1-ams(1573753606) Pool0 9QJ7WS0R fas3270cl1-ams(1573753606)

In this example, the local System ID for the new Controller is 1943753293. The old MB System ID was 1573753606 (disk show -v from Section V). The disks need to be reassigned to the local System ID. Example Only

XVI.Step

STOP

1 Follow the correct Procedure below based on your system configuration and status.

IF ONTAP version is 8.2.x or higher , follow the linked process here IF ONTAP version is 7.3.x, 8.0.x or 8.1.x, follow Procedure-A below.

Procedure-AONTAP < 8.0.3 will give an error message if more than 500 disks are attempted to be reassigned from the partner.If system is exposed, a system outage is required to do the disk reassign - Read this link for options >here

A1 Login to the PARTNER node (7-Mode=root , C-Mode=admin ). Engage end-user for password. A2 C-Mode only: Enter: run local to enter the nodeshell.A3

A4

A5 Enter: priv set advanced at the prompt for the following command to work. Prompt will include " * ".A6

NOTE The disk reassignment process takes several seconds and a message is printed for each disk that is reassigned.

second takeover/giveback from the "target" (repaired) node must be executed later in this AP. Ref TSB-1110-04

to cancel the 'disk reassignment' and follow the steps >> here.If re-dispatched for the disk reassign, start at the beginning of this Procedure- A.

A7A8

A9 Continue with step 2 on next page.

ONTAP 8.2.x introduced a new automatic 'disk reassign' process for "HA" systems provided the partner took over the target node. Follow step 1 CAREFULLY!

Page 21 of 27

V-FAS3200 Family: Disk Reassign Action Description

Engage the customer and ask: Are there any Windows applications running that would inhibit a cf giveback at this time? (open cifs sessions)" If the customer states the giveback cannot be performed now, answer n

Partner has taken over target controller running ONTAP 7.3.x, 8.0.x, 8.1.x;

Bug537799

Partner "takeover" Verification Check: Check the console prompt as follows: Case1: IF the prompt has the word (takeover) in it (Example: nodeB (takeover)> ) , continue with step A4. Case2: IF the prompt has a "/" in it (Example: nodeA / nodeB> ) , enter: partner and then press enter key twice to exit the partner shell. IF the prompt has "(takeover)" in it - continue with step A4, otherwise NGS. Case3: IF either case1 or case2 does not match your console prompt, verify with cf status. IF no "takeover", follow Procedure-B on next page. Questions or if partial takeover, NGS.

If the giveback can be preformed now, enter y and continue. The next console message confirms the disk ownership update to the new system-ID. Enter y to the question.

STOP If the console messages stated the giveback must be completed immediately, do not enter any other commands on the partner node until "after" the disk ownership on the down node is verified and the giveback is completed.

Enter: partner aggr status -f Confirm no disks are listed as they will have to be replaced before the disk reassignment and giveback. IF failed disk(s), inform customer and call NGS.

Enter: disk reassign -s -d Cut and paste the old and new System IDs from the console Log.

STOP

IF the following highlighted console message is displayed: 1) The giveback cannot be postponed and 2) A

Follow Procedure-B on next page.

Target Node Status

(i) "HA" and the node was successfully taken over by its partner

(ii) The partner did NOT takeover ORthis node is a Single controller configuration

Procedure to be followed

Disk ownership will be updated on all disks previously belonging to Filer with sysid 1573753606. Would you like to continue (y/n)? y Enter: y

Command Example Only: partner-system name(takeover)*> disk reassign -s 1573753606 -d 1943753293

A console message will be displayed for each disk changing ownership (System ID)


disk reassign: A giveback must be done immediately following a reassign of partner disks. After the partner node becomes operational, do a takeover and giveback of this node to complete the disk reassign process. Do you want to continue (y/n)?

Execute all of these steps on the PARTNER node.

XVI.Step Action Description

Procedure-BB1

B2 IF Single Controller configuration, follow steps (i)-(ii). IF the partner did NOT takeover, skip to step B3.(i)

(ii)

B3 IF the partner did NOT takeover, follow steps (i)-(iii).(i)

(ii)(iii)

2a)

b)

3 Continue with Section XVI on next page.

Page 22 of 27

V-FAS3200 Family: Disk Reassign (cont.)

At the maintenance mode " * > " prompt enter: disk reassign -s -d Cut and paste the old and new System IDs from the console Log.

Use for Single Controller configuration or if HA, the partner did NOT takeover.

From the console port on "target" controller on which you replaced the MB (in maintenance mode): Enter: disk show -s No disks or V-Series LUNs should be listed as shown in console window below. (The "-s old-sysID" was specified in the disk reassign step1)IF any disks/V-LUNs are listed, a reservation may not have released, continue with step 2(b). IF no output, skip to step 3.

Not all disks re-assigned: Re-issue the disk reassign, from the node it was entered on, to see if the reservation releases. Then repeat the disk show -s command in step 2(a). IF disks/V-LUNs are still listed in the output, call Support.

Enter: y to question "Would you like to continue (y/n)?" Skip to step 2.

Enter: n to question "Abort reassignment (y/n)?" Enter: y to question "Would you like to continue (y/n)?" Continue with step 2.


*> disk show -s 1573753606 Local System ID: 1943753293 DISK OWNER POOL SERIAL NUMBER HOME ---------- ------------- ----- ------------- ------------- *>

Example Only

IF all disks properly reassigned, there should be no disks/V-LUNs listed in the output.

Execute all of these steps from Maintenance mode on the repaired node.

Command Example Only: *> disk reassign -s 1573753606 -d 1943753293

Command Example Only: Disk ownership will be updated on all disks previously belonging to Filer with sysid 1573753606. Would you like to continue (y/n)? y Enter: y

A console message will be displayed for each disk changing ownership (System ID)

Command Example Only: Partner node must not be in Takeover mode during disk reassignment from maintenance mode. Serious problems could result!! Do not proceed with reassignment if the partner is in takeover mode. Abort reassignment (y/n)? n Proceeding with disk reassignment. Disk ownership will be updated on all disks previously belonging to Filer with sysid 1573753606. Would you like to continue (y/n)? y

Enter: n

Enter: y A console message will be displayed for each disk changing ownership (System ID)

XVI.Step Action Description

4

5 Skip to step 6 if this is not a V-Series (V3200). If V-Series perform the below Additional Steps.

6 At the maintenance mode prompt: " * >", enter halt to exit to LOADER-A|B.7 Go to Section XVII, "Boot PROM Variable Checks" on next page.

Page 23 of 27

V-FAS3200 Family: Disk Reassign (cont.)

Enter: disk show -v Verify the "Local System ID" matches node's disks listed in the "HOME" column. READ STOP below.

STOP!BEFORE the "giveback" can be executed, you must verify the system-id for this node's disks listed under "HOME" if the column exists, use OWNER if not, and the "Local System ID" match. If not, confirm the correct system-ids were entered on the 'disk reassign' command. If problems, do NOT proceed, call NGS for assistance.


*> disk show -v Local System ID: 1943753293 DISK OWNER POOL SERIAL NUMBER HOME ------------ ------------- ----- ------------- ------------- 1b.02.4 fas3270cl2-ams(1573753632) Pool0 9QJ7VRRF fas3270cl2-ams(1573753632) 1b.02.3 fas3270cl2-ams(1573753632) Pool0 9QJ7WMNQ fas3270cl2-ams(1573753632) ..... 0d.41 fas3270cl2-ams(1573753632) Pool0 JLVT29GC fas3270cl2-ams(1573753632) 0d.43 fas3270cl2-ams(1573753632) Pool0 JLVT7BUC fas3270cl2-ams(1573753632) ..... 0c.21 -------------------------- Pool0 JLVT0KDC (1943753293) 0c.18 -------------------------- Pool0 JLVT2HZC (1943753293) ..... 0a.01.13 -------------------------- Pool0 9QJ7W3XZ (1943753293) 0a.01.21 -------------------------- Pool0 9QJ7WSX8 (1943753293) 0a.01.16 -------------------------- Pool0 9QJ7W3YT (1943753293) 0a.01.12 -------------------------- Pool0 9QJ7WS0R (1943753293) ..... .....

Example Only

The new local System ID for the Controller is 1943753293. The owner name (fas3270cl1-ams) may or may not be shown. But those disks should reflect the new local System ID.

The "disk show -v" command displays the connectivity to the third party array and any (optional) NetApp disks. If the storage listing is missing or incomplete: (i) Confirm all FC connections are in the correct HA port and firmly seated on the NetApp controller. (ii) If the connections are correct and firmly seated, confirm the WWPNs were correctly entered by end-user by reviewing the output of the "fcadmin channels" command BEFORE and AFTER the MB swap - Engage NGS for assistance.

S T O P

XVII. V-FAS3200 Family: Boot PROM Variable ChecksStep Action Description

1

2

STOP

Table-2

Step

A)

B)

C)

D) bootarg.storageencryption.support true

kmip.init.interface eXy kmip.init.ipaddr xxx.xxx.xxx.xxx

key_manager setup later in this AP) kmip.init.netmask xxx.xxx.xxx.xxx Does this system have NSE Disks? Click > here kmip.init.gateway xxx.xxx.xxx.xxx

3 If any of the variables are missing or values are not set correctly, set them now one by one by following steps a-d below.a) Open the console log file that was previously captured from the old MB and scroll to the "printenv" output listing. b) Use the "Find" function and search for the variable required and its value. Sample "printenv" output herec)

d) Enter: savenv to save the variable and value.

PROM will accept improper variable names. Recheck they were entered properly by issuing printenv again.4 Go to Section XVIII, "Boot the Operating System - 'giveback' if applicable" on next page.

Page 24 of 27

IF ONTAP version is < 8.0.2 (ONTAP 8.0.2, 8.1 and > are not affected), unset the variable bootarg.init.wipeclean. (copy-n-paste) Bug 462894.

Validate the boot env variables:At the LOADER-A|B> prompt, enter: printenv to list "all" the boot PROM variables. Search the output to confirm the required variables are properly set based on your system type and configuration . Follow steps A-D in Table-2 below.

IF V-Series: fc-non-array-adapter-list is only required if the V-Series is also hosting NetApp multipath shelves.

fc-non-array-adapter-list

If you accidently issued a "set-defaults", all the configuration variables are unset - You have to examine the printenv capture of the original MB to determine which ones will need to be manually entered in step 3 for your System Type & Configuration.

System Type & Configuration Variable name and Value to be validated

IF HA: Verify this variable is set with correct partner sys-id value.

partner-sysid xxxxxxxxxx IF C-Mode: Verify this variable is set to true.

bootarg.init.boot_clustered true

STOP

IF NSE disk based system: Verify these 5 variables and values are set correctly. (More variables will be set and verified by running

Once identified, copy them using "Ctrl-C", then at the LOADER prompt enter: setenv

One of these two variables "may" be set if the storage switches are McData.

fc-no-mgmt-ports OR (only one of these two may be set) fc-no-mgmt? true

LOADER-A> setenv variable-name value LOADER-A> savenv

LOADER-A> unsetenv bootarg.init.wipeclean


XVIII.Step Action Description

1 At the LOADER-A|B prompt, enter: autoboot to boot ONTAP.2

a) If the system booted up to a "login>" prompt, example below, continue with step 2b, otherwise skip to step 3.

b) Is this system HA Configured?

No, the system is stand-alone (non-HA head): Skip to Step 14 on page 27.3

4 Login into the PARTNER node (7-Mode=root , C-Mode=admin ). Engage end-user for password. 5

6 IF Pre-ONTAP 8.2 and 7-Mode: Ask the customer if there are any heavy NDMP, SnapMirror or SnapVault processes running. If Yes, they should be disabled due to bug 489060. The procedure to disable the processes is here.

7 Enter the proper controller giveback command(s) based on the mode running as follows:

NOTE

8

a) IF Cluster-Mode: Confirm the "giveback" status of the storage, refer this doc > ONTAP 8 failover show

NOTE

9 Continue with Section XVIII on next page.

Yes, but the partner did not takeover: Go to the STOP under Step 10 on next page.

V-FAS3200 Series: Boot the Operating System - 'giveback' if applicable

A giveback can fail due to: "a failed disk" - remove failed disks or for "Open CIFS sessions". If due to open CIFS sessions, check with customer how to close out CIFS sessions. Terminating CIFS can cause loss of data if DBs are running under CIFS.

Wait! 90 seconds for 7-Mode or 3 minutes for Cluster-Mode after giveback reported complete. Check controller failover status by entering the appropriate command shown for the specified ONTAP Mode.

If the "giveback" is incomplete, wait 2 minutes and re-check. If still not complete after 10 minutes, contact Support. Do not proceed to next step if 'incomplete or partial giveback'!

If the system booted up to a "Waiting for giveback>" prompt (press the key) , example below, the node was part of an HA configuration and was taken over by its partner.

Check takeover status by entering the appropriate command shown for the specified ONTAP Mode. If "partner not ready" may have to wait 2-4 minutes for the NVRAMs to synchronize.

Page 25 of 27

After the console stops printing messages, press the key.

Loading X86_64/freebsd/image1/kernel:0x100000/3375736 0x538280/3221872 ..... ..... ******************************* * Press Ctrl-C for Boot Menu. * ******************************* ..... login:

These are typical Boot strap console

messages. If the partner did not takeover OR this is a stand-alone

system, you should eventually get a "login" prompt when you

hit .

.... Many typical system startup messages removed for clarity

Phoenix TrustedCore(tm) Server ..... ..... ******************************* * Press Ctrl-C for Boot Menu. * ******************************* ..... ..... Waiting for giveback...(Press Ctrl-C to abort wait)

NOTE 3: If you see this message, this node is part of a HA configuration and the partner node took over for it. If the "controller giveback" fails due to partner "not ready", wait 5 minutes for the NVMEMs to synchronize. If the giveback fails due to "open CIFS sessions", failed disks or for any other reason, contact NGS.

"...." = Deleted lines to save space

7-Mode partner(takeover)> cf giveback

Cluster-Mode cluster::> storage failover giveback -fromnode local

7-Mode partner(takeover)> cf status

Cluster-Mode cluster::> run local cf status

Cluster-Mode cluster::> storage failover show

7-Mode partner> cf status Controller Failover enabled, XYZ is up.

Follow step (a) Look for failover enabled


XVIII.Step

10

STOP

a)

11 From the "repaired node", execute a takeover using the proper command below to sync the sys-IDs.

a)

b)

c)

(i)

Click > ONTAP 8 failover show to see examples of output. Issues? Call NGS.12 Continue with Section XVIII on next page.

Wait Again! This time 90 seconds for 7-Mode or 3 miniutes for Cluster-Mode and then check giveback status by entering the appropriate command below for the proper ONTAP mode. For 7-Mode look for failiover enabled , for Cluster-Mode follow step (i).

For ONTAP Cluster Mode, storage failover show should not show any "partial" givebacks. If there are, wait another 60 seconds and recheck. Some large systems may take up to 10 minutes to complete.

After the appropriate Wait period in step 11a) and the cf status reports: "Ready for giveback" , enter the proper "giveback" command below. This is the final synchronization of the system-Ids across the HA pair.

Page 26 of 27

Action DescriptionIF Motherboard was Replaced and the partner printed out the following highlighted message after the "disk reassign" command was executed, go to Step 11. If no message reported, skip to step 12. (Ref TSB-1110-04).

IF this system has a partner controller, but the partner did not takeover, (Disks were assigned in maintenance mode) continue with step 10a (Ref Internal TSB-1209-02). Note, the console message above is not displayed when disks are reassigned in maintenance mode.

Login to the "repaired node" (target) and re-enable "controller failover" using proper command syntax below (copy-and-paste).

V-FAS3200 Series: Boot the Operating System - 'giveback' if applicable (cont.)

Wait! 60 seconds for 7-Mode or 90 seconds for Cluster-Mode after takeover reports complete- Then check takeover status by entering the appropriate command shown for the specified ONTAP Mode.

7-Mode target> cf takeover

Cluster-Mode (run from clustershell )

cluster::> storage failover takeover -bynode local

7-Mode

target> cf enable

Cluster-Mode (run from clustershell )

(1st cmd is for 2-node clusters ONLY, 2nd cmd is for 3 or more node clusters)

cluster::> cluster ha modify -configured true OR cluster::> storage failover modify -node local -enabled true

disk reassign: A giveback must be done immediately following a reassign of partner disks. After the partner node becomes operational, do a takeover and giveback of this node to complete the disk reassign process.

7-Mode

target(takeover)> cf status Cluster-Mode

cluster::> run local cf status

Cluster-Mode cluster::> storage failover show

7-Mode target> cf status Controller Failover enabled, XYZ is up. Follow step (i).

7-Mode

target(takeover)> cf giveback Cluster-Mode

cluster::> storage failover giveback -fromnode local


XVIII.Step

13 IF Cluster-Mode: Follow steps (a-b) below, otherwise skip to Step 14.a)

Example of output here> net int show

b)

Example of output here> net int show14

Follow Procedure A steps > here.

XIX. V-FAS3200 Series: NetApp Storage Encryption (NSE) System? Step Action DescriptionSTOP

1 Enter: key_manager setup on the target (repaired) node to update the boot PROM variables and to regenerate the key for the new system-ID. Follow these additional steps here.

a) Next, login to the partner node and enter: key_manager setup to update it's boot parameters.XX.

Step1 IF NDMP, SnapMirror or SnapVault options were disabled, enable them now. Refer to page 2 of doc > > here2 C-Mode Only: Re-enable "auto-giveback" options if they were disabled on either node. C-Mode command here3 Ask customer if using Operations Manager? If so, can they still access the controllers?

4

NOTE5 Ask end-user if using "AutoSupport"? If YES, perform step 5(a). If NO, perform step 5(b).

a)

b) If ASUP is disabled: Call NGS CSR and provide the new MB serial number so they can register it as the new system s/n.6 Email the console log with the NetApp Reference Number in the Subject Line to [email protected] Place the defective part in the antistatic bag and seal the box.8 Follow the return shipping instructions on the box to ship the part(s) back to NetApps RMA processing center. If the

shipping label is missing see process to obtain a shipping label here > Missing Shipping Label?9 Verify with customer that the system is OK and if working with NGS ask them if it is OK to be released.

10 Close dispatch per Rules of Engagement.

Procedure A: SP FW update from ONTAP prompt after the controller comes back online (giveback completed if HA)This method requires downloading the SP firmware 'zip' binary from the NetApp support site and saving it in the "/etc/software" folder on the controller. It is installed using the ONTAP "software install" command.

IF this system is using NSE disks continue with step 1, otherwise skip to Section XX.

IF a SP Firmware update is required and the update was delayed until the node is back on line, upgrade to the latest recommended version, currently 1.3.1 and follow "Procedure A" below. If FW=1.3.1 or > skip to Sec XIX.

Bug 583160

SNMP v3 access may fail despite the user existing with the correct permissions after a takeover or giveback. WORKAROUND: Change the password or re-add the user. Engage Customer to step through the Work-around.

Page 27 of 27

V-FAS3200 Series: Boot the Operating System - 'giveback' if applicable (cont.) Action Description

From the clustershell on each node, enter the command below to list the logical interfaces that are not on their home server and port.

If any interfaces are listed as "false" in the above command, enter the command below to revert them back to their home port. Issues? Call NGS.

Service entitlements break when the MB is swapped because the new motherboard changes the system serial number.

V-FAS3200 Series: Complete system setup, Registration, Submit logs, Part return.

IF the Remote Support Agent (rsa) configuration captured in Section IV step 6 stated: "Remote Support Agent is enabled.", a) Enter: ^G (Ctrl-G) and login to the Service Processor. b) Using the captured info from the "rsa show", run: rsa setup The customer will have to provide the HTTP password. (Hint: administrator or root password? If unknown, can be setup later)

ASUP Enabled System: Request end-user to send NetApp an ASUP Message from the target node so the configuration setup can be verified and the new system serial number can be registered by NGS. If the target system is not UP, send ASUP from its partner. Use the corresponding command for the mode of ONTAP running. Enter your dispatch's 7-digit FSO number (begins with 5).

Action Description


Cluster-Mode cluster::> net int show -is-home false

Cluster-Mode cluster::> net int revert *

7-Mode filer> options autosupport.doit 5xxxxxx

Cluster-Mode cluster::> invoke * -type all -message 5xxxxxx

README1234 (2)567-new8910111213141516-new17N18N192021-8.2-new222324252627

fas32xx service v2

Documents