upgrading and downgrading firmware •firmwaremanagement,onpage1...

Upgrading and Downgrading Firmware

• Firmware Management, on page 1• Upgrading and Downgrading Considerations, on page 3• Upgrading the Fabric, on page 4

Firmware ManagementACME Inc., in partnership with Cisco, has evaluated the requirements for their deployment based on thesoftware features required, the support for the hardware platforms they have selected, and the maturity of thesoftware releases. They have selected a target version of software for their deployment. Additionally, theyhave put a proactive plan in place to revisit this decision periodically to determine if future upgrades arerequired.

Firmware VersionsThe software versions for Cisco Application Centric Infrastructure (ACI) are listed in the following format:

major.minor(maintenance)

• major—Represents major changes in the product architecture, platform, or features content.

• minor—Represents a minor release with new software features.

• maintenance—Represents bug fixes to a feature release of Application Policy Infrastructure Controller(APIC). This changes when there are fixes for product defects in the software, but no additional newfeatures.

The following example shows some APIC versions:1.0(1e)1.1(1j)1.2(1i)

Both the software for the APIC and the fabric nodes are denoted by the same version scheme. For example,the APIC 1.2(1i) release corresponds to the switch software 11.2(1i) release. The release notes for the APICversions reference the corresponding switch versions, and vice versa.

All components of the ACI infrastructure including the APIC, leaf switches, and spine switches, should beon the same version. While at the time of upgrading, disparate versions may exist between APIC and theswitches, do not operate the fabric for extended periods of time in this state.

Upgrading and Downgrading Firmware1

When considering the impact and risk of upgrading, you can assume that a maintenance version upgrade,such as upgrading from 1.1(1j) to 1.1(1o), will have less impact than a major/minor version upgrade, as therewill be only bug fixes and no new features added.

Firmware ComponentsThere are three main components that can be upgraded:

• Switches (leaf and spine)• Application Policy Infrastructure Controller (APIC)• Catalog firmware

Firmware Policies

Firmware Groups

Firmware group policies on the Application Policy Infrastructure Controller (APIC) define the group of nodeson which firmware will be upgraded. For most deployments, a single firmware group is adequate.

Maintenance Groups

Maintenance group policies define a group of switches that will be jointly upgraded to the associated firmwareset. Maintenance groups can be upgraded on demand or according to a schedule, making it possible to deferan upgrade task to a business maintenance window. Typically, there are two maintenance groups, eachcontaining a set of leafs and spines. Each maintenance group is upgraded separately.

Controller Firmware

The APIC firmware policy applies to all controllers in the cluster, but the upgrade is always done sequentially.The APIC GUI provides real-time status information about firmware upgrades. Controller firmware policiescan be upgraded on demand or according to a schedule.

Catalog Firmware

Each firmware image includes a compatibility catalog that identifies supported switch models. The APICmaintains a catalog of the firmware images, switch types, and models that are allowed to use that firmwareimage. The APIC, which performs image management, has an image repository for compatibility catalogs,APIC firmware images, and switch images.


Upgrading and Downgrading FirmwareFirmware Components

Figure 1: Firmware Upgrade Policy Relationships

Upgrading and Downgrading ConsiderationsBefore starting the upgrade or downgrade process, verify the following things:

• Application Policy Infrastructure Controller (APIC) cluster health—Before starting the upgrade process,your controllers should be in good health. Verify that the health state of all of the controllers in the clusterare Fully Fit before you proceed. To resolve issues for controllers that are not fully fit see theTroubleshooting Cisco Application Centric Infrastructure document.

• Configuration backup—Before starting any upgrade, always export your configuration to an externalsource. For information about exporting configurations, see the "Import and Export Policies."

• Permissions—A user must have the fabric administrator role to perform firmware upgrade tasks.

• Verify free space—Confirm that the /firmware partition is not filled beyond 75%. If the partition isfilled beyond 75%, you might be required to remove some unused firmware files from the repository toaccommodate the compressed image as well as provide adequate space to extract the image. The APICautomatically extracts the image.

• Upgrade order—Typically, the controllers should be upgraded first, followed by the switch nodes. Alwaysrefer to the relevant release notes of the destination firmware version for any changes to this order.

• Maintenance windows—Although it is possible to upgrade the fabric without impacting the dataplane,you should perform an upgrade during a scheduledmaintenance window according to your change controlpolicy. This window should account for any unforeseen issues that might arise during the upgrade, andallocate enough time to troubleshoot or perform a rollback.

• Maintenance groups—To help minimize the impact to hosts during an upgrade, you should set up at leasttwo separate maintenance groups. A common separation is by odd and even node IDs. Assuming thatyour hosts are dual-connected to at least one odd and one even leaf node, there should not be any impactto your hosts. Maintenance group creation is covered in detail later in the chapter. Another considerationis that your leaf vPC pairs should contain one odd and one even node.


Upgrading and Downgrading FirmwareUpgrading and Downgrading Considerations

• Upgrading a fabric with the Application Virtual Switch (AVS) deployed—The AVS software is notspecifically tied to the APIC or switch software version.

• Device packages—Device packages are not always tied to the APIC software. You can confirm thedevice compatibility for Layer 4 to Layer 7 devices using the online Cisco Application CentricInfrastructure (ACI) Compatibility tool.

Upgrading the Fabric

Downloading the Firmware Images Using the GUIYou must download both the controller software package and switch software package for the ApplicationPolicy Infrastructure Controller (APIC) from Cisco.com.

Procedure

Step 1 On the menu bar, choose Admin > Firmware.Step 2 In the Navigation Pane, choose Fabric Node Firmware.

In the Work pane, the list of all switches in the fabric and the status of when the firmware was last upgradedare displayed.

Step 3 In the Navigation Pane, choose Download Tasks.Step 4 In the Work pane, choose Actions > Create Firmware Download Task.Step 5 In the Create Firmware Download Task dialog box, perform the following actions:

a) In the Source Name field, enter a name for the switch image, such as "apic_1.2.1i".b) For the Protocol radio buttons, click the Secure copy or HTTP radio button.c) In the URL field, enter the URL from where the image must be downloaded.

• HTTP Example: http://192.168.0.50/aci-apic-dk9.1.2.1i.iso• SCP Example: 192.168.0.50:/tmp/aci-firmware/aci-apic-dk9.1.2.1i.iso

• For SCP, enter your username and password.

d) Click Submit.

Step 6 (Optional) You can instead upload the image from your local machine by performing the following actions:a) In the Navigation pane, choose Download Tasks.b) Right click and choose Upload Firmware to APIC.c) Browse to the image that is saved on your local machine.d) Click Submit.

Step 7 In the Navigation Pane, choose Download Tasks.Step 8 In the Work pane, choose the Operational tab to view the download status of the images.Step 9 Repeat this procedure for the switch image.Step 10 After the download reaches 100%, in the Navigation pane, choose Firmware Repository.


Upgrading and Downgrading FirmwareUpgrading the Fabric

Step 11 In the Work pane, choose the Images tab to view the downloaded version numbers and image sizes.

Downloading the Firmware Images Using the Object Model CLIYou must download both the controller software package and switch software package for the ApplicationPolicy Infrastructure Controller (APIC) from Cisco.com.

Procedure

Step 1 SSH to an APIC in the fabric.# ssh admin@node_name

Step 2 Switch to the object model CLI:apic1# bashadmin@apic1:~>

Step 3 Place the image into the image repository:admin@apic1:~> firmware add ver_no.iso

Step 4 Verify that the software has been added to the repository:admin@apic1:~> firmware listName : aci-apic-dk9.1.2.1i.binType : controllerVersion : 1.2(1i)

Upgrading an APIC Using the GUIThe catalog firmware image is upgraded when an Application Policy Infrastructure Controller (APIC) imageis upgraded. You do not need to upgrade the catalog firmware image separately.

To upgrade an APIC:

1. On the menu bar, choose Admin > Firmware.2. In the Navigation pane, click Controller Firmware.3. In the Work pane, choose Actions > Upgrade Controller Firmware Policy.4. In the Upgrade Controller Firmware Policy dialog box, perform the following actions:

1. In the Target Firmware Version field, from the drop-down list, choose the image version to whichyou want to upgrade.

2. In the Apply Policy field, click the Apply now radio button. Alternately, you can apply a schedulepolicy if you wish to defer the task to a specific date/time.

3. Click Submit to complete the task.

The Status dialog box displays the "Changes Saved Successfully" message, and the upgrade processbegins. The APICs are upgraded serially so that the APIC cluster is available during the upgrade.

5. Verify the status of the upgrade in the Work pane.


Upgrading and Downgrading FirmwareDownloading the Firmware Images Using the Object Model CLI

Each APIC takes about 10 minutes to upgrade. Once an APIC image is upgraded, it drops from the clusterand reboots with the newer version while the other APICs in the cluster are still operational. Once theAPIC reboots, it joins the cluster again. Then, the cluster converges and the next APIC image starts toupgrade. If the cluster does not immediately converge, and is not fully fit, the upgrade will wait until thecluster converges and is Fully Fit. During this period, a "Waiting for Cluster Convergence" message isdisplayed in the Status column for each APIC as it upgrades.

When the APIC that the browser is connected to is upgraded and it reboots, the browser displays an errormessage.

During the upgrade process, while the APIC reboots with the newer image, you will not be able to use theGUI of that specific APIC. If you are logged into the APIC GUI during the upgrade process, you may receivea browser error message and may be logged off. Once the status of that specific APIC if Fully Fit, you canlog in to that APIC again.

Note

Upgrading an APIC Using the NX-OS-Style CLIYou can upgrade an Application Policy Infrastructure Controller (APIC) using the NX-OS-style CLI. Beforeyou upgrade the switches, the APICs must have completed upgrading and have a health state of Fully Fit. Inthe NX-OS-style CLI, you must first set the catalog firmware. The following procedure sets the catalogfirmware and starts the upgrade.

Procedure


Step 2 Enter the configure mode:apic1# configureapic1(config)#

Step 3 Enter the firmware mode:apic1(config)# firmwareapic1(config-firmware)#

The firmware mode allows you to set the catalog version.

Step 4 Set the catalog version:apic1(config-firmware)# catalog-version aci-catalog-dk9.1.2.0.225.bin

Now you are ready to update the controller firmware.

Step 5 Enter the controller-group mode and verify the current version:apic1(config-firmware)# controller-groupapic1(config-firmware-controller)# show versionRole Id Name Version---------- ---------- ------------------------ --------------------controller 1 apic1 1.2(0.139g)


Upgrading and Downgrading FirmwareUpgrading an APIC Using the NX-OS-Style CLI

controller 2 apic2 1.2(0.139g)controller 3 apic3 1.2(0.139g)

Step 6 Set the controller firmware to the version that you want:apic1(config-firmware-controller)# firmware-version aci-apic-dk9.1.2.0.225.bin

Step 7 Start the upgrade.

You can specify a time for the upgrade to start, or you can start the upgrade immediately.

• To specify the time for the upgrade to start, enter:apic1(config-firmware-controller)# time start 23:30

You must always specify a time; specifying the date is optional.

• To start the upgrade immediately, enter:apic1(config-firmware-controller)# exitapic1(config-firmware)# exitapic1(config)# exitapic1# firmware upgrade controller-group

Upgrading an APIC Using the Object Model CLIThe catalog firmware image is upgraded when an Application Policy Infrastructure Controller (APIC) imageis upgraded. You do not need to upgrade the catalog firmware image separately. Cisco recommends that youperform the firmware upgrade from the GUI.When you use the GUI, the APIC performs additional verificationand integrity checks on the software image.

To upgrade an APIC using the object model CLI:

1. List the current software in the repository that was previously downloaded.

Example:admin@apic1:~> firmware listName : aci-apic-dk9.1.1.1j.binType : controllerVersion : 1.1(1j)

2. Upgrade the firmware on the APICs.

Example:admin@apic1:~> firmware upgrade controllers ver_no .bin

The APICs are upgraded serially so that the APIC cluster is available during the upgrade. The upgradeoccurs in the background.

3. Check the status of the upgrade.

Example:admin@apic1:~> firmware upgrade statusNode-Id Role Current- Target- Upgrade- Progress-Percent

Firmware Firmware Status (if inprogress)--------- ----------- ------------ ------------------ ---------- ------------------1 controller 1.1(1.200j) apic-1.2(1.202i) complete 0


Upgrading and Downgrading FirmwareUpgrading an APIC Using the Object Model CLI

2 controller 1.1(1.200j) apic-1.2(1.202i) inprogress 03 controller 1.1(1.200j) apic-1.2(1.202i) inqueue 0

The Upgrade-Status field will show "inqueue", "inprogress", or "completeok". If you see "unknown" in thisfield, the APIC has upgraded and is rebooting. During this time, you may lose connectivity to the APIC CLIand have to relog in to the CLI.

Upgrading a Switch Using the GUIBefore you upgrade the switches, the Application Policy Infrastructure Controllers (APICs) must havecompleted upgrading and have a health state of Fully Fit.

To upgrade a switch using the GUI:

1. On the menu bar, choose Admin > Firmware.2. In the Navigation pane, choose Fabric Node Firmware.

In the Work pane, the switches that are operating in the fabric are displayed.

3. If you have not created a firmware group, perform the following substeps:

1. In the Navigation pane, choose Fabric Node Firmware > Firmware Groups.2. In the Work pane, choose the Policy tab.3. Choose Actions > Create Firmware Group.4. In the Create Firmware Group dialog box, perform the following actions:

1. In the Group Name field, enter the name of the firmware group.2. In the Target Firmware Version drop-down list, choose the firmware version to which you will

upgrade.3. In the Group Node IDs field, enter a comma-separated list or a range of node IDs to include in

the group. For example, "101, 103-105, 108".4. Click Submit.

5. To verify that the firmware group was created, in the Navigation pane, choose Fabric Node Firmware> Firmware Groups > new_firmware_group. The Work pane displays details about the firmwarepolicy that was created earlier.

4. If you have not created maintenance groups, perform the following substeps:

1. In the Navigation pane, choose Fabric Node Firmware >Maintenance Groups.

Cisco recommends that you create twomaintenance groups for all of the switches. For example, createone group with the even-numbered nodes and the other group with the odd-numbered nodes. Ensureat least one spine and one leaf are in a different maintenance group than others so as not to lose totalconnectivity.

2. In the Work pane, choose Action > Create POD Maintenance Group.3. In the Create POD Maintenance Group dialog box, perform the following actions:

1. In theGroup Name field, enter the name of the maintenance group. For example, "Even-Nodes".2. For the Run Mode drop-down list, choose Pause Upon Upgrade Failure. This is the default

mode.3. In the Group Node IDs field, enter a comma-separated list or a range of node IDs to include in

the group. For example, "102, 104, 106, 108, 110".


Upgrading and Downgrading FirmwareUpgrading a Switch Using the GUI

4. In the Scheduler drop-down list, you can choose to create a schedule for upgrading or leave thedrop-down list blank so that you can upgrade on demand.

5. Click Submit.6. Repeat this step for the second maintenance group. For example, a group named "Odd-Nodes".

4. Verify that the maintenance group was created.

1. In the Navigation pane, choose Fabric Node Firmware >Maintenance Groups >new_maintenance_group

2. Choose the name of the maintenance group that you created.

3. In the Work pane, verify that the nodes are attached to that maintenance group.

5. Right-click one of the maintenance groups that you created and choose Upgrade Now.6. In the Upgrade Now dialog box, for Do you want to upgrade the maintenance group policy now?,

click Yes.

Note: In the Work pane, the Status displays that all the switches in the group are being upgradedsimultaneously. The default concurrency in a group is set at 20. Therefore, up to 20 switches at a timewill get upgraded, and then the next set of 20 switches are upgraded. In case of any failures, the schedulerpauses and manual intervention is required by the APIC administrator. The switch upgrade takes up to12 minutes for each group. The switches will reboot when they upgrade, connectivity drops, and thecontrollers in the cluster will not communicate for some time with the switches in the group. Once theswitches rejoin the cluster after rebooting, you will see all the switches listed under the controller node.If there are any VPC configurations in the cluster, the upgrade process will upgrade only one switch at atime out of the two switches in a vPC domain.

7. In the Navigation pane, click Fabric Node Firmware.

Note: In the Work pane, view all of the switches that are listed. In the Current Firmware column, viewthe upgrade image details listed against each switch. Verify that the switches in the fabric are upgradedto the new image.

Upgrading a Switch Using the NX-OS-Style CLIYou can upgrade a switch using the NX-OS-style CLI. Before you upgrade the switches, the APICs musthave completed upgrading and have a health state of Fully Fit. The following procedure upgrades a switch.

Procedure


Step 2 Add images to the firmware repository:apic1# firmware repository add aci-n9000-dk9.11.2.0.225.bin

Step 3 Enter the configure mode:apic1# configureapic1(config)#


Upgrading and Downgrading FirmwareUpgrading a Switch Using the NX-OS-Style CLI

Step 4 Enter the firmware mode:apic1(config)# firmwareapic1(config-firmware)#

Step 5 Check the firmware version:apic1(config-firmware)# show versionRole Id Name Version---------- ---------- ------------------------ --------------------leaf 101 176-Leaf-1 n9000-11.2(0.65l)leaf 102 176-Leaf-2 n9000-11.2(0.65l)spine 201 176-Spine-1 n9000-11.2(0.65l)spine 202 176-Spine-2 n9000-11.2(0.65l)

Step 6 Enter the firmware-switch mode by creating a switch-group:apic1(config-firmware)# switch-group EvenNodesapic1(config-firmware-switch)#

Step 7 Add switches to the switch-group:apic1(config-firmware-switch)# switch 102, 202

Step 8 (Optional) Verify that the switches were added:apic1(config-firmware-switch)# show run# Command: show running-config firmware switch-group all-nodes# Time: Fri Nov 6 15:18:34 2015firmwareswitch-group EvenNodesswitch 102switch 202

Step 9 Set the switch firmware to the version that you want:apic1(config-firmware-controller)# firmware-version aci-apic-dk9.1.2.0.225.bin

Step 10 Set the switch run-mode to pause-on-failure so that the upgrade will pause in the event of any failures:apic1(config-firmware-switch)# run-mode pause-on-failure

Step 11 Start the upgrade.

You can use a scheduler specify a time for the upgrade to start, or you can start the upgrade immediately.

• To use a scheduler, enter:apic3(config-firmware-switch)# schedule upgradetimerEvenNodes

• To start the upgrade immediately, go back to execsh mode and enter:apic1# firmware upgrade switch-group

Upgrading a Switch Using the Object Model CLIBefore you upgrade the switches, the Application Policy Infrastructure Controllers (APICs) must havecompleted upgrading and have a health state of Fully Fit.

To upgrade a switch using the object model CLI:


Upgrading and Downgrading FirmwareUpgrading a Switch Using the Object Model CLI

1. Check that the output of the following command appears like the output shown below, with the correctversion number:

Example:admin@apic1:~> firmware listName : aci-n9000-dk9.11.2.1i.binType : switchVersion : 11.2(1i)

The name changes from ".iso" to ".bin".

2. Upgrade the switches.

Example:admin@apic1:~> firmware upgrade switch node 101 ver_no.binFirmware Installation on Switch Scheduled

You must upgrade each switch separately.

3. Check the upgrade status for the switch. The output that appears from the following command will appearlike the following sample:

Example:admin@apic1:~> firmware upgrade status node node_idNode-Id Role Current- Target- Upgrade- Progress-Percent

Firmware Firmware Status (if inprogress)--------- ----------- ------------------- ------------------ ---------- ------------------1017 leaf n9000-11.1(1.869S1) n9000-11.2(1i) completeok 100

You can check the status of all nodes at once, by entering the firmware upgrade status command.

4. Repeat Steps 2 and 3 for each additional switch.

Verifying Cluster Convergence Using the GUIYou can monitor the progress of the cluster convergence after a scheduled maintenance. You can view theprogress on theController Firmware screen of the GUI, which presents you with a series of messages duringthe process of converging. These messages are displayed in the Status field.

As the controller and switches move through the upgrade, you will see messages about the number of nodesqueued and the number in the process of upgrading, as well as how many have upgraded successfully.

The following are the possible upgrade states for a node:

• NotScheduled: No upgrade is currently scheduled for this node.• Scheduled: Upgrade is scheduled for this node.• Queued: There is a currently active window (schedule) and the node is requesting permission to upgrade.• Inprogress: Upgrade is currently in progress on this node.• CompleteOK: Upgrade completed successfully.• CompleteNOK: Upgrade failed on this node.• Inretryqueue: Node is queued again for upgrade retry (5 attempts are made before declaring failure).

This may take a while. When all the clusters have converged successfully, you will see "No" in theWaitingfor Cluster Convergence field of the Controller Firmware screen.


Upgrading and Downgrading FirmwareVerifying Cluster Convergence Using the GUI

Troubleshooting Failures During the Upgrade ProcessThere is one scheduler per maintenance policy. By default, when an upgrade failure is detected, the schedulerpauses, and no more nodes in that group begin to upgrade. The scheduler expects manual intervention todebug any upgrade failures. Once manual intervention is complete, you must resume the paused scheduler.

If you notice that switches are in the "queued" state, then check the following:

• Is the controller cluster healthy? The controller cluster must be healthy. If you see"waitingForClusterHealth = yes" in the API or "Waiting for Cluster Convergence" showing "Yes" in theGUI, that means the controller cluster is not healthy. Until the controller cluster is healthy, switcheswhich have not already started their upgrade will be in "queued" state.

• Is the switch maintenance group paused? The group will be paused if any switch fails its upgrade.

If the system takes longer than about 60 minutes for a switch to display "waitingForClusterHealth = no" inthe API or "Waiting for Cluster Convergence" showing "No" in the GUI, you should work through the stepsfor verifying a pause in the scheduler.

For additional troubleshooting procedures, see Troubleshooting Cisco Application Centric Infrastructure.


Upgrading and Downgrading FirmwareTroubleshooting Failures During the Upgrade Process

upgrading and downgrading firmware •firmwaremanagement,onpage1...

Documents