1712 dell emc cloud for microsoft azure stack 1712 dell emc cloud for microsoft azure stack patch...
TRANSCRIPT
1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide
Version A01 Dell Engineering January 2018
2 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A00
Revisions
Date Version Description
Jan 2018 A00 Initial release
Jan 2018 A01 HLH updates, Meltdown, and Spectre
THIS GUIDE IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES.
THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND BY DELL EMC or MICROSOFT
Copyright © 2018 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All
other marks and names mentioned herein may be trademarks of their respective companies.
3 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Table of contents Revisions............................................................................................................................................................................. 2
Patch and Update Overview ............................................................................................................................................... 5
Goal . ................................................................................................................................................................ 5
About Speculative Execution Side-Channel Vulnerabilities ......................................................................................... 5
Installing Patch and Updates ........................................................................................................................................ 6
Phase 1: Installing Microsoft updates ................................................................................................................................. 7
1a. Updating the Hardware Lifecycle Management Server [HLH] ............................................................................... 7
Installing the Windows Server 2016 Cumulative Update on the HLH Host ................................................................. 7
Enable the Mitigations by Setting Windows Server 2016 Registry Values .................................................................. 9
Updating the OME and OMNM VMs ............................................................................................................................ 9
1b. Running the Microsoft Patch and Updates on Azure Stack Scale Nodes. .......................................................... 13
Build reference ........................................................................................................................................................... 13
Prerequisites ............................................................................................................................................................... 13
Procedure ................................................................................................................................................................... 14
Phase 2: Running Dell EMC firmware Patch and Update framework .............................................................................. 22
Downloading contents ................................................................................................................................................ 22
Preparing the stamp for updates ................................................................................................................................ 22
2a. Applying the Dell EMC firmware Patch and Update on the Hardware Lifecycle Management Host [HLH] ......... 25
2b. Applying the Dell EMC firmware Patch and Update on Azure Stack Scale Nodes ............................................. 31
Draining the node – Maintenance mode .................................................................................................................... 31
Invoking the Dell EMC Patch and Update Script ........................................................................................................ 33
How to review updates ...................................................................................................................................................... 41
Monitor updates in Azure Stack using the privileged endpoint .................................................................................. 47
Verify the cmdlets are available ................................................................................................................................. 47
Use the update management cmdlets ....................................................................................................................... 48
Connect to the privileged endpoint and assign session variable ............................................................................... 49
Get high-level status of the current update run .......................................................................................................... 49
Get the full update run status with details .................................................................................................................. 49
Get the verbose progress log ..................................................................................................................................... 50
Actively view the verbose logging .............................................................................................................................. 50
Resume a failed update operation ............................................................................................................................. 51
Troubleshooting .......................................................................................................................................................... 51
4 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
New features and fixes ............................................................................................................................................... 51
Windows Server 2016 new features and fixes ........................................................................................................... 51
Known issues with the update process ............................................................................................................................. 52
Microsoft Known Issues ............................................................................................................................................. 52
Known issues (post-installation) ................................................................................................................................. 52
DELL EMC Known Issues .......................................................................................................................................... 55
Appendix A: Updating Security Policies on the OME VM ................................................................................................. 56
5 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Patch and Update Overview
Goal .
Azure Stack operators are faced with the enormous challenge of keeping their solution both secure and
functional. They must ensure the solution is not vulnerable to threats–external or internal–while maintaining
negotiated service-level agreements.
About Speculative Execution Side-Channel Vulnerabilities
Dell EMC is aware of the side-channel analysis vulnerabilities (also known as Meltdown and Spectre)
affecting many modern microprocessors that were publicly described a team of security researchers on
January 3, 2018. This document addresses the specific steps for securing the servers within the Dell EMC
Cloud for Microsoft Azure Stack from these specific vulnerabilities.
In general, there are three steps that must be taken to implement full mitigations against these
attacks. These can be summarized as follows:
1. Patch the Operating System on the server (install Windows Server 2016 KB4056890)
2. Enable specific mitigations within the Operating System (apply registry modifications)
3. Update the Intel processor microcode on the server (flash the server BIOS).
In the context of the Azure stack solution:
The Installing the Windows Server 2016 Cumulative Update on the HLH Host, Enable the Mitigations
by Setting Windows Server 2016 Registry Values, and Updating the OME and OMNM VMs
subsections within this document explain how to apply these same updates to the physical HLH host,
as well as the OpenManage Essentials / Support Assist Enterprise and OpenManage Network
Manager Virtual Machines that reside on that host.
The Microsoft Azure Stack 1712 Update (Build 20180106.1) addresses steps #1 and #2 for the scale
unit hosts and the infrastructure VMs that comprise the Azure Stack solution. Following the
procedures in the 1b. Running the Microsoft Patch and Updates on Azure Stack Scale Nodes section
of this document will apply the necessary OS updates and registry configuration settings for the
physical scale unit hosts and the infrastructure VMs.
As of January 22, 2018, Intel has communicated new guidance regarding "reboot issues and
unpredictable system behavior" with the microcode included in the BIOS updates released to address
Spectre (Variant 2), CVE-2017-5715. Dell is advising that all customers should not deploy the BIOS
update for the Spectre (Variant 2) vulnerability at this time. We have removed the impacted BIOS
updates from our support pages and are working with Intel on a new BIOS update that will include
new microcode from Intel. For the latest information and recommended BIOS versions, please refer
to: http://www.dell.com/support/article/us/en/04/sln308588/microprocessor-side-channel-
vulnerabilities-cve-2017-5715-cve-2017-5753-cve-2017-5754-impact-on-dell-emc-products-dell-
enterprise-servers-storage-and-networking-?lang=en
6 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
When these BIOS updates are released, Dell EMC will issue an updated Dell EMC Tools package for
Microsoft Azure Stack. This update will contain the BIOS update packages that can be applied using
the steps in 2a. Applying the Dell EMC firmware Patch and Update on the Hardware Lifecycle
Management Host [HLH] (which explains the procedure for the physical HLH host, including
suspending BitLocker prior to performing the update) and 2b. Applying the Dell EMC firmware Patch
and Update on Azure Stack Scale Nodes (which explains how to apply these updates to the physical
scale unit nodes).
Installing Patch and Updates
Installing Patch and Updates includes firmware updates for the hardware, and software updates for the
operating system and drivers. The Patch and Update process is a two-phase process:
1. Running Microsoft software Patch and Update framework
a. Hardware Lifecycle Management Server [HLH]
b. Azure Stack Scale Nodes
2. Running Dell EMC firmware Patch and Update framework
a. Hardware Lifecycle Management Server [HLH]
b. Azure Stack Scale Nodes
IMPORTANT: Normally, firmware patches and updates need to be installed first before running software
patches and updates. For 1712, due to side-channel analysis vulnerabilities (also known as Meltdown
and Spectre), the Microsoft Patch and Update process will need to be run first, and Dell EMC Patch and
Update process second.
7 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Phase 1: Installing Microsoft updates Section 1a below provides the procedure for updating the hardware lifecycle manaement server. Section 1b
provides the details for running the Microsoft P&U framework against the scale nodes. These operations can
be performed in either order.
1a. Updating the Hardware Lifecycle Management Server [HLH]
Installing the Windows Server 2016 Cumulative Update on the HLH Host Complete the following steps to update Windows Server 2016 on the HLH host:
Step Activity
1 Log in to your HLH server.
2 Browse to C:\DELLEMCTools\ folder on your OME-VM where you extracted the DellEMC toolkit.
Go to the HLH_Cumulative_Windows_Update folder for the Windows Server 2016 x86_64
Cumulative Update for January 2018 (KB 4056890) and copy it onto the HLH host.
3 Run the update package.
Click “Yes” to allow the update to execute.
8 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
4 The update will progress through multiple stages and take several minutes to complete.
Note: This may take 25 minutes or more to complete.
5 When the update package has finished running, click “Restart Now” to reboot the computer and finish applying the updates.
9 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
6 Log in to Windows and open an elevated (Administrator) PowerShell.
Issue the following command and verify that KB4056890 has been installed:
Get-HotFix
Enable the Mitigations by Setting Windows Server 2016 Registry Values Complete the following steps to set registry values that enable the mitigations:
Step Activity
1 Open an elevated (Administrator) PowerShell. Issue the following three commands to set the registry values (each command should be entered as one continuous line, despite the wrapping in this document):
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f
reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f
For more information, refer to https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution
2 Execute the Restart-Computer cmdlet to reboot the HLH.
Updating the OME and OMNM VMs To further ensure that guest-to-host and guest-to-guest memory access is protected against potential exploits,
the OS update and registry settings should also be applied to the virtual machines that run on the HLH host.
10 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Install the Cumulative Update on the OME and OMNM Virtual Machines Complete the following steps to update Windows Server 2016 on the OME and OMNM virtual machines:
Step Activity
1 From the HLH console, connect to the OME VM via RDP (or from Hyper-V Manager). Log on as the local Administrator.
2 Either place a copy of the Windows Server 2016 x86_64 Cumulative Update for January 2018 (KB 4056890) locally on the VM, or connect to a file share location that contains the update package.
3 Run the update package.
Click “Yes” to allow the update to execute.
11 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
4 The update will progress through multiple stages and take several minutes to complete.
Note: This may take 25 minutes or more to complete.
5 When the update package has finished running, click “Restart Now” to reboot the computer and finish applying the updates.
Note: The reboot may take several minutes due to portions of the update that run before Windows Server 2016 shuts down.
12 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
6 Log in to the VM (as in step 1) and open an elevated (Administrator) PowerShell. Issue the following command and verify that KB4056890 has been installed:
Get-HotFix
7 Repeat these same steps to apply the update on the OMNM virtual machine.
13 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Enable Mitigations on the OME and OMNM Virtual Machines Complete the following steps to set registry values that enable the mitigations:
Step Activity
1 From the HLH console, connect to the OME VM via RDP (or from Hyper-V Manager). Log on as the local Administrator.
2 Open an elevated (Administrator) PowerShell. Issue the following three commands to set the registry values (each command should be entered as one continuous line, despite the wrapping in this document):
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f
reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f
For more information, refer to https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution
3 Execute the Restart-Computer cmdlet to reboot the virtual machine.
4 Repeat these same steps to apply the update on the OMNM virtual machine.
1b. Running the Microsoft Patch and Updates on Azure Stack Scale Nodes.
IMPORTANT: This update package is only applicable for Azure Stack integrated systems. Do not apply
this update package to the Azure Stack Development Kit.
Build reference The Azure Stack 1712 update build number is 180106.1. If a customer has deployed 180103.2 previously, you
do not need to apply 180106.1.
Prerequisites You must first install the Azure Stack 1710 Update and 1711 Update before applying this update.
14 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Procedure The following procedure shows how to import and install updates as an Azure Stack operator.
Step Activity
1 Download the update package from Microsoft from the The Azure Stack 1712 update build number 180106.1 website. Scroll down the page to the section “Download the update” and download the package. An update package will typically consist of a single self-extracting executable (.exe), corresponding bin files (.bin) and a single metadata (.xml) file.
The <package>.exe file contains the payload for the update, for example the latest cumulative update for
Windows Server.
The corresponding <package>.bin file(s) provide compression for the payload as associated with the
executable.
The metadata.xml file contains essential information about the update, for example the publisher, name,
prerequisite, size and support path URL.
15 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
2 To import the update package to Azure Stack, in the administrator portal, under Data + Storage, click Storage Accounts.
3 In the filter box, type update, and select the updateadminaccount.
16 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
4 In the updateadminaccount storage account details, under Services, select Blobs.
5 On the Blob service tile, click + Container to create a new container, give it a name (for example, update-1709), and then click OK.
17 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
6 After the container is created, click Upload to upload the <package>.exe, any associated .bin files, and the metadata.xml files into the container.
18 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
7 Browse to the <package>.exe file, and then click Open in the file explorer window.
8 Next, click Upload in the administrator portal.
19 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
9 Do the same for the <package>.bin and metadata.xml files.
10 When done, you can review the Notifications. A notification should indicate that upload has completed.
20 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
11 Navigate back to the Update tile to review the newly-added update package.
12 To install an update, select the package marked as Ready and either right-click and select Update now, or click Update now in the command bar at the top of the window.
22 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Phase 2: Running Dell EMC firmware Patch and Update framework
Note: When you run Dell EMC firmware Patch and Update and it informs that no updates available, that
means you are at the latest level of firmware level.
Downloading contents
Download the DellEMC Tools compressed file (Cloud for Microsoft Azure Stack 13G Toolkit
1.0.1712.2.zip) from the Dell EMC Support Downloads Webpage onto your OME-VM. Right click the zip file to
extract its contents into a folder, for example C:\DELLEMCTools. This toolkit has, among various DELLEMC
tools, the framework and firmware installation files required for running Dell EMC post-deployment / FRU
scenario firmware Patch and Update process.
Preparing the stamp for updates
Step Activity
1 Browse to the folder C:\DELLEMCTools where the extracted contents are located. In this folder, the contents required for the post-deployment Dell EMC Firmware Patch and Update are:
The Post Deploy Firmware Patch and Update Azure Stack folder
The Firmware folder.
23 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
2 Right-click the Firmware folder (this folder holds all the firmware executables that iDRAC will use to update the firmware). Now select the Sharing tab. Then click Advanced Sharing… and share this folder. In our example we called the share “Firmware”, which is the default. You can call it a different name.
IMPORTANT: Please take note of this share name as we will use it later when we invoke the Firmware P&U script. Click OK. After share creation, we move to draining the node into maintenance mode.
3 Check if the SMB v1 is enabled on the share.
On the “firmware” share that was created in the prior step, where the DUPs executables are located, perform the following:
a. Check to see if SMBv1 is enabled in the server configuration by running the following command: Get-SmbServerConfiguration
Note: The commands indicated must be run in a PowerShell window as Administrator.
24 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
b. If SMBv1 is enabled (displays True; highlighted yellow in the following screenshot), proceed to the next step (Draining the Node – Maintenance Mode).
i. Post completion of P&U framework disable the SMBv1 configuration (See steps below to disable
SMBv1 configuration) c. If SMBv1 support is disabled, then run the following command to enable SMBv1 configuration on the
node and acknowledge the operation with a "Y" when prompted. Sample output is listed below: PS C:\Windows\system32> Set-SmbServerConfiguration -EnableSMB1Protocol $true
i. Confirm ii. Are you sure you want to perform this action? iii. Performing operation 'Modify' on Target 'SMB Server Configuration'. iv. [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): Y v. Ensure that the configuration now indicates that SMBv1 is enabled (show in the image above).
d. Run the P&U process as per the documentation. e. Post completion of P&U process disable the SMBv1 configuration (see the steps below).
Steps to disable SMBv1 configuration
Note: The commands indicated must be run in a PowerShell window as Administrator.
1. Disable SMBv1 support by running the following command:
Set-SmbServerConfiguration -EnableSMB1Protocol $false
a. Acknowledge by typing in a "Y" at the prompt.
b. Ensure that SMBv1 support is now off by using the Get-SmbServerConfiguration cmdlet. Now
the EnableSMB1Protocol will display False.
25 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
2a. Applying the Dell EMC firmware Patch and Update on the Hardware
Lifecycle Management Host [HLH]
1. Suspend BitLocker by performing the following steps:
Step Activity
a Log into your HLH server.
b Verify that a BitLocker Recovery Password for the C: drive is available: a) Open an elevated (Administrator) PowerShell. b) Run the following command:
(Get-BitLockerVolume -MountPoint “C:”).KeyProtector
c) Make sure that a Key Protector of Type “RecoveryPassword” is listed, and that this numerical password has been saved somewhere outside of the HLH host. This will make it possible to boot the OS in the event that something goes wrong with the update process.
d) If a RecoveryPassword is NOT present, run the following command:
Add-BitLockerKeyProtector –MountPoint “C:” –
RecoveryPasswordProtector
The new password will be displayed on screen when it is created, or you can repeat the command from “b” above to display it again. It is also advisable to check whether a Recovery Password is available for the D: drive, and to create this protector if it is not present. Simply substitute “D:” in place of “C:” in the commands within this step to check for the presence of the password or to create it if it is absent.
26 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
c Suspend BitLocker on the OS volume: a) Open an elevated (Administrator) PowerShell. b) Run the following command:
Suspend-BitLocker –MountPoint “C:”
c) Verify that the “Protection Status” shows “Off”.
2. Now switch over and log into the OME VM to invoke the Firmware P&U script.
3. Open a new PowerShell window with Administrator privileges.
4. Browse to the C:\DELLEMCTools folder where you extracted the DELLEMC Tools zip file and
change directory to the “Post Deploy Firmware Patch and Update Azure Stack” folder.
5. Type the following command at the prompt and press Enter to load all the modules required for
Firmware Patch and Update framework.
Import-Module .\DELLEMCFirmwareUpdate.ps1
27 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
6. Type the following command at the prompt.
Note: For the -IPAddress parameter, enter the IP address of the [HLH] server’s iDRAC. The -
Remediate paramater installs the firmware. If you do not use the -Remediate parameter, only the
inventory of the firmware will be printed out. Press Enter.
Invoke-CheckFirmwareBaseline -IPAddress 10.10.10.10 -Remediate
Note: After invoking the script if you receive and error that looks something like “WARNING: Caught
exception -> WinRM cannot complete the operation”, please consult the Dell EMC Known issues
section
Note: If you encounter errors such as “Cannot invoke method. Method invocation is
supported only on core types in this language mode.” when attempting to execute
the Dell EMC firmware updates from the OME VM, see Appendix A: Updating Security Policies on the
OME VM.
7. After you press Enter, you will be prompted for the iDRAC username (idracUser). Enter the correct
iDRAC username. In the example below, we used root.
28 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
8. Next, you will be prompted to provide the iDRAC password (idracPass). Enter the correct iDRAC
password.
9. Next, you will be prompted to provide the name of the share where the firmware executables are
located (FirmwareShare). This is the same share we created in Step 2 of Preparing the stamp for
updates. In our example, we called it Firmware.
10. Next, you will be prompted for the Host IP address where the Firmware Share is located
(FirmwareShareHost). This is the IP address of the OMEVM where we created the share and
executing this script. Since we are invoking the scripts from OME-VM, it will be the local IPv4 address
of the OME-VM.
29 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
11. Next, you will be prompted to enter the credentials for the account on the OMEVM that has access to
the Firmware Share (FirmwareShareCredential). In our example below, we used Administrator.
At this step, the framework will parse the catalog file and perform health checks, among other
functions.
30 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
The framework will now compare the firmware inventory against the catalog file (Support Matrix
compatible) and print out the results of firmware that is not compliant and needs remediation.
If -Remediate option was used, the framework will start the remediation process now. This includes
creating update jobs and then polling the firmware update jobs. This can be seen from the few
screenshots below.
IMPORTANT: The OME-VM runs on the HLH server. Since the OME-VM is trying to update the HLH
Firmware, the host will restart and the PowerShell session will be lost. Once HLH reboots, please log
into the OME-VM and run the same Firmware Update process for HLH until all the Firmware have
been remediated
31 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
2b. Applying the Dell EMC firmware Patch and Update on Azure Stack
Scale Nodes
Draining the node – Maintenance mode
1. Log in to the Azure Stack Administration portal, and from the Dashboard, click the Region
management tile.
32 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
2. In the left pane, click Scale Units and then in the right pane, click S-Cluster.
Clicking the S-Cluster displays all the nodes in the cluster.
3. Click the first node, (in the example below; SAC21-Node01). A new tile appears on the right, showing
the option to drain the node into maintenance mode. Click the Drain button for the first node.
33 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
The notification center will indicate Draining the node – Running.
Once the node has successfully drained, the Notification Center will show the message “Successfully
drained the node …”. Now the node is in maintenance mode and ready for firmware updates. In the
next section, we will invoke the Dell EMC firmware updates on the drained node.
Invoking the Dell EMC Patch and Update Script
To invoke the Firmware P&U script:
1. Log into the OME VM.
2. Open a new PowerShell window with Administrator privileges.
3. Browse to the folder where you extracted the DELLEMC Tools zip file and change directory to the
“Post Deploy Firmware Patch and Update Azure Stack” folder.
34 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
4. Type the following command at the prompt and press Enter to load all the modules required for
Firmware Patch and Update framework.
Import-Module .\DELLEMCFirmwareUpdate.ps1
5. Type the following command at the prompt.
Note: For the -IPAddress parameter, enter the IP address of the host’s iDRAC that is in
Maintenance Mode. The -Remediate paramater installs the firmware. If you do not use the -
Remediate parameter, only the inventory of the firmware will be printed out. Press Enter.
Invoke-CheckFirmwareBaseline -IPAddress 10.10.10.10 -Remediate
Note: After invoking the script if you receive and error that looks something like “WARNING: Caught
exception -> WinRM cannot complete the operation”, please consult the Dell EMC Known issues
section.
Note: If you encounter errors such as “Cannot invoke method. Method invocation is
supported only on core types in this language mode.” when attempting to execute
the Dell EMC firmware updates from the OME VM, see Appendix A: Updating Security Policies on the
OME VM.
35 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
6. After you press Enter, you will be prompted for the iDRAC username (idracUser). Enter the correct
iDRAC username. In the example below, we used root.
7. Next, you will be prompted to provide the iDRAC password (idracPass). Enter the correct iDRAC
password.
8. Next, you will be prompted to provide the name of the share where the firmware executables are
located (FirmwareShare). This is the same share we created in Step 2 of Preparing the stamp for
updates. In our example, we called it Firmware.
9. Next, you will be prompted for the Host IP address where the Firmware Share is located
(FirmwareShareHost). This is the IP address of the OMEVM where we created the share and
executing this script.
36 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
10. Next, you will be prompted to enter the credentials for the account on the OMEVM that has access to
the Firmware Share (FirmwareShareCredential). In our example below, we used Administrator.
At this step, the framework will parse the catalog file and perform health checks, among other
functions.
37 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
The framework will now compare the firmware inventory against the catalog file (Support Matrix
compatible) and print out the results of firmware that is not compliant and needs remediation.
If -Remediate option was used, the framework will start the remediation process now. This includes
creating update jobs and then polling the firmware update jobs. This can be seen from the few
screenshots below.
39 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
You can also open an iDRAC virtual console and the Job Queue during the updates process to get
real-time updates. If the iDRAC itself needs updating, you will lose access to it and will have to
reconnect.
40 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Once the firmware updates are complete, go back to the Azure Stack Administration portal and
Resume the node.
IMPORTANT: Now repeat the same process for the remaining scale unit nodes.
41 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
How to review updates You can drill into the Update tile to view information about updates that may have already been imported, or
updates you plan to install on a certain date.
After an update package is uploaded to Azure Stack, the top-level Update tile will indicate that an update is
available, and show the current version of the stamp. See the following screenshots to review the in-line
logging and “download full logs” features. Sometimes the update availability information is not updated right
away, so please refresh the portal and it should reflect “Update available”.
46 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
You could also click on the “Download full logs” to get the upgrade summary information log in JSON format.
47 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Monitor updates in Azure Stack using the privileged endpoint
The following information is also available at Microsoft’s Azure Stack website, Monitor updates in Azure Stack
using the privileged endpoint. It is recommended to visit the website in order to get the latest updates and
changes made by the Microsoft Azure Stack team.
Applies to: Azure Stack integrated systems
You can use the privileged endpoint to monitor the progress of an Azure Stack update run, and to resume a
failed update run from the last successful step.
The following new PowerShell cmdlets for update management are included in the 1710 update for Azure
Stack integrated systems.
Cmdlet Description
Get-AzureStackUpdateStatus Returns the status of the currently running, completed, or failed
update. Provides the high-level status of the update operation,
and an XML document that describes both the current step and
the corresponding state.
Get-AzureStackUpdateVerboseLog Returns the verbose logs that are generated by the update.
Resume-AzureStackUpdate Resumes a failed update at the point where it failed. In certain
scenarios, you may have to complete mitigation steps before you
resume the update.
Verify the cmdlets are available Because the cmdlets are new in the 1710 update package for Azure Stack, the 1710 update process needs to
get to a certain point before the monitoring capability is available. Typically, the cmdlets are available if the
status in the administrator portal indicates that the 1710 update is at the Restart Storage Hosts step.
Specifically, the cmdlet update occurs during Step: Running step 2.6 - Update PrivilegedEndpoint whitelist.
You can also determine whether the cmdlets are available programmatically by querying the command list
from the privileged endpoint. To do this, run the following commands from the hardware lifecycle host or from
a Privileged Access Workstation. Also, make sure the privileged endpoint is a trusted host. For more
information, see step 1 of Access the privileged endpoint.
48 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
1. Create a PowerShell session on any of the ERCS virtual machines in your Azure Stack environment
(Prefix-ERCS01, Prefix-ERCS02, or Prefix-ERCS03). Replace Prefix with the virtual machine prefix
string that’s specific to your environment.
$cred = Get-Credential $pepSession = New-PSSession -ComputerName <Prefix>-ercs01 -Credential $cred -ConfigurationName PrivilegedEndpoint
When prompted for credentials, use the <Azure Stack domain>\cloudadmin account, or an account
that's a member of the CloudAdmins group. For the CloudAdmin account, enter the same password
that was provided during installation for the AzureStackAdmin domain administrator account.
2. Get the full list of commands that are available in the privileged endpoint.
$commands = Invoke-Command -Session $pepSession -ScriptBlock { Get-Command }
3. Determine if the privileged endpoint was updated.
$updateManagementModuleName = "Microsoft.Azurestack.UpdateManagement" if (($commands | ? Source -eq $updateManagementModuleName)) { Write-Host "Privileged endpoint was updated to support update monitoring tools." } else { Write-Host "Privileged endpoint has not been updated yet. Please try again later."
}
4. List the commands specific to the Microsoft.AzureStack.UpdateManagement module.
$commands | ? Source -eq $updateManagementModuleName
For example:
$commands | ? Source -eq $updateManagementModuleName CommandType Name Version Source PSComputerName ----------- ---- ------- ------ -------------- Function Get-AzureStackUpdateStatus 0.0 Microsoft.Azurestack.UpdateManagement Contoso-ercs01 Function Get-AzureStackUpdateVerboseLog 0.0 Microsoft.Azurestack.UpdateManagement Contoso-ercs01 Function Resume-AzureStackUpdate 0.0 Microsoft.Azurestack.UpdateManagement Contoso-ercs01
Use the update management cmdlets
Note: Run the following commands from the hardware lifecycle host or from a Privileged Access
Workstation. Also, make sure the privileged endpoint is a trusted host. For more information, see step 1
of Access the privileged endpoint.
49 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Connect to the privileged endpoint and assign session variable Run the following commands to create a PowerShell session on any of the ERCS virtual machines in your
Azure Stack environment (Prefix-ERCS01, Prefix-ERCS02, or Prefix-ERCS03), and to assign a session
variable.
$cred = Get-Credential $pepSession = New-PSSession -ComputerName <Prefix>-ercs01 -Credential $cred -ConfigurationName PrivilegedEndpoint
When prompted for credentials, use the <Azure Stack domain>\cloudadmin account, or an account that's a
member of the CloudAdmins group. For the CloudAdmin account, enter the same password that was
provided during installation for the AzureStackAdmin domain administrator account.
Get high-level status of the current update run To get a high-level status of the current update run, run the following commands:
$statusString = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateStatus -StatusOnly } $statusString.Value
Possible values include:
Running
Completed
Failed
Canceled
You can run these commands repeatedly to see the most up-to-date status. You don't have to re-establish a
connection to check again.
Get the full update run status with details You can get the full update run summary as an XML string. You can write the string to a file for examination,
or convert it to an XML document and use PowerShell to parse it. The following command parses the XML to
get a hierarchical list of the currently running steps.
[xml]$updateStatus = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateStatus } $updateStatus.SelectNodes("//Step[@Status='InProgress']")
In the following example, the top-level step (Cloud Update) has a child plan to update and restart the storage
hosts. It shows that the Restart Storage Hosts plan is updating the Blob Storage service on one of the hosts.
[xml]$updateStatus = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateStatus }
50 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
$updateStatus.SelectNodes("//Step[@Status='InProgress']") FullStepIndex : 2 Index : 2 Name : Cloud Update Description : Perform cloud update. StartTimeUtc : 2017-10-13T12:50:39.9020351Z Status : InProgress Task : Task FullStepIndex : 2.9 Index : 9 Name : Restart Storage Hosts Description : Restart Storage Hosts. EceErrorAction : Stop StartTimeUtc : 2017-10-13T15:44:06.7431447Z Status : InProgress Task : Task FullStepIndex : 2.9.2 Index : 2 Name : PreUpdate ACS Blob Service Description : Check function level, update deployment artifacts, configure Blob service settings StartTimeUtc : 2017-10-13T15:44:26.0708525Z Status : InProgress Task : Task
Get the verbose progress log You can write the log to a file for examination. This can help you diagnose an update failure.
$log = Invoke-Command -Session $pepSession -ScriptBlock { Get-AzureStackUpdateVerboseLog } $log > ".\UpdateVerboseLog.txt"
Actively view the verbose logging To actively view the verbose log during an update run, and jump to the most recent entries, run the following
commands to enter the session in interactive mode, and to show the log:
Enter-PSSession -Session $pepSession Get-AzureStackUpdateVerboseLog -Wait
The log updates every 60 seconds, and new content (if available) is written to the console.
During long-running background processes, the console output may not be written to the console for some
time. To cancel the interactive output, press Ctrl+C.
51 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Resume a failed update operation If the update fails, you can resume the update run where it left off.
Invoke-Command -Session $pepSession -ScriptBlock { Resume-AzureStackUpdate }
Troubleshooting
The privileged endpoint is available on all ERCS virtual machines in the Azure Stack environment. Because
the connection is not made to a highly available endpoint, you may experience occasional interruptions,
warning, or error messages. These messages may indicate that the session was disconnected or that there
was an error communicating with the ECE Service. This behavior is expected. You can retry the operation in
a few minutes or create a new privileged endpoint session on one of the other ERCS virtual machines.
New features and fixes This update includes the following improvements and fixes for Azure Stack.
New features Test-AzureStack cmdlet to validate Azure Stack Cloud available via privileged endpoint
Ability to register a disconnected deployment of Azure Stack
Monitoring alerts for certificate and user account expiration
Added Set-BmcPassword cmdlet in PEP for BMC password rotation
Network logging updates to support on-demand logging
Support reimage operation for Virtual Machine Scales Sets (VMSS)
Enable kiosk mode on ERCS VM for CloudAdmin login
Tenants can activate Windows VMs automatically
Fixes Fix to show Node Operational Status in maintenance while running repair
Fix to correct Public IP usage records time/date stamp
Various other performance, stability and security fixes
TimeSource and Defender privileged endpoint module bug fixes
Windows Server 2016 new features and fixes
January, 3rd - 2018—KB4056890 (OS Build 14393.2007)
o This update includes the software fixes for the industry-wide security issue described by MSRC
Security Advisory ADV 180002.
52 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Known issues with the update process
Microsoft Known Issues
This section contains known issues that you may encounter during the 1712 update installation.
1. Symptom: Azure Stack operators may see the following error during the update process: "Type 'CheckHealth' of Role 'VirtualMachines' raised an exception:\n\nVirtual
Machine health check for -ACS01 produced the following errors.\nThere was
an error getting VM information from hosts. Exception details:\nGet-VM :
The operation on computer 'Node03' failed: The WS-Management service
cannot process the request. The WMI \nservice or the WMI provider returned
an unknown error: HRESULT 0x8004106c."
a) Cause: This issue is caused by a Windows Server issue that is intended to be addressed in subsequent Window server updates.
b) Resolution: Contact Microsoft Customer Service and Support (CSS) for assistance.
2. Symptom: Azure Stack operators may see the following error during the update process:"Enabling the seed ring VM failed on node Host-Node03 with an error: [Host-Node03]
Connecting to remote server Host-Node03 failed with the following error
message : The WinRM client received an HTTP server error status (500), but
the remote service did not include any other information about the cause
of the failure."
a) Cause: This issue is caused by a Windows Server issue that is intended to be addressed in subsequent Window server updates.
b) Resolution: Contact Microsoft Customer Service and Support (CSS) for assistance.
Known issues (post-installation) This section contains post-installation known issues with build 20171201.3.
Portal It may not be possible to view compute or storage resources in the administrator portal. This indicates
that an error occurred during the installation of the update and that the update was incorrectly
reported as successful. If this issue occurs, please contact Microsoft CSS for assistance.
You may see a blank dashboard in the portal. To recover the dashboard, select the gear icon in the
upper right corner of the portal, and then select Restore default settings.
When you view the properties of a resource group, the Move button is disabled. This behavior is
expected. Moving resource groups between subscriptions is not currently supported.
For any workflow where you select a subscription, resource group, or location in a drop-down list, you
may experience one or more of the following issues:
You may see a blank row at the top of the list. You should still be able to select an item as expected.
If the list of items in the drop-down list is short, you may not be able to view any of the item names.
If you have multiple user subscriptions, the resource group drop-down list may be empty.
53 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Note: To work around the last two issues, you can type the name of the subscription or resource
group (if you know it), or you can use PowerShell instead.
Deleting user subscriptions results in orphaned resources. As a workaround, first delete user
resources or the entire resource group, and then delete user subscriptions.
You are not able to view permissions to your subscription using the Azure Stack portals. As a
workaround, you can verify permissions by using PowerShell.
Health and monitoring If you reboot an infrastructure role instance, you may receive a message indicating that the reboot
failed. However, the reboot actually succeeded.
Marketplace Some marketplace items are being removed in this release due to compatibility concerns. These will
be re-enabled after further validation.
Users can browse the full marketplace without a subscription, and can see administrative items like
plans and offers. These items are non-functional to users.
Compute Users are given the option to create a virtual machine with geo-redundant storage. This configuration
causes virtual machine creation to fail.
You can configure a virtual machine availability set only with a fault domain of one, and an update
domain of one.
There is no marketplace experience to create virtual machine scale sets. You can create a scale set
by using a template.
Scaling settings for virtual machine scale sets are not available in the portal. As a workaround, you
can use Azure PowerShell. Because of PowerShell version differences, you must use the -
Name parameter instead of -VMScaleSetName .
Networking You can't create a load balancer with a public IP address by using the portal. As a workaround, you
can use PowerShell to create the load balancer.
You must create a network address translation (NAT) rule when you create a network load balancer.
If you don't, you'll receive an error when you try to add a NAT rule after the load balancer is created.
You can't disassociate a public IP address from a virtual machine (VM) after the VM has been created
and associated with that IP address. Disassociation will appear to work, but the previously assigned
public IP address remains associated with the original VM. This behavior occurs even if you reassign
the IP address to a new VM (commonly referred to as a VIP swap). All future attempts to connect
through this IP address result in a connection to the originally associated VM, and not to the new one.
Currently, you must only use new public IP addresses for new VM creation.
Azure Stack operators may be unable to deploy, delete, modify VNETs or Network Security Groups.
This issue is primarily seen on subsequent update attempts of the same package. This is caused by a
packaging issue with an update which is currently under investigation.
54 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Internal Load Balancing (ILB) improperly handles MAC addresses for back-end VMs which breaks
Linux instances.
55 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
SQL/MySQL It can take up to an hour before tenants can create databases in a new SQL or MySQL SKU.
Creation of items directly on SQL and MySQL hosting servers that are not performed by the resource
provider is not supported and may result in a mismatched state.
Note: You should not have impact to your existing SQL or MySQL resource provider users when
updating your Azure Stack Integrated Systems to the 1712 version. You can continue to use your
current SQL or MySQL resource provider builds until a new Azure Stack update is available.
App Service A user must register the storage resource provider before they create their first Azure Function in the
subscription.
Identity In Azure Active Directory Federation Services (ADFS) deployed environments,
the azurestack\azurestackadmin account is no longer the owner of the Default Provider Subscription.
Instead of logging into the Admin portal / adminmanagement endpoint with
the azurestack\azurestackadmin, you can use the azurestack\cloudadmin account, so that you can
manage and use the Default Provider Subscription.
IMPORTANT: Even though the azurestack\cloudadmin account is the owner of the Default Provider
Subscription in ADFS deployed environments, it does not have permissions to RDP into the host.
Continue to use the azurestack\azurestackadmin account or the local administrator account to login,
access and manage the host as needed.
DELL EMC Known Issues
Symptom: Sometimes when running the DELLEMC Firmware Patch and Update framework, right at the start
of validating the firmware, the PowerShell outputs the following exception.
WARNING: Caught exception -> WinRM cannot complete the operation. Verify that
the specified computer name is valid, that the computer is accessible over the
network, and that a firewall exception for the WinRM service is enabled and
allows access from this computer. By default, the WinRM firewall exception for
public profiles limits access to remote computers within the same local subnet.
Cause: The root cause of this particular issue is unknown at the moment.
Resolution: The workaround is simply to re-try the command again and when you try the second time, the
framework connects to the iDRAC just fine and validates the firmware baseline.
56 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Appendix A: Updating Security Policies on the OME VM If you encounter errors, such as “Cannot invoke method. Method invocation is supported
only on core types in this language mode.” when attempting to execute the Dell EMC firmware
updates from the OME VM, it will be necessary to update the security policies to white-list the scripts that
perform the updates.
Step Activity
1 Download the Latest DellEMC Tools from the following link:
https://support.emc.com/downloads/42238_Cloud-for-Microsoft-Azure-Stack
2 a.) Extract the update files from the DellEMC Tools package to C:\PU on the OME VM.
b.) Copy “UpdateWDAC\” to C:\Security on your OME VM.
3 Open a PowerShell session with Administrator privileges on the OME VM.
4 Navigate to C:\Security\UpdateWDAC.
5 Run the following script:
.\CreateOMEAuditPolicy.ps1
6 Reboot the VM.
7 Log back in to the OME VM and Open PowerShell with Administrator privileges.
57 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
8 Run the following command, which will clear the existing code integrity events.
Wevtutil.exe cl Microsoft-Windows-CodeIntegrity/Operational
9 Navigate to C:\PU to locate the SupportAssist Enterprise installer.
10 Execute SupportAssistEnterprise_1.2.0.36.exe and select “Upgrade”.
11 Navigate back to C:\Security\OMEWDACUpdate.
58 1712 Dell EMC Cloud for Microsoft Azure Stack Patch and Update Guide | version A01
Step Activity
12 Create the final enforced whitelist by executing the following command:
.\CreateOMEEnforcedPolicy.ps1
14 Reboot the OME VM so that the policy takes effect.
15 The DellEMC update scripts should now be white-listed, and should execute without errors.