0

Mechanic v. Surgeon(photos from istockphoto.com)

1

2007: AIX Kernel hotpatch support available with AIX 6.12008: Ksplice Linux hotpatch support (based on MIT student’s master’s thesis)

• Stops all running processes – can’t continue if a thread in a function to be patched

2011: Ksplice acquired by Oracle, no new subscribers2014: kGraft: patching by SUSE

• No patches to kernel data structures possible• Uses both old and new versions of patched functions until complete (all

threads exit kernel space)2014: kpatch: patching by Red Hat

• No patches to kernel data structures possible• Relies on stack backtraces (questionable reliability)• Stops all running processes during patching – can’t continue if a thread in a

function to be patched

https://lwn.net/Articles/634649/

(photo from istockphoto.com)

2

Step 1:

• Prepare the new rootvg to be able to start the surrogate.

• Normal alt-disk-copy / update process – but transparent to admin because this is for temporary use.

• Customize the new rootvg with the live update environment

Step 2:• Initiate from AIX using HMC APIs• Reserve CPU/MEM resources from the HMC

Step 3:• Initiate from AIX using HMC K2 APIs that interact with VIOS• Same storage accessible from both LPARs (split the paths)• Original rootvg is mirrored (mirror will be split later)• Private vlan required between LPARs for the move

Step 4:• Some “conditioning” of the surrogate LPAR required

• identical device configuration (same devnos)• network interface not yet configured until IP takeover•Prepare scratch filesystem on the current boot rootvg for chrooted environment

Step 5:• minimal “WPAR-style” checkpoint of each process

• designated pids are not checkpointed• app memory moved asynchronously• Sync and split the mirrored VG on the original• Import the original vg in the chrooted env.• all communication is via temporary, private vlan• force same pids, tids, IPC ids for restarted processes• original IP address configured before restart• update runs chrooted in original rootvg on surrogate LPAR

• normal in-place update, but when complete, no reboot required• Prepare the original rootvg to be the primary boot disk for the surrogate

• Monitor the transfer of mem (async mobility).

• For alt_disk_install compatibility, prepare the mirror of the original rootvg (In case, it needs to be used to restore the previous level)

Step 6:

• Shutdown the original

• Return cpu and mem, resources to the HMC

• Mirror of original rootvg is available if needed to go back to pre-update state

Required lvupdate.data file: /var/adm/ras/liveupdate/lvupdate.data(Start with the lvupdate.template file in same directory)

general:kext_check = no

disks:nhdisk = <hdisk name(s) for surrogate boot>mhdisk = <hdisk name(s) for new rootvg mirror>

hmc:lpar_id = <requested new lpar_id>user = <HMC user name>management_console = <hostname or IP addr>

Command LineAuthenticate with HMC

hmcauth –u hscroot –a hmc_namePreview

geninstall –k –p –d /tmp IZ12345.140806.epkg.ZRun Live Update

geninstall –k –d /tmp IZ12345.140806.epkg.Z

10

SMITsmitty -> Software Installation and Maintenance -> Install and Update

Software -> Install Software -> enter input device

Change the option: INVOKE live update? to yes

NIM SetupDefine NIM environment:Generate HMC Password Key

# /usr/bin/dpasswd -f /export/eznim/passwd/hmc_passwd -U hscroot -P abc123

Use Key to Define HMC object# nim -o define -t hmc -a if1="find_net hmc_object 0" -a net_definition=“ent255.255.255.0 9.1.2.1” -a passwd_file=/export/eznim/passwd/hmc_passwdhmc_object

Define Managed System of NIM Standalone# nim -o define -t cec -a hw_type=8203 -a hw_model=E4A -a hw_serial=0123456 -a mgmt._source=hmc_object cec1

Exchange SSH keys between HMC and NIM master# dkeyexch -f /export/eznim/passwd/hmc_passwd -I hmc -H hmc_object

Define the NIM standalone pointing to the CEC# nim -o define -t standalone -a if1=find_net mac1 0” -a net_definition=“ent255.255.255.0 9.1.2.1” -a net_setting1=“100 full” -a mgmt_source=cec1 -a identity=<lpar_id> client1

Note: NIM Live Update will call hmcauth during the cust operation to authenticate the NIM client with the HMC using the HMC passwd_file.

NIM Usage•From NIM master

•Using a NIM live_update_data resource, lvup# nim -o cust -a live_update=yes -a live_update_data=lvup -a lpp_source=720lpp -a fileset=IZ12345.140806.epkg.Z client1

•Using the client’s /var/adm/ras/liveupdate/lvupdate.data file# nim -o cust -a live_update=yes -a fileset=IZ12345.140806.epkg.Z client1

•Preview# nim -o cust -a live_update=yes -a live_update_data=lvup -a install_flags=“-p” -a lpp_source=720lpp -a fileset=IZ12345.140806.epkg.Z client1

10

•From NIM client•Using Separate Operations to Allocate and Run Live_Update

# nimclient -o allocate -a lpp_source=720lpp -a live_update_data=lvup# nimclient -o cust -a live_update=yes -a fileset=IZ12345.140806.epkg.Z

•Allocate and Run Live_Update# nimclient -o cust -a live_update=yes -a lpp_source=720lpp -a live_update_data=lvup -a fileset=IZ12345.140806.epkg.Z

•Preview# nimclient -o cust -a live_update=yes -a lpp_source=720lpp -a live_update_data=lvup -a install_flags=“-p” -a fileset=IZ12345.140806.epkg.Z

(diagram from istockphoto.com)

10

Requisites:System firmwareAx730_066*Ax740_043*Ax770_063Ax773_056Ax780_056Ax810 or later

* Limitation: PowerVC can not seamlessly manage the updated LPARHardware Management Console (HMC)

840Virtual I/O Server

2.2.3.502.2.4

RSCT (if required)3.2.1.0

PowerHA® (if required)7.2.0

PowerSC™ (if required)1.1.4.0

Subsystem Device Driver Path Control Module (SDDPCM) (if required)TBD

11

I/O restrictions• Any Coherent Accelerator Processor Interface (CAPI) device must not be open

during the Live Update operation. • No physical or virtual tape or optical device is supported. These devices must be

removed before the Live Update operation can proceed. • The mirrorvg utility can mirror up to 3 copies. If the root volume group of the

original partition is already being mirrored with 3 copies, the Live Update operation cannot proceed.

• The Live Update operation is not supported on diskless AIX clients. • The Live Update operation is not supported in a multibos environment. • Data Management API (DMAPI) is not supported by the Live Update feature. • Virtual Small Computer System Interface (vSCSI) support for the Live Update

operation is only for those logical unit numbers (LUNs) that are backed by physical volumes, not logical volumes.

• The vSCSI disk support excludes the option where the vSCSI server adapter can be mapped to any partition or partition slot.

Security restrictions• The Live Update operation is not supported when a process is using Kerberos

authentication. • The Live Update feature does not support PowerSC Trusted Logging. • The Live Update feature is not supported by an active Department of Defense

(DoD) security profile. • The Live Update feature is not supported when audit is enabled for a stopped

workload partition (WPAR). • The Live Update feature does not support Public-Key Cryptography Standards # 11

(PKCS11). The security.pkcs11 fileset cannot be installed. • The Live Update feature is not supported by any of the following Trusted Execution

options in the trustchk command:•TEP=ON •TLP=ON •CHKSHLIB=ON and STOP_UNTRUSTD=ON •TSD_FILES_LOCK=ON

Reliability, availability and serviceability (RAS) restrictions• System trace of the Live Update operation is not possible if channel 0 is already in

use. • The Live Update feature is not supported when ProbeVue is running. The ProbeVue

session needs to be stopped to run the Live Update operation. • User storage keys are not supported in the Live Update environment. • The system dump that is present on the root volume group of the original LPAR is

11

not available after a successful Live Update operation.

Miscellaneous restrictions• The interim fix must have the LU CAPABLE attribute, which means the interim

fix must be compatible with the Live Update operation. The emgr command can display this attribute. Ideally, all the interim fixes can be applied with the Live Update operation, but there might be some exceptions.

• The destination of the interim fixes must be on the root volume group of the client partition in either /, /usr, /home, /var, /opt, or /tmp file systems.

• The Network File System (NFS)-mounted executables must not be running during a Live Update operation.

• Active WPARs must be stopped before the Live Update operation. • RSCT Cluster Services are stopped during a Live Update operation, and then

restarted before the Live Update operation completes. • A configuration with 16 MB page support is not allowed. The promoted (16 MB

Multiple Page Segment Size (MPSS)) pages by Dynamic System Optimizer (DSO) are supported by the Live Update operation.

• The Live Update operation is supported when the DSO running, but DSO optimization is reset by the Live Update operation. The optimization begins again based on workload monitoring after the Live Update operation.

• The Live Update feature is not supported on a partition that participates in Active Memory Sharing (AMS).

• The Live Update feature is not supported on a remote restartable partition. • If an interim fix is installed without the Live Update operation that requires a

restart, the restart must be completed before a subsequent Live Update operation can be started.

(photo from istockphoto.com)

11

Demo video

12

No commitment expressed or implied for future releases. All plans subject to change without notice.

(Photo by Aaron Sheffield (acsheff.wordpress.com), used by permission)

13

Compare Live Update to LPM … will take time to build this out so it’s widely applicable. Let’s accelerate the process for Live Update! Sign up for the Early Ship Program (see final 2 slides), put 7.2 in a test environment, put some real workloads in there, and wring this out so we can make a No Reboot Required world a reality.

(photo from istockphoto.com)

14

15

16

17

18