kd-2014-technical-tooling-for-a-major-kfs-upgrade

Technical Tooling for a Major KFS Upgrade

Shyam Gedela ([email protected])

Harsha Ummerpillai ([email protected])

Michigan State University

11/11/2014

1

Kuali Days 2014Indianapolis 2

Overview

• Upgraded from KFS 3/R1 to KFS5/R2• Overall Goal

– Vanilla– No functional impact - technical upgrade– Shared Rice so KC can integrate

• Emphasis on tooling, not upgrade process


Background

• Foundation• Liquibase scripts were not enough• Scripts to update status code not fast• Rice maintenance document update too simplistic• No easy way to take changes like RC Code

• MSU• No central issue tracking• Non standard commit comments• 3.0/R1 to 5.0/R2• Most functionality will go forward

5

Understand the problem.

Analysis


Collect data, correlate and make it searchable.1. Functionality changes – Use Revision Tool

1. Application Enhancement Requests (AER) list

2. Defect tracking tool (DTT) list

3. Application Support Request (ASR) list

4. Code changes (Fisheye)

Analysis


Searchable database using code and issue ID - AER, ASR, DTT, Fisheye1. Identify issues with code impact (F)

2. Correlate items from different lists (F)

3. Search unit of work using issue ID or code file name (T)

4. Categorized each issue into a decision matrix (F)1. MSU Only (e.g. Interfaces, Conversions, some Features)

2. Contribute

3. Already in foundation

4. Remove from production• Change in business process

F – Functional, T - Technical

Analysis – Revision Tool

8

Revision Tool


1. Code upgrade.

2. Database upgrade

1. Metadata upgrade.

2. Data split and upgrade.

3. KIM and Workflow upgrade.

3. Split Rice

4. Automate

5. Practice makes perfect.

Overall Technical approach

10

Customizations created difference between the foundation and MSU version.

Code Upgrade


Code Upgrade Approach

• Transfer all MSU code from 3 to 5– Tool to compare deltas before and after

• Technical review to ensure complete transfer• Remove code based on functional approval• Git tooling for mixing bowl approach


Code Upgrade KFS 3.0 to 5.0

13

Database upgrade involved metadata and data upgrade.

Database Upgrade


Metadata Approach


Metadata Upgrade

• Submit all changes to functional review.• Blanket Rules

– Accept new columns.– Accept columns made null.– Accept column size increase– Accept column data type changes.– Accept new tables and renames.– Accept new and modified Foreign Keys.– Accept new and modified Primary Keys.– Ignore dropped columns.– Ignore columns made not-null.– Ignore column size decrease.– Ignore FKs removed.– Accept bootstrap/mixed data changes

16

Data Approach


Database Upgrade KIM

• Primary Keys changed• Complexities with uniquely identifying and

comparing data.• Special JDBC programs to print diff reports.• Needed functional review.• Custom/manual SQL script generation.• Review and apply against production data.• Run custom comparators to review impact after

script is applied.


Database Upgrade KEW

• Exported production version of KEW XML files.• Exported vanilla version of KEW XML files.• Compare XML files using “WinMerge”.• Functional review of differences.• Manual edit of production XML files.• Combine, sort and merge all XML files into a

single DocTypes.xml per application.• Ingest into upgraded database to verify the

impact.

19

Phase 1 is done! Onto Phase 2


Split Rice • Create new Rice environment• Split Rice data from applications• Workflow doc IDs didn’t collide

21

Production Move


Upgrade Automation


Production Move Plan & ExecutionID Pred Task Owner

Planned Start Start Time Duration Prod Notes

1Verify PROD shared rice setup (DS xml files, Lucene and configuration properties) MW 3/6/2014 8am 30 mins

2Copy rice root war to rice prod server to check URL is working MW 3/6/2014 8am 15 mins FN-D will provide root war to MW

3

On all application servers (6 servers total) change log level for org.apache.cxf to ERROR in jboss-log4j.xml located at /home2/jboss/server/ais/conf MW 3/6/2014 8am 30 mins

applied change to prod rice servers but needs to wait for Jboss conversion on kfs to make those changes. Will be finished in step 6

4 Create release tag and reconfigure bamboo plans FN-D 3/6/2014 9am 30 mins

5 Post Service Alert - Finance & Data ServicesIntegration 3/10/2014 12pm 15 mins

6 Prep for Jboss upgrade MW 3/12/2014 12pm 2 hrsTar up new version and other preparation

7Verify KFS DBA login is working in FINPRD as designed - remove after use DBA 3/13/2014 9:00am 15 mins

8Backup (level 0) FINPRD and MSUDIM and MSUEDW DBA 3/13/2014 11am 6 hrs

9Ensure times across prod servers are synced (including Rice) VSS 3/13/2014 12pm 30 mins

automate this by creating daemon process

10 1 Verify new sentinel app id RICE Security 3/13/2014 1pm 30 mins

11 10Verify that authorize existing application (KFS, KMM, OOI) users to Shared Rice (RICE) Security 3/13/2014 1:30pm 30 mins

12Remove D6501.IDM.DAILY.PF from Saturday night’s schedule Prod Svcs 3/13/2014 11am 15 mins

13 Prep for bamboo plans FN-D 3/13/2014 9am 1 hr

14

disable UC4 jobs - N81.DBA.ORACLE.STORES.TO.MSUEDW.COPY and N81.DBA.ORACLE.KFS.TABLES.COPY Prod Svcs 3/14/2014 1pm 15 mins disable sat. & sun.

15 Create EDW and DIM restore point for ETL and BI DS 3/14/2014 8am 4 hrs

16 Run N81.FP.PCARD.NIGHTLY.REVIEW.PF at noon Prod Svcs 3/14/2014 12pm 15 mins

17 16 Functional review of Pcard File FN-F 3/14/2014 12:15pm 15 mins

18 17 Run N81.FP.PCARD.NIGHTLY.DOCUMENT.PF Prod Svcs 3/14/2014 12:30pm 30 mins

19Lock Finance App logins except l ist of validation NetIDs MW 3/14/2014 7pm 30 mins script

20 Shut down Paybase Servers (including PBPRD) VSS/DBA 3/14/2014 7pm 10 mins

21 19Run KFS 3 Nightly Batch (with N81.CAB.EXTRACT Job disabled) Prod Svcs 3/14/2014 7:30pm 2 hrs

22 21

Verify N81.Kuali.Nightly.PF (KFS nightly batch) and D6501.KFS.Apply.Updates (from IDM) are complete. Stop Queue.N81.KFS. Prod Svcs 3/14/2014 9:30pm 15 mins

23 22 Generate CAB Reconcil iation List in Production FN-D 3/14/2014 9:45pm 15 mins

24 22 Check that all Log Activity stopped FN-D 3/14/2014 9:45pm 30 mins

25 24 Post KFS Redirect page (update F5) MW 3/14/2014 10:15pm 15 mins

26 24 Stop XI Communications N82.BC.X.STOP.CC.PF Prod Svcs 3/14/2014 10:15pm 5 mins BASIS backup may be needed

27 25 Disable WebMon MW 3/14/2014 10:30pm 10 mins

28 26 Run N81.SYS.STOP.ALL.SERVERS.PF Prod Svcs 3/14/2014 10:20pm 15 mins

29 28Backup (level 1) FINPRD and MSUDIM and MSUEDW DBA 3/14/2014 10:35pm 30 mins plus Sergey

30 29

Pre-snapshot DBA task (stop OEM, check/update autostart, disable backup, change process parameter - 1200) DBA 3/14/2014 11:05pm 15 mins

31 30 Perform snapshot VSS 3/14/2014 11:20pm 30 minsShutdown DB VMs except ebsprd100d1 and app VMs

32 31 Run EDW/DIM Finance/OOI Nightly Batch Prod Svcs 3/14/2014 11:50pm 5 hrs

33 31 Jboss Upgrade and File Configuration MW 3/14/2014 11:50pm 3 hrs

clean out all files in ais-deploy directory, copy new wars from jboss-deployment, keep logs directory, rename latest server.log. /home2/n81/jboss-deployment - location to copy wars from, copy out opt files rice-config and rice keystore to each production

34 32Check if EDW/DIM-FIN Batch run is complete. Stop Queue.N85.EDW.Finance Prod Svcs 3/15/2014 5am 15 mins

35 32 Build shared Rice schema (All except 4 tables) DBA 3/15/2014 6am 2 hrs

36 34 Create EDW/DIM restore point DBA 3/15/2014 8am 15 mins


Resources

• https://tools.test.ebsp.msu.edu/

https://tools.test.ebsp.msu.edu/

https://tools.test.ebsp.msu.edu/

kd-2014-technical-tooling-for-a-major-kfs-upgrade

Documents

kuali days

indianapolis code upgrade

indianapolis metadata

workflow upgrade

msu code

indianapolis metadata

status code

code file