Eric Westfall – Indiana University

James Bennett – Indiana University

ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE

KUALI RICE @ INDIANA UNIVERSITY

• Integrated with 26 enterprise applications

• Hosting 50 eDocLite applications

• Over 400 different KEW Document Types

• 24 x 7 x 365 availability

• Approximately 99.8% uptime in the last year (including maintenance)

EXPANDING IMPLEMENTATION

2003 2004 2005 2006 2007 2008 2009 20100

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

9000000

10000000

11000000

12000000

13000000

Documents RoutedRequests GeneratedActions Taken

USAGE

2003 2004 2005 2006 2007 2008 2009 20100

10000

20000

30000

40000

50000

60000

70000

80000

Number of Unique Users

# of Unique Users

ARCHITECTURE AND INFRASTRUCTURE

• Red Hat Enterprise Linux on VMware

• Load balancer with four clustered Kuali Rice standalone server instances

• Java 6

• Tomcat servlet container

• Shared file system mount for attachments

• Oracle 11g database

IU KUALI RICE ARCHITECTURE

MAINTAINED BY

• A team of 3 FTE, who do…

• Production Support

• System Maintenance

• Upgrades

• Integration Projects

• Application and eDocLite Development

PRODUCTION SUPPORT

• Areas of concern:

• Database resources

• Service bus

• Ingestion

• Clustering/Load Balancing

• Tools we use

DATABASE RESOURCES

• Recently updated to Oracle 11g

• Clients primarily use embedded

• Concerns:

• Connection pool limit

• Database blocking

• Client proxy accounts

• Export/import job for refreshing test environments

MANAGING THE SERVICE BUS

• Bad service endpoints

• Clean shutdown of client applications

• Monitoring/maintaining the message queue

CLUSTERING/LOAD BALANCING

• Horizontally scalable

• Session affinity

• Session replication required for full failover

• Unique endpoints per machine

TOOLS - DYNATRACE

• Performance monitoring

• Error reporting

• Drill down into requests

• Take memory/thread dumps

DYNATRACE EXAMPLE

TOOLS - DATABASE

• Reporting:

• Stuck documents

• Database blocking

• Cleanup

• User options table

• Message queue

SYSTEM MAINTENANCE

• Rice customization/patches

• Environment configuration

• Load Testing

• Functional testing

• Data migration

RICE CUSTOMIZATION/PATCHES• Two options:

• Override Spring bean (ex: IU Group Service)

• Override a particular file

• Process

• Copy source

• Make necessary changes

• Build the standard Rice source

• Overlay customized files

• Deploy or distribute changes

ENVIRONMENT CONFIGURATION

• Java environment managed by a J2EE group

• Build process deploys:

• Application code

• Settings/security files

• Most Java applications share the same build process

• http.url configuration parameter

LOAD TESTING

• Using Apache JMeter

• Perform standalone and coordinated load tests

• Load testing transactions include

• Document search

• Action list

• Taking mass actions

• Viewing route logs

• Creation and routing of documents

LOAD TESTING EXAMPLE OUTPUT

FUNCTIONAL TESTING

• Uses Selenium

• Runs nightly through Bamboo

• Functional test suite includes:

• Most eDocLite applications

• Maintenance documents (group, rule, etc)

• Document search

• Action List/Preferences

DATA MIGRATION

• Database job to move production data into testing environments

• Use the “export” feature

Dev Test Pre-production Production

UPGRADING RICEProcess:

1. Bring in new Kuali Rice foundation jars

2. Make changes to the IU version of Rice which includes:

• Updating customizations

• Merging "patched" code

• Produce IU Rice client jars

3. Perform DB updates to test environment

4. Bring up standalone testing environments

5. Work with clients to bring in IU Rice client jars and update code

6. Testing before making the change to production

INTEGRATION

• Client application configuration

• Keystore Administration

CONFIGURING CLIENT APPLICATIONS

• Spring files (ex KIM overrides)

• http.url configuration parameter

• Document handler URL

• Namespace (if appropriate)

• Roles/permissions

• Security file deployment

KEYSTORE ADMINISTRATION

• Keystore needed for KSB security

• Keytool commands

• genkey

• selfcert

• export

• import

• list

KEYSTORE ADMINISTRATION

Items to consider in a production setting:• Keep certificate expiration dates in mind

• Testing and prod keystores should not be the same

• New applications must:

• Create a certificate

• Include a copy of the KR certificate in their keystore

• Give their certificate to be included in the KR keystore

EDOCLITE DEVELOPMENT

Process:

1. Request comes in for a new eDocLite

2. Meet to review requirements

3. Develop initial eDocLite

4. Test and make changes until ready for prod

5. Ingest eDocLite into production

IU EDOCLITE EXAMPLES

IU EDOCLITE EXAMPLES

EDOCLITE MANAGEMENT• Currently have 50 eDocLite apps in production

• Reasons for changes to existing eDocLites

• Annual changes due to nature of form

• Addition of a new field

• Change to drop down contents

• Change to field validation

QUESTIONS?