take your oracle weblogic applications to the next … · take your oracle weblogic applications to...

Take Your Oracle WebLogic Applications to The

Next Level with Oracle Enterprise Manager 12c

Mojahedul Hoque Abul Hasanat CTO, Therap Services

Neelima Bawa Consulting Tech. Lead, SCP, EM, Oracle

Agenda

• Background of Therap Services

• The problem

• Application Performance Management

• Quick description of Oracle’s APM offering

• How OEM and RUEI helped us

• Actual scenarios

• The future

• Tips

• Q&A

Therap Services - OOW 2013 2

Therap Services, LLC

• Documentation and Communication Software for MR/DD

• EHR for the DD industry is the closest for describing us

• Niche segment in the health sector

• Improve quality of life for people with DD by improving efficiency of delivery through communication

• SaaS business model

• 150K+ active users

• 1000+ providers in 48 states

• State customers

• Extensive usage for DD in DHS ND and DHHS NE

• 150+ employees

• Based in CT, dev center in Bangladesh

• http://www.therapservices.net


http://www.therapservices.net

The Application

• The application is our business

• 1M+ lines of code

• 60+ modules

• 1M+ sustained HTTP requests/hour

• 30K+ peak requests/minute

• 6000+ concurrent users

• Based on JEE and the Spring Framework

• Hibernate

• Seam

• GRAILS


Delivery Platform

• 2 identical sites in two states

• Primary hosts (per site):

• 4 WebLogic application servers in cluster

• 1 Memory based data server (in-house, java)

• 1 Oracle database server

• 1 NetApp storage (SAN)

• 1 F5 Load balancer

• Supporting hosts

• Use Dyn for site high availability

• Data replication with Oracle Golden Gate


What Matters

• Availability

• Application is used 24x7

• Application use is critical to the business of our customers

• Performance

• A user needs to spend as little time as possible in our application

• Most users use it daily, multiple times

• Data integrity

• Fast development turnaround


Evolution of Therap

• Improved testing

• Formal code review

• Improved processes

• Removed repeating problems

• Now all problems we face are new

• Acquired large customers

• Availability and performance have become critical factors


Before OEM & RUEI

• Heavy use of logging

• Nagios

• Cacti

• kill -3


The Problem

• Diagnosis of performance issues

• Has become much harder with the growing system

• Application availability

• With a larger customer base, uptime has become a major factor

• Complexity of the system increases difficulty

• Limits of logging

• Works for known unknowns

• Need infrastructure to visualize and store historic data

• Limits of OS based monitoring

• Limited metrics

• Limits of simple JMX monitoring


Application Performance Management

• Deep insight into running application

• Profiling at runtime

• Some bottlenecks are only visible at runtime

• Historic data

• Invaluable for preventing performance regression


Which Vendor?

• We were moving from JBoss to WebLogic

• JVM Diagnostics

• Extensive WebLogic metrics

• Probably the best database diagnostics

• Our team was already familiar with OEM

• Deep integration with the database

• Integration of JVMD and app server metrics

• Expect better support from Oracle


Oracle’s APM

• Oracle Enterprise Manager 12c

• WebLogic Metrics

• Middleware Diagnostics Advisor

• JVM Diagnostics

• Configuration Management

• Incident Management

• Lifecycle Management

• Oracle Real User Experience Insight

• Oracle Business Transaction Management


WebLogic Metrics

• Pro-active monitoring

• Helps us in avoiding downtime

• Correlation between various metrics

• Middleware Diagnostics Advisor


JVM Diagnostics

• Deep insight into the JVM

• Invaluable for understanding application performance issues

• Helped us in identifying log4j bottleneck

• Early identification of performance problems


Oracle Real User Experience Insight

• Measure performance seen from the customer end

• Detect performance regression

• Enables shorter release cycles

• Quick and real feedback for performance tuning

operations


Our Journey


Timeline

• Evaluation of various vendors – July 2012

• Purchase of WebLogic, OEM, RUEI – Nov 2012

• Start JBoss to WebLogic migration

• Start building expertise on OEM

• Start using OEM in test environment

• Fix problems found through OEM, JVMD

• Production deployment – Mar 2013


How OEM and RUEI helped us


The log4j bottleneck

• During load testing, we could not increase load beyond a

certain point

• CPU load was low

• JVMD showed us something that we could hardly believe

• Many threads were contending for lock for writing to the

log file

• The contention only shows up at high loads

• Used JVMD heavily to find the best logging backend and

the best configuration


log4j…


log4j...


EJB Transaction Optimization

• Noticed abnormally high number of bean transaction

commits

• We had forgotten to optimize some frequently used EJB

• Read-only methods do not need to be transactional


EJB Transaction Optimization…


Unexpected Top Method

• Noticed a JMS listener in the top method list

• In production!

• Did not show up during synthetic load testing

• We forgot to add a “message selector” on the listener


Top Method…


The MDA Catch

• MDA reported an unexpected “The EJB is taking too long

to execute”

• Related method was showing in the top methods list

• There were extraneous calls to the EJB

• The method did not need to be in an EJB


The MDA Catch…


The Slow Library

• A library call for producing JSON showed on the top

method list

• JSON is needed for AJAX

• It was totally unexpected

• The library was old and inefficient

• Replaced it with a newer and more efficient library


The Slow Library…


In-efficient Network Write

• Initially discovered in production through JVMD

• There were instances of high network waits

• Methods a certain module in the application showed up in

the top list during the high network wait periods

• Discovered a 3 level loop that writes data

• Further inspection through JProfiler confirmed it


In-efficient Network Write…


Automatic Thread Snapshots

• Previously, relied on kill -3

• Manual, missed dumps at crucial moments

• Now, JVMD takes thread snapshots when an abnormal

thread state is reached on any WebLogic server

• Combined with auto-restart from WebLogic, eliminated

unplanned downtime


Compliance

• Helps us identify patches needed for:

• WebLogic

• Oracle Database

• OEM

• Downtime log

• Useful for tracking operations improvement


The JDK Upgrade

• Upgraded JDK from 1.6.0_29 to 1.6.0_45

• At that time, we have not brought RUEI into our regular

operations process

• Started getting slowness complaints after a few days

• A look into RUEI instantly revealed performance

regression

• Downgrading the JDK fixed the regression completely


The JDK Upgrade…


The JDK Upgrade…

• We could not reproduce the performance regression in lab

environment

• We now have a new procedure to upgrade JDK

• Do normal tests and load tests as before

• In production, upgrade the JDK of one server only

• Wait a few days

• Compare performance

• Decide whether to upgrade or rollback


The Results

• 1 unplanned downtime in the last 4 months!

• Improving DevOps culture

• Foster collaboration between dev, ops and DBA

• No more fighting between dev and DBA

• With RUEI, even the business team joins the fun


Most

Importantly

I can do this!


Future with OEM and RUEI


Future

• Involve more people

• Use KPI in RUEI

• Integrate KPI with OEM by defining a Business

Application

• Leverage alerting and incident management in OEM


Challenges

• Steep learning curve

• Took us time to understand the role of BTM and JRF

• Better documentation available now

• Lack of full WebLogic 12c support when we started

• Should be solved in latest release of OEM

• Support has been fanatical!

• Thank you Oracle!


Tips

• Understand what you need

• Any web app with performance requirements needs RUEI

• If you face performance issues, you will need JVMD

• Engage as many people in your team as possible

• Give wide access to many

• The learning curve

• The problems these tools help you with are also complex


Q&A


Other Sessions

• JVM Diagnostics: Java Profiling in Production

Environments [CON9571]

• Thursday 2:00pm – 3:00pm @ Moscone North - 130



Contact

[email protected]

http://neelimabawa.blogspot.com

[email protected]

@Masum6

mailto:[email protected]

http://neelimabawa.blogspot.com

mailto:[email protected]

take your oracle weblogic applications to the next … · take your oracle weblogic applications to...

Documents