application and performance management techniques for j2ee “management built in” for the...
TRANSCRIPT
Application and Performance Management Techniques for J2EE
“Management Built In” for the Adaptive
Enterprise
Scott L. Williams HP
Agenda
• Challenges: Testing, Management, & Repair of Applications• Embracing Management Throughout The Application Lifecycle• Management Tools for Operations• Instrumenting Your Applications For The Operational Environment• Demonstration of Application Instrumentation Tools• Call to Action
Challenges: Testing, Management, & Repair of Applications
RequirementsGathering
Design &Develop
Testing Staging Production
1.
Application is “hanging” in production. Use OV to do “root cause” analysis and send results to developers.
2.
Use “root cause” analysis results to narrow down which software contains bug. Fix bug.
3.
Deploy bug fix to staging. Execute regression tests to verify integrity of bug fix.
4.
Release bug fix to production.
The Ideal: Simple, Quick Resolution & Repair
Application Development
IT/Operations
erroroccurs
No way to help developers find error within the application!
the application
No Shared “Context” Difficult For IT/Ops To Correct Problems
Examples of common pain points today
• Application integration (many technologies, techniques, etc.)
• Poor communication (business+dev+operations)
• Deployment failures (insufficient information)
• Difficulty configuring application security (technology, process)
• Maintenance (again, insufficient shared information)
Most time and money is dedicated to maintenance in production
10%
Operation
Consolidation/Integration
Innovation
30%
60%
IT Today
Source: HP IT department
Removing The Barriers To SuccessFree up developer resources
– Today: developers often spend too much time supporting their applications when those applications degrade or fail in production
– Goal: spend less time supporting applications and more time creating new ones (greater alignment with new business needs)
Free up IT resources– Today: Large numbers of operations staff; manual processes;
“archeology” of poorly documented applications– Goal: Well documented apps; automated troubleshooting; better
management at all levels (infrastructure, apps, business processes)
Drive complexity out of the software lifecycle– Today: Hidden assumptions, poor documentation, “silos”– Goal: Meta-data (documentation); “management built in” to applications;
context shared throughout entire lifecycle (business-development-operations communicate);
Adaptive Enterprise design principles
+
+
+Easily manage and modify the environment through a uniform system of relationships.
Improve performance by managing infrastructure components discretely or collectively.
Standardize the way IT assets are used with common components and processes.
Simplify applications and systems to reduce complexity and risk.
Applied consistently across:
•Business processes
•Applications•Infrastructure
Simplification
Standardization
Modularity
Integration
What is “Management for the Adaptive Enterprise?”
• Do more with less• Remove barriers to success• Shift IT Investment from maintenance to innovation• Be ahead of the competition• Be competitive or be outsourced
Moving from maintenance to innovation
10%
40%
Operation
Consolidation
Innovation30%
60%
Today3 years
from now
20-25%
35-40%
• In large enterprises up to 75 % are personnel or services costs
• Hardware, software and infrastructure only 25 %
Source: HP IT department
Embracing Management Throughout The Application
Lifecycle
Application Development
IT/Operations
erroroccurs
No way to help developers find error within the application!
the application
Application Development
IT/Operations
Shared “Context” IT/Ops Can Correct Problems
erroroccursUse “context” to communicate problem
back to application developers
the application
CONTEXT: instrumentation points, performance metrics,app meta data, etc.
SoftwareConstruction& Instrumentation
Business, Application, & Environment Context
business context, KPIs, SLAs, etc.
performance metrics
application meta-data
Packaging
Deploy to Production
OVO
OVTA
OVIS
OVOOVO
OVTA
OVIS
Manage in Production
update
OVO
OVTA
OVIS
OVOOVO
OVTA
OVIS
OVO
OVTA
OVIS
OVOOVO
OVTA
OVISApplication Bug: help development find and fix
erroroccurserroroccurs
use context to help development find and fix bug.
Testing/ Staging
how do the components interrelate with each other…
knowledge base of stress, expected loads, etc.
configure
update
Functional Requirements& Contracts + SLAs
create
track
monitor
controltrack
monitor
control
track
monitor
control
Start Here
Roles:Who Manages Apps in Production?• Application “Owners” (business sponsors)
– Customer Management– Renegotiate Contracts– Measure/Monitor Compliance with SLAs
• IT/Operations (traditional, platform management)– Ensure health of hardware, OS, some middleware– 1st line of defense when problems occur
• Application Managers (newer concept)– Ensure health of application containers and custom applications– 2nd line of defense. Can often alter/control/repair without needing dev support.
• Software Developers– Last line of defense.– When all else fails, they use all runtime context to debug & repair problems
Market Observations & Trends – Gartner
0
20
40
60
80
%
Approach
Preferred Technology Approach
Best of breed
Single vendor
Outsource
Other
0
10
20
30
40
%
Group
Responsibility for app
management
Appdevelopment
IT operations
App support
LOB
outsourcer
other
0
5
10
15
20
25
30
%
Type
Most management need
Custom
ERP
J2EE
CRM
other
• Best-of-breed is most popular strategy. • Application development & support are most
often responsible for application management.• Custom applications are most need of
management• “Gartner believes …that responsibility for
application management increasingly resides outside of the traditional IT operations organizations”.
Gartner “Application Management Poll Reveals New Trends” April 15, 2004
BSM AM
Whom HP OpenView Application Management Targets Today
CIO
App Manager
LOB Owner
App Dev App Support
IT Ops Manager
Network Mgr
ServiceDeskMgr
Biz/IT Liaison
Bus ProcOwner
IT Controller
Infra Svcs Manager
ServerMgr
App QA
LOBFinance
Software Developer
Tools: “Enterprise debugger” IDE Architecture Models
Application Owners (business sponsors)
Tools: “BPM/BAM Dashboards” CRM/Service Desk
Operations/IT
Tools: Performance Monitoring Operational Status Monitoring Infrastru. Troubleshoot Tools
Application Management
Tools: Application Troubleshooting (JMX info, etc.) Service Management Tools
Business,Application & Environment Context
enterprise “coredump”: call stack traces performance info audit trail
business runtime: SLO’s, KPI’s, etc. business processes
operational runtime:mgmt events, infra. “health”, etc.
application runtime: JMX, WMI transaction traces, etc.
Management Tools for Operations
3 Pillars of Management
• Tracking– observe aspects of a single thread across multiple components. – technologies: logging & auditing
• Monitoring– capture runtime and historical events of a particular component, for the
purpose of reporting and notifications.– technologies: Real-time statistics (JMX), logging, & auditing
• Control – alter the behavior of the managed component directly without recourse to
interrupt the operation of the component.
– technologies: JMX, custom techniques
What is “Tracking?”• Measurement of “operational performance”• Combination of JVM profiling & application logging• Often is “durable” recording of what occurred.• Like “frames in a movie”• Some tools could “play back” the recording.• Helpful during testing & staging
What is “Monitoring?”• Measurement of “business performance” (SLAs, KPIs)• Measurement of current “operational health” and “business health”• Much “context” (operational & bus) only known by applications: so use application
logging & JMX• Some is “durable” recording of what occurred.• Some is transient or “current snapshot”• Management Events & Corrective Triggers
What is “Control?”• Automated changes or repair of application.• Manual changes or repair to application.• Often in response to management events “fired” during monitoring.• Sometimes modifies operational aspects of applications.• Sometimes modifies business aspects of applications.• Integral to building an adaptive enterprise and adaptive applications.• J2EE Applications often expose JMX MBeans for this purpose.
Management Products for “Tracking”
• HP OpenView Transaction Analyzer • HP OpenView Internet Services – measures availability (and response-time) by creating
synthetic transactions (aka “robots”) • Borland ServerTrace• Actional (web services)• JVM-level Profiling Tools• Etc.
Management Products for “Monitoring”
• HP OpenView for Operations, Reporting, Graphing• HP Service Desk• HP OpenView Internet Services + OVTA (real end user response times and volume metrics, essential for SLA
monitoring -- e.g. response-time < 5 secs AND volume < 10,000 TPS)
• BMC• CA Unicenter• Microsoft Operations Manager & Performance Monitor• Actional (web services)• Etc.
Management Products for “Control”
• HP OpenView for Operations • BMC• CA Unicenter• Microsoft Operations Manager• Etc.
Instrumenting Your Applications For The Operational Environment
What should you manage?• Think in terms of an application’s “management interface”
• Use SLAs, business requirements, & IT/operations needs to help you decide what management to support
• Management is a combination of what can be done at the platform (JVM, J2EE Containers, etc.) & what you need to code
• You only need to code context that isn’t available from the platform: (business-specific context, operational statistics not gathered by platform, etc.)
managementrequirements
starting-pointmanagement“best practices”Solution Architects,
Industry Trends, etc.
operational management
new requirementslearned over timethrough experience
operational management
IT Operations
contracts with end users, servicelevel agreements,etc.
business management
CustomersMarketing, Analysts, Architects
Solution (business logic, data, etc.)
Management Interface of Solution
management tools
IT Operations
managementrequirements
communicate…
are specifications for
are specifications for
manage…
determine “dials” and “knobs” for mgmt
functional &systems
requirements
“HR
” A
pp
Managem
ent
Inte
rface
s
“Call
Cente
r” A
pp
Managem
ent
Inte
rface
s
IT
ManagementSolution(s)
TrackingWhat To Do:
– Logs and trace for method “entry” and “exit.”– Performance metrics (at JVM-level as well as app container
level)– These are like frame-by-frame recordings of program
executions. They are durable
Available Technologies & APIs:– JDK 1.4 or later: use standard java logging– JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport )– JVMTI (JVM Tooling Interface) for tracking/monitoring tools
MonitoringWhat To Do:
– Logs and trace for operational as well as “business” statistics.
– Business audit logs for historical business data (store exchanged business documents, business events, etc.)
– JMX and WMI for application instrumentation– Performance metrics (at JVM-level as well as app
container level)
Available Technologies & APIs:– JDK 1.4 or later: use standard java logging– JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport )– JMX (MBeans) – “custom” MBeans as well as container-provided
ControlWhat To Do:
– react to predictive “triggers” and actually change running servers & code
– modify the behavior of servers & code “in flight” without shutdown– solve application problems/issues without always requiring
developers
Available Technologies & APIs:– Custom-coded “control point” management objects using
JMX (MBeans)– Custom-coded “control points” using: config files, web
services, RMI, etc.– Container-level “control points” accessible via JMX,
config files, etc.– React to change of state of “control point” objects
Best Practices for coding mgmt interfaces
• Use JMX for operational and business “health” & as “control points”
• Use JMX to provide “rollups” such as:– averages over time– business-specific stats: # of “high value”
customers this hour– current operational load on system– current business activity rollup: $ processing right
now, etc.
• MBeans are for monitoring & “control”: Change their state to control the application.
• Could provide averages over time: “transactions per hour”, “high value customers served today”.
• But!, better to put these calculated metrics outside of the application and outside of MBeans:– Management system configurations (OVO templates)– Management “rules” engines, event engines, etc.
• Provide simple, atomic data through JMX. Aggregate it externally.
Averages & Roll-Ups Belong Outside of application’s source
Don’t count on “native” JMX Management Events
• With JMX you can subscribe to and generate events.
• JMX events are “seen” within the JVM. But aren’t sent outside to the management system.
• Different management products have their own event mechanisms. And WSDM (Web Services Distributed Management) standard has another.
• If you will be in a heterogeneous environment with many technologies, don’t use JMX or WBEM notifications.
• Consider not using “native” JVM management events
public interface MoviePreviewStatsMBean { public int getPreviewDownloadsCount (); } public class MoviePreviewStats implements MoviePreviewStatsMBean, java.io.Serializable { public MoviePreviewStats () {} public int getPreviewDownloadsCount () { return m_nPreviewDownloadsCount; }
private int m_nPreviewDownloadsCount = 0 protected void resetPreviewDownloadsCount () { m_nPreviewDownloadsCount = 0; } protected void incrementPreviewDownloadsCount () { m_nPreviewDownloadsCount++; } }
A Very Simple! Standard MBean
Standard MBeans– Useful when the structure of managed data is well defined and unlikely to change– Named with “MBean” added on the interface– Attributes with getter/setter methods– When to use: When first using MBeans; when the management interface isn’t changing much.
Dynamic MBeans– must implement “DynamicMBean” interface– is like DII in CORBA – your MBean doesn’t need a method signatures that match how the outside world “sees” it– You supply the “MBeanInfo” meta data for the bean– When to use: When you want a level of indirection between the signature of your MBean and the actual implementation; When impl side is dynamic (at runtime as well as design often changing).
Model MBeans– (A subtype of dynamic MBeans)– “RequiredModelMBean” always available and implements the “ModelMBean” interface– The infrastructure implements most of the state caching, methods, etc. you would normally write yourself.– When to use: when you want to configure MBeans from outside of the source code (like from XML file)
Different “Flavors” of MBeans
Publishing Your MBeans
• If you want the outside world (management systems) to see your MBeans, you “publish” them to an MBeanServer.
• Each MBean that is published is like a “pointer” to an actual Java object.
• Consumers of MBeans never “touch” the actual MBeans. Instead they ask the MBeanServer to talk to the MBeans.
• Some Best Practices:– As often as possible, use the standard JMX API for this (for
instance, BEA lets you use an MBean EJB)– Use unique “domain” names for each application.
Code Sample for Publishing an MBean/**
* Create the MBean. Our application will be responsible for
* updating its value throughout the running of the application. */
Employee employeeMBeanRef = new Employee ();
/**
* Access the MBeanServer. */
MBeanServer mbeanServer = MBeanServerFactory.createMBeanServer();
/**
* Create “object name” for Bean. Publish it into the MBeanServer, using
* "HR_domain" as the domain part of the MBean's name attibute. */
ObjectName beanName =
new ObjectName("HR_domain:Name=Employee&type=manager");
mbeanServer.registerMBean( employeeMBeanRef , beanName);
Demonstration of Application Instrumentation Tools
J2EE Container
your application
MBeans(the managementinterface)
HP OpenView for Operations
SPI Configuration:user defined metrics
JMX SPI
JMX Metric Builder
edit configuration file
Container MBeans
Call To Action
• Think about app instrumentation “holistically” rather than as some APIs. (Think about the “management requirements.”)
• Work within a development lifecycle that embraces “manageability built in” as fundamental to success
• Determine your management requirements and management interfaces up front.
• Build your applications to support: tracking, monitoring, & control.
• Use technologies such as JMX, web services & SOA to help you create flexible & robust applications for the adaptive enterprise.
Questions