best practices for performance tuning jboss enterprise ... · pdf filebest practices for...

www.jboss.com

2 In search of hIgh-performance applIcatIons

2 performance tunIng prIncIples

2 Why tune for performance?

2 Performance priorities

3 A word about performance benchmarks

5 JBoss eap 5.0 tunIng

5 Connection pooling

6 Thread pooling

8 Object and component pools

9 Logging

9 Caching

11 JBoss EAP performance tuning summary

11 Linux-specifictuning:Largememorypages

12 Linux-specifictuning:Tuningthevirtualmemory manager

12 Database tuning update

12 Case study

17 applIcatIon performance tunIng: a contInuous process

18 appendIx: usIng large-page memory (lInux-specIfIc InstructIons)

Best practices for performance tuning JBoss enterprIse applIcatIon platform 5 tIps and trIcks for optImIzIng your applIcatIon’s performance

2 www.jboss.com

in search of high-performance applications

Overall,performancetuningisaveryimportantpartofcreating,maintaining,anddeployingasuccessfulbusinessapplication.Whetheryouarebuildingcustomapplicationsordeployingcommercial,off-the-shelfsolutions,youwilllikelyneedtotunetheapplication,thedatabase,themiddleware,orallthree.Infact,75%of performance issues originate with the application itself.

Whenorganizationsselectapplicationmiddleware,performanceisalwaysoneoftheirmostimportantselec-tioncriteria,ifnotthemostimportant.InmanycasesourcustomerstellustheychoseJBossEnterpriseApplication Platform (EAP) because of its superior performance. They know that many users of JBoss EAP areachievingsuperiorapplicationperformancedayafterday.

Togetthemostfromyourcompany’sinvestmentinmiddleware,developersandarchitectsneedtoknow thespecificwaystheycanachievesuperiorperformancewithJBossEAP.Whilewewouldallliketothinkthatanapplicationcouldperformwellstraightoutofthebox,thisisnotusuallythecase.Applications canhavewidelyvaryingcharacteristics,andwhilesomeapplicationsmightperformwellwithdefault middlewaresettings,otherswillnot.

IfyouarenewtoJBossEAPorperformancetuning,thispaperwillintroduceyoutobestpracticesthatcanhelpyouavoidcommonperformancepitfallsasyouprepareyourapplicationforproduction.Ifyou’reanoldhandatapplicationperformanceissues,youknowthattechnologyisconstantlychanging.Youmaybenefitfrom an update on best practices in JBoss EAP performance tuning.

performance tuning principles

Why tune for performance?

Performance was once considered just another feature of an application. Today it is frequently consid-eredthemostimportantcharacteristicoftheapplication—onethatcanhaveasignificantimpactonyourbusinessandyourproductivity.Consideryourreactiontoaslowwebsite.Ifyou’relikemostpeople,youbecomefrustrated,losepatience,andgoelsewhere.IfthatcompanyiscountingonrevenuefromWebsales,ithasnotonlylostyourattention—ithaslostbusiness.Evenforinternalapplications,poorperformancecanaffectproductivityifusershavetowaitordealwithunpredictablebehavior.Atbest,theymightfeelannoyed,losealittletime,orformanegativeopinionoftheirITdepartments.Atworst,businesstrans-actionsmaybelost,orcustomersmaygowithoutimportantserviceifusersmustworkaroundapoorlyperforming application.

But user experience is not the only reason to tune for performance. A well-performing application will generallyusefewerhardwareandsoftwareresources.Acompanycanoptimizeitsinvestmentinhardwarewhenapplicationsaretunedappropriately,whetherthatmeansusingoldersystemslonger,purchasingnewsystemsthataremoremodestlysized,orusingfewersystemsoverall.Onthesoftwareside,awell-performingapplicationwillgenerallyneedtousefewerCPUcountsorsoftwarelicenses,nomatterwhattypeofsoftwareisinvolved.Reducingsoftwarecostscansavethecompanysignificantmoneyovertime.

performance prIorItIes

Superiorperformancecomesfrommanylayersoftheapplicationstack,notjusttheapplicationserver.Inmanycases,thewaytheapplicationisdesignedandhowitconnectstothedatabaseandothersoftwarecomponentscanhavealargeimpactonoverallapplicationperformance.Manyorganizationsspendmore

BestpracticesforperformancetuningJBossEnterpriseApplicationPlatform5.0

www.jboss.com 3

timetuningtheircustom-builtapplicationsanddatabasesthantheirunderlyingapplicationservers.Sokeepinmindthatasuperiorlyperformingapplicationservermayhaveonlyaminorimpactontheoverallperfor-mance of your applications.

a Word aBout performance Benchmarks

Some organizations rely on industry-standard performance benchmarks when selecting middleware. While benchmarkscanhelpvendors,theycandeceiveyouasanevaluatorbecausebenchmarkapplicationsareusuallyverydifferentfromtheapplicationsyouwillruninproduction.Thecurrentbenchmarkforapplica-tionservers,forexample,SPECjAppServer2004,isa2004applicationthatdoesn’ttakeadvantageofmanyrecentdevelopmentsinJava.Butasidefromthat,everyapplicationisdifferentandthesystemsonwhichbenchmarkapplicationsarerunmaybeverydifferentfromyours.

Understandinghowagivenbenchmarkrunsmaytellyouverylittleabouthowyourapplicationwillrunonyourhardwarewithyoursettings.Beforeyourelyonabenchmark,youneedtoinvestigatethesoftwareandhardwareconfigurationsused,theserverandnetworkconfigurations,thesettingsused,thearchitectureoftheapplication,andhowallofthesecomparewithyourownenvironment.Werecommendthatbeforeselectingasystem,youtestitwithanapplicationandhardwareconfigurationasclosetoyoursaspossible.1

performance tunIng prIncIple #1: understand your performance requIrements

Thefirststepintuningyourapplicationforperformanceistounderstandtheconditionsunderwhichitwillneedtoperform.Ifyourapplicationisareplacementforanexistingsolution,thenyourorganizationalreadyhassignificantexperiencewiththatsolution.Chancesareyouhavemetricsavailablesuchasthenumberofusers,thenumberoftransactionsperday,thevariationsintransactionloadortypeoverthecourseofaday,week,month,oryear,andsoon.

Ifyouaredeployingacompletelynewsolution,youwillneedtostudythatapplication’sbusinesscontextverycarefully.Themoreyouunderstandexactlyhowyourapplicationwillbeused,themoresuccessfulyourperformancetuningwillbe.Sometimesassumptionsdonotreflectreality.Inonecase,adevelopmentteamcreatedandtesteditsapplicationbasedonanassumedworkloadof60,000transactionsperday.Thefirstdayinproduction,sixmilliontransactionsoccurred.

performance tunIng prIncIple #2: plan for peaks, not averages

Asyouexamineperformancerequirements,oneofyourgoalsshouldbetodevelopaprofileofyourapplica-tion’sworkloadwithspecialattentiontothepeaks.Forexample,manybusinessapplicationsexperiencedailypeaksinthemorningandafternoonwithavalleyduringthemiddleofthedaywhenemployeesareeatinglunch.Someapplicationsexperiencepeaksattheendofthemonth,quarter,orseason.Yourapplication’sworkloadprofilewilldependonthespecificsofyourbusiness,butyoushouldalwayspayparticularattentiontoperiodsofpeakworkload.Oneofthebiggestmistakesdevelopersmakeistorelyonanaverage(averagedailyworkload,forexample).Averagesarenotsufficienttoensurethatyourapplicationwillperformduringperiods of peak load.

1 Foramorein-depthexplanationoftheprosandconsofperformancebenchmarks,seeDoPerformance Benchmarks&ComparisonsMatter?—AGuidetoAssessingApplicationServerPerformanceResults.


4 www.jboss.com

performance tunIng prIncIple #3: alWays Instrument your applIcatIon

Allapplicationsshouldbeinstrumentedtoprovideinformationforperformanceanalysis.Businesscondi-tions,includingcustomerbehaviorandworkloadcurves,canchangedramaticallyovertime,soevenifanapplicationonceperformedwell,itmaynotperformwellundertoday’sconditions.Ifyourunintotrouble,andyourapplicationhasnotbeeninstrumentedforperformance,thenyouhavenoeasywaytoknowwhereyourproblemsare.Butifyourapplicationisappropriatelyinstrumented,thenyou’llbeabletomonitorchanges in business conditions easily and tune your applications to match before problems occur.

Inthepast,adeveloperneededtoembedcodewithintheapplicationitselfinordertoinstrumentanappli-cation.Todaymanysolutionsprovideinformationwithoutrequiringdeveloperstocode.WithJBossEAP,thecallstatisticsinthecontainerprovideyouthenumberofcalls,concurrentcalls,mincalltime,maxcalltime,averagecalltime,andthestandarddeviationofcalltimes.Hibernatestatisticsprovidequeryexecu-tiontimes.ThenewJBossEAP5AdminConsole,aswellasandJBossOperationsNetwork(JBossON)Monitoringcandisplayallofthisinformationandgraphtheresultsinrealtime.Additionally,numerous third-partyapplicationperformancemanagement(APM)productsareavailablefromcertifiedRedHatpartnerstoprovideapplicationperformanceinformationaswell.

Finally,forperformance-criticalsituationswhereoneoftheothertoolsdoesn’tprovidetheinformationyouneed,youcanwriteyourowninstrumentationusingtheJBossAOPframework.Thisframeworkoffersquitea few features that enable you to see exactly what is happening at runtime. Whether you choose to write yourowninstrumentationwilldependontheapplication,yourskillsandavailabletime,andtheimportanceoftheapplication’sperformancecharacteristicstoitsoverallsuccess.

performance tunIng prIncIple #4: understand Where your applIcatIon spends Its tIme

One important reason to instrument your application is to understand where time is being spent. While you wanttoknowabouttimeacrosseachlayerofyourapplicationstack,thetoolswithinJBossEAPwillhelp youunderstandonlypartofthisequation.Ifyourapplicationisspendingtoomuchtimeinthedatabase, forexample,thenyoumayneedtofocusonyourdatabasestatisticstopindowntheproblem.

Byunderstandingwhereyourapplicationspendsitstime,youwillbeabletoavoidthe“shotgunmethod”of performance tuning — trying multiple solutions to common problems without knowing whether any of themarerelevanttoyourproblem.Youmightsolveyourproblemthisway,buttheprobabilityislow.Manydeveloperswhoassumetheyknowwhatiscausingtheirperformanceproblemsaremistaken,resultinginproblemsthatpersistformonthsorsometimesyears.Thatiswhyhavinganobjectivewaytoknowwhereyourapplicationisspendingtimewillhelpyouavoidspendingtoomuchtimesolvingperformanceproblems.


www.jboss.com 5

performance tunIng prIncIple #5 replIcate or model your productIon envIronment

Asyouworktounderstandexactlyhowyourapplicationwillperforminproduction,youwillneedtorunitinatestenvironment.Somecompaniesmakeitstandardpracticetoimplementtestenvironmentsthatareexactreplicasoftheirproductionenvironments.Thisisanidealapproachinmanyrespects,becausedevel-opersdon’thavetowonderhowtheirapplicationswillscaleforlargersystems.

Formanyorganizations,though,thecostofreplicatingtheproductionenvironmentisasignificant obstacle.Theirtestenvironmentstypicallyemploysmallersystems.Ifthisisyoursituation,thenyouwillneedtocreateamodelthatdefinestherelationshipbetweenperformanceinyourtestenvironmentandperformanceintheproductionenvironment.Thatis,youneedtomodelhowtheapplicationwillscale.Asyoudoso,keepthefollowinginmind:

do not assume a linear relationship.• Performancedoesn’toftenscalelinearly.Whilevendors maytellyoutheyexperiencelinearresults,thisisrarelythecaseforrealproductionapplications.

Be conservative in creating your model, and use actual historical data wherever possible.• Ifyou canbaseyourmodelondatafrompastexperiencewithotherapplications,youwillbeabletorefine itovertimeandincreaseyourconfidenceinitsaccuracy.

JBoss eap 5 tuning

While75%ofallperformanceproblemsaretheresultoftheapplication,notthemiddlewareorthe operatingsystem,youstillneedtoknowhowtotunesettingsinthemiddlewaretoimproveperformance andthroughput.Dependingonyourapplication,oneormoreofthefollowingmayneedsomeattention:

Connection pooling•

Thread pooling•

Object and component pools•

Logging•

Caching•

Thefollowingsectionsprovideanoverviewofeachoftheseareas.

connectIon poolIng

Connection pooling and thread pooling are the most important areas to consider when you want to maximize throughputonmodernhardware.Intermsofsystemresources,databaseconnectionsareexpensivetosetupandteardown.Butinspiteofthis,someapplicationscreateanewconnectiontothedatabasewitheveryqueryortransactionandthenclosethatconnectionimmediately.Thispracticeaddsagreatdealofoverheadtotransactionprocessingandcanleadtopoorperformance.

TotakeadvantageoftherobustconnectionpoolinginJBossEAP,startbyadjustingconnectionpoolsettingsonthedatasourcedefinitionsthatyoucansetupinthedeploydirectory.Settheminimumpoolsizetothelevelyouwanttotunefor,thensetthemaximumatleast25-30%higher.Don’tbeconcernedaboutsettingthemaximumtoohigh,becauseifyoudon’tneedthatmanyconnections,thepoolwillshrinkautomatically.


6 www.jboss.com

Todeterminethepropersizing,youcanmonitoryourconnectionusage.Apoolthatistoosmallwillthrottletheapplication,asJBossEAPwillqueuetherequestforadefaultof30,000milliseconds(or30seconds)beforegivingupandthrowinganexception.Ifyoustartseeingalotof30-secondtimeouts,thatisastrongcluethatyouneedtolookatyourconnectionpooling.YoucanmonitortheconnectionpoolutilizationfromtheEAPJMXconsole,fromJBossON,orfromdatabase-specifictools.

Hereisanexampleofconnectionpoolsettingsforadatasource:

<datasources> <local-tx-datasource> <jndi-name>MySQLDS</jndi-name> <connection-url>jdbc:mysql://[host]:3306/[database]</connection-url> <driver-class>com.mysql.jdbc.Driver</driver-class> <user-name>someuser</user-name> <password>somepassword</password> <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor. MySQLExceptionSorter</exception> <min-pool-size>75</min-pool-size> <max-pool-size>100</max-pool-size> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <prepared-statement-cache-size>100</prepared-statement-cache-size> <shared-prepared-statements>true</shared-prepared-statements> </local-tx-datasource> </datasources>

thread poolIng

Thread pooling is the next most important area to consider as you tune your application for performance. JBossEAPhasrobustthreadpooling,butbeforeyoucansizethethreadpoolsappropriately,youneedtoknow how they are used and which ones might be affecting your application’s performance. The character-isticsofyourspecificapplicationwilldeterminewhichthreadpoolsareusedandwhichonesmightbecomebottlenecks.Thiscanvarysignificantlyfromapplicationtoapplication.Thetablebelowprovidesasummaryof how each thread pool is used.


www.jboss.com 7

removIng connectors

Ifyouarecertainthataparticularconnectorwillnotbeusedinyourapplication,considerremovingit. Forexample,manyapplicationsuseeitherthehttpdthreadpoolorthemod_jkthreadpoolbutnotboth. Soyoucanremovetheoneyoudon’tneed.monItorIng and tunIng thread pools

MonitoryourthreadpoolsviatheEAP5AdminConsole,whichdisplaysnotonlythenumberofactivethreadsforeachpoolbutalsothequeuesize.InboththeEAP5AdminConsoleandJBossON,youcanadjustanddefinethreadpoolsettingsthatpersist.SettingsadjustedviatheJMXconsolewillnotsurvive areboot.IfyouusetheJMXconsole,remembertogobackandeditthefileifyouwanttheadjustments to be permanent.

thread pool Where is it defined? hoW is it used?

System thread pool Injboss-service.xmlintheconfdirectory ForJNDInaming—thedefaultsettingisfineformost cases

HTTPdthreadpool in JBoss Web

Intheserver.xmlfileunder<server>/deploy/jboss-web-sar.

WhenmakingHTTPrequestsdirectlytoEAP

AJP thread pool Intheconnectorsectionofserver.xml WhenmakingHTTPrequeststhroughmod_jkormod_cluster

JCA thread pool (also called the WorkManagerthreadpool)

Definedin<server>/deploy/ jca-jboss-beans.xml

InconjunctionwithJMS,asJBossMessagingusesJCAinflowastheintegrationintoEAP

JBossMessagingthreadpool (for remote clients)

Definedin<server>/deploy/messaging/remoting-bisocket-service.xml

Pools the TCP sockets

JBossMessagingthreadpool (inJVMclients)

All the processing will occur on the JCA thread pool(WorkManager)

Notethatifyouhaveamessagedrivenbean,thatinvokesotherbeans,suchasstatelesssessionbeans,thosebeanswillalsorunontheJCA thread pool.

EJB3(sameJVM) ClientsinthesameJVMwillrunonwhateverthread pool they are already using

Forexample,awebrequestcomesinthroughtheAJPconnector.WhenitcallsanEJB3bean,it will continue executing on the AJP connector thread pool.

EJB (remote clients) <server>/ejb3/deployer/META-INF/ jboss-service.xml


8 www.jboss.com

example: http thread pool

<Connector port=”8080” address=”${jboss.bind.address}” maxThreads=”250” maxHttpHeaderSize=”8192” emptySessionPath=”true” protocol=”HTTP/1.1” enableLookups=”false” redirectPort=”8443” acceptCount=”100” connectionTimeout=”20000” disableUploadTimeout=”true” />

example: mod_Jk or aJp thread pool

 <Connector port=”8009” address=”${jboss.bind.address}” protocol=”AJP/1.3” emptySessionPath=”true” enableLookups=”false” redirectPort=”8443” maxThreads=”200” />

example: Jca thread pool

 <bean name=”WorkManagerThreadPool” class=”org.jboss.util. threadpool.BasicThreadPool”>  <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name=”jboss.jca: service=WorkManagerThreadPool”, exposedInterface=org.jboss.util.threadpool.BasicThreadPoolMBean.class)</annotation>  <property name=”name”>WorkManager</property>  <property name=”maximumQueueSize”>1024</property>  <property name=”maximumPoolSize”>100</property>  <property name=”keepAliveTime”>60000</property> </bean>

oBJect and component pools

Object pools and component pools are essentially the same thing. Their settings represent the number of objectinstances.ForEJB3,twotypesofpoolsaredefinedin<server>/deploy/ejb3-interceptors-aop.xml.ThesearetheThreadLocalPoolandtheStrictMaxPool.Bydefault,StatelessSessionandStatefulSessionBeansusetheThreadLocalPool,whichisbackedbyanInfinitePoolwithnomaximumsize.Therefore,itgrowsaccordingtovolumeinyourapplication.Thishasthedistinctadvantageofnotneedingtobetuned.Bydefault,MessageDrivenBeans(MDBs)usetheStrictMaxPool.Thispoolactuallyobeysamaximum,willqueueuprequestswhenthatmaximumhasbeenreached,andwilltimeoutanythinginthequeueifthereisnotanavailablereferencefromthepool.Inthiscase,thesystemwillthrowanexceptionandiftheproblemoccurredinmid-transaction,youwillexperienceatransactionrollback.Giventheimpactfailedtransactionscanhaveonyourbusiness,youshouldmonitortheStrictMaxPoolcloselyviatheJMXconsole.

www.jboss.com 9

loggIng

Developersshouldtakefulladvantageoflogginginthedevelopmentandtestingphasesoftheapplicationlifecycle.Inproduction,however,loggingcancausebottlenecks.Youwanttobesurethatloggingprovidesyou with useful information without hurting application throughput. Consider making the following changes asyoupromoteyourapplicationintoproduction:

turn off console logging in production.• InJBossEAP’sdefaultconfiguration,consoleloggingisenabled,whichmeansyouhavetheopportunitytoseeallthelogsfromyourIDE.Inproduction,thisisanexpen-siveprocesswithunbufferedI/O.Whilesomeapplicationsmaybefinewithconsolelogging,high-volumeapplicationsbenefitfromturningitoff.JBossEAP5providesaconfigurationset,namedProduction,whichgivesdevelopersabetterstartingpointforcreatingaproductionenvironment.IntheProductionconfiguration,consoleloggingisturnedoff.

turn down logging verbosity.• Thelessyoulog,thelessI/Owilloccur,andthebetteryouroverallappli-cationthroughputwillbe.Loggingisalwaysatradeoff,sothinkcarefullyabouthowmuchloggingyoureally need in production.

use asynchronous logging.• This can make a big difference for high-throughput applications. With asynchronouslogging,logmessageswillgointoaqueueandcontrolreturnstotheapplicationasif the logging had been completed. Then a separate thread executes the log operations from the queue.

Wrap debug log statements with• If(debugEnabled()). This simple practice can make a huge differ-enceifyourapplicationcontainsalotofdebuglogstatements.Withoutthisconditionset,yourapplica-tioncreatesallofthestringobjectsforeachofthelogstatements,andLog4jcreatestheLoggingEventobjectforeachlogstatementregardlessoftheloglevelthatissetbecausetheloglevelischeckedonlyafteralloftheseobjectshavebeencreated.InsomecasesthiscanleadtocreationofthousandsandthousandsoftemporaryStringandLoggingEventobjects,resultinginmemoryandgarbagecollectionissues and reducing throughput dramatically. By placing a conditional wrapper around your debug log statements,youcanensurethatunnecessarylogprocessingdoesnotaffectyourthroughput.

cachIng

Caching,whileoftenveryhelpful,maybeoneofthetrickiestareastotunecorrectly.JBossCacheisanintegralpartoftheJBossEAPinfrastructure,andyourapplicationcanuseittocacheanythingyoulike.Cachingisespeciallyvaluableforapplicationsthatperformalotofreadoperationsagainstdatathatiseithercompletelystaticordoesn’tchangefrequently,suchasreferencedata.Forapplicationswithheavyuseofwriteoperations,cachingmaysimplyaddoverheadwithoutprovidinganyrealbenefit.Ifnotconfig-uredproperly,cachingmayactuallyreducethroughput.

OneoftheeasiestpotentialperformanceenhancementsyoucanmakeistocacheEJB3entities.Todefinewhichentitiesyouwantcached,modifythefilepersistence.xml that you deploy with your EJB 3 applica-tion(anexampleisshownbelow)[email protected]@Cachedannotation,youspecifytheusageasbeingoneofthefollowing:

CacheConncurrencyStrategy.READ_ONLY, READ_WRITE, NONSTRICT_READ_WRITE, or TRANSACTIONAL.


10 www.jboss.com

Youdefinethecachesizeandevictionpolicyinthefile

<server>/deploy/cluster/jboss-cache-manager.sar/META-INF/jboss- cache-manager-jboss-beans.xml.

Thesedefinitionsmayrequiresometrialanderrortogetright.Rememberthatyouareworkingwithlimitedheapspace,andthatamisguidedcachingstrategycanbeworsethannoneatall.

Evictionpolicywilldependonthespecificsofyourapplication.Youcandefinethecacheastransactional orread-write.Withverylargedatasets,notethatcachingmaynotprovidenoticeableperformancebenefits.Inthiscase,shrinkingthecachescanimproveperformance.Also,ifyourapplicationisverywrite-heavy,itmaynotbenefitfromcaching.Testingvariouscachingandnon-cachingconfigurationswillhelpyoudeter-mine this.

example: settIngs for cachIng In persIstence.xml

<persistence xmlns=”http://java.sun.com/xml/ns/persistence” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd” version=”1.0”> <persistence-unit name=”services” transaction-type=”JTA”> <provider>org.hibernate.ejb.HibernatePersistence</provider> <jta-data-source>java:/MySQLDS</jta-data-source> <properties> <property name=”hibernate.default_catalog” value=”EJB3”/> ... <property name=”hibernate.cache.region.factory_class” value=”org.hibernate.cache.jbc2.JndiMultiplexedJBossCacheRegionFactory”/> <property name=”hibernate.cache.region.jbc2.cachefactory” value=”java:CacheManager”/> <property name=”hibernate.cache.use_second_level_cache” <property name=”hibernate.cache.region.jbc2.cfg.entity” value=”mvcc-entity”/> <property name=”hibernate.cache.region_prefix” value=”services”/> ... </properties> </persistence-unit> </persistence>

Notethatthereareactuallyfourdifferentcacheregionfactoryclasses.BesidesJndiMultiplexedJBossCacheRegionFactory,thereisalsoMultiplexedJBossCacheRegionFactory,SharedJBossCacheRegionFactoryandJndiSharedJBossCacheRegionFactory.Youshouldconsult theHibernatedocumentationaboutthedifferencesbetweenthethree.

InadditiontoJBossCache,anotherformofcachingthatcanbenefitmanyapplicationsispreparedstate-mentcaching.Preparedstatementcachingcanbesetinyourdatasourceconfiguration(seeprevioussectiononConnectionPooling).Bythissimplechange,itispossibleforsomeapplicationstoexperienceasignificantimprovementinthroughput.Inthesameexampledatasourcedefinitionusedbefore,thepreparedstatementcachingparametersappearinbold:


www.jboss.com 11

<datasources> <local-tx-datasource> <jndi-name>MySQLDS</jndi-name> <connection-url>jdbc:mysql://[host]:3306/[database]</connection-url> <driver-class>com.mysql.jdbc.Driver</driver-class> <user-name>someuser</user-name> <password>somepassword</password> <exception-sorter-class-name>org.jboss.resource.adapter.jdbc. vendor.MySQLExceptionSorter</exception> <min-pool-size>75</min-pool-size> <max-pool-size>100</max-pool-size> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <prepared-statement-cache-size>100</prepared-statement-cache-size> <shared-prepared-statements>true</shared-prepared-statements> </local-tx-datasource> </datasources>

Asalways,itisrecommendedthatanalysisbeconductedwithinyourenvironmenttodeterminehowlarge tomakethepoolsizeasthiswilldependonthepreparedstatementsthatarespecifictoyourapplicationand data source.

JBoss eap performance tunIng summary

Tosummarize,keepthefollowingperformancerecommendationsinmindwhentuningJBossEAP5.0.

DefinedatasourcesinthedeploydirectoryandtakeadvantageofEAP’srobustconnectionpooling.•

Understandwhichthreadpoolsareactuallyusedbyyourapplication,removepoolsthatarenot •needed,monitorthenumberofactivethreadsandthequeuesize,andadjustpoolsizeifneeded.

Ifyouareusingmessage-drivenbeans,monitortheStrictMaxPoolcloselytoensurethatthemaximum•object pool size is not a bottleneck.

Employdifferentloggingstrategiesfordevelopmentandproduction.Especiallyforhigh-volumeappli-•cations,turnconsoleloggingoff,reduceloggingverbosity,useasynchronouslogging,andalwayswrapdebug log statements with If (debugEnabled ()).

Takeadvantageofcachingifyourapplicationisread-heavyormakessignificantuseofreferencedata.•Avoidcachingforverylargedatasetsorwrite-heavyapplications.

lInux-specIfIc tunIng: large memory pages2

Fortoday’s64-bitsystems,useoflargememorypagescanimproveperformancesignificantly.Thedefaultmemorypagesizeistypically4kB.Whenyouareaddressinglargeamountsofmemory,thisquicklyadds uptoalargenumberofmemorypages—justonegigabyterequires262,144memorypages.That’salotfor asystemtokeeptrackof,whichtranslatestoalotofsystemoverhead.

2 WhilethissectionprovidesLinux-specificsettings,theprinciplesoflargememorypageuseareapplicabletoany64-bitsystem.


12 www.jboss.com

Asidefromhelpingyouavoidtheoverheadofmappingsomanymemorypages,largememorypagesonLinuxcannotbeswappedtodisk.Thisisamajoradvantagebecausehavingyourheapspaceswaptodiskcanwreckhavocontheperformanceofyourapplication.

Largepagesupportbeginsat2MBandcanrunashighas256MBonsomehardwarearchitectures.Thesenumberswillvary,andyouwillneedtofindoutthevaluesforyourspecificserver.AllthemajorJVMsystemssupportlargememorypagesonLinux.Becauseitcanbetrickytosetup,weprovidesomespecificinstructions in the appendix.

Whenyouuselarge-pagememory,keepinmindthatthememoryisnotavailabletoapplicationsingeneral.Tootherapplications,yoursystemwilllookasthoughithashadmemoryremovedfromit,becausethismemorywillbededicatedtoyourspecificapplication.Refertotheappendixformorespecificinformationaboutconfiguringyourapplicationtouselargememorypages.

lInux-specIfIc tunIng: tunIng the vIrtual memory manager

InLinux,thevirtualmemorymanageristunable.Insomecases,youcanachieveperformancebenefitsbychangingthesettings.Youwon’tbeusingthefilesystemcachemuch,andyoudon’twantthesystemtofavorthefilesystemcacheoveryourapplications,whichcansometimesoccurwiththedefaultsettings.

In/etc/sysctl.conf you can set vm.swappinessto1topreventapplicationsfrombeingswappedto disk when there is memory pressure.

dataBase tunIng update

Moderndatabases,especially64-bit,areextremelyefficientatcachingdata.Intheearlydaysof64-bit databases,largebuffersizeswouldslowperformanceduetoelongatedsearchtimes,butthisisnolongertrue.RedHat'stesting,hasresultedinOLTPapplicationswithverylargebuffercachesthatshowvery good results.

Consideryourapplication’scharacteristics—especiallyitsread/writeratios—whendecidingwhetherandhowtousedatabasecaching.Themajorityofapplicationsweseetodayareread-intensive,asmostapplicationsdrivetheirbusinesslogicbyreadingdatafromthedatabase.Themoreread-intensivetheapplicationis,themorecachingcanhelp.Iftheapplicationiswrite-heavy,orthedatasetisverylarge(fromadatawarehouse,forexample),thenalargecachewon’thelpandmayslowdowntheapplication.

YoushoulduseDirectI/Oifyourdatabasesupportsit.Especiallywithalargecacheorawrite-intensiveworkload,youshouldavoiddoublebufferingwiththefilesystembuffercache.NotethatMySQL5documen-tationindicatesthatqueriesmayslowdownbyuptoafactorofthreewhenusingDIRECTI/O,butRedHathasnotexperiencedthisincaseswhereaproperlysizedbufferpoolisused.IfyouareusingDIRECTI/O,youshouldbesuretousethevirtualmemorysettingmentionedearlier.ConsiderusingasynchronousI/Oifyourdatabase supports it.

case study

TodemonstratethedramaticresultsthatcanbeachievedwithJBossEAP5performancetuning, an experiment was performed using a sample application.

ItbeganwithJBossEAP’sdefaultconfiguration,alongwithallLinuxparametersattheirdefaults;RedHatEnterpriseLinuxwasusedforalltests.UsingGrinder,anopensourceJavaLoadTestingFramework,thehighestthroughputthatcouldbeachievedwithallofthesesettingswasmeasured.Usingthesameappli-cation,manyoftheoptimizationscoveredinthisguidewereapplied.Allotherhardware,database,andnetworkconfigurationsremainedthesamethroughouteachtestscenario.


www.jboss.com 13

Virtual users

1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90

80

70

60

50

40

30

20

10

0

Tran

sact

ion

s pe

r se

con

d

JBoss EAP 5 GA ResultsMean, no optimizations

figure 1a

ThesampleapplicationwasanEJB3applicationwithtwoservletsfortheUI,statelessandstatefulsessionbeansformostofthebusinesslogic,amessagedrivenbeanforsomeasynchronousprocessing,andentitiesforthepersistence.Fortheinitial,non-optimizedtests,defaultJBossEAPconfigurationswereused(threadpoolof64,defaultheapof1.3GB).

Fortheoptimizedtestruns,configurationsweretunedspecificallywithscalinginmind;heapsizewasadjustedto3.7GBwithlargepagememory,thethreadpoolwasincreasedupto150,andthedatabaseconnectionpoolwasalsoat150.Additionally,thefollowingJVMoptimizationswereaddedtotheconfigura-tion:-XX:+AggressiveOptsand+XX:DoEscapeAnalysis.

Performanceresultsforbothscenariosareshownbelowinthefollowingsetofgraphs.Thefirstsetofresults measures scalability using mean transactions per second (TPS). The initial test run was conducted withdefault,non-optimizedconfigurations(Figure1A).Thesecondtestrunusedthesameapplication,butleveragedtheperformanceoptimizationshighlightedpreviously(Figure1B).


14 www.jboss.com

Virtual users

JBoss EAP 5 GA ResultsMean, optimized

95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170

140

120

100

80

60

40

20

0

Tran

sact

ion

s pe

r se

con

d

figure 1B

Notethedramaticimprovementsbetweeneachscenario.Thefirst,non-optimizedtestscaledupto90virtualusers,achievingmorethan75,000peakTPS.ByoptimizingtheJBossEAPconfigurations,we wereabletoalmostdoubletheamountofvirtualusersaddedto170andmorethandoubledthepeakamountoftransactionsatoverthan140,000TPS.

Thenextsetofgraphsfocusonapplicationresponsivenessunderload.Withoutoptimizations(figure2A), theapplicationpeakedat90virtualuserswitha300millisecondaveragerequestresponse.


www.jboss.com 15

1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90

300

250

200

150

100

50

0

Mill

isec

on

d re

spo

nse

tim

e

Virtual users


figure 2a


16 www.jboss.com

Virtual users


95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170

300

250

200

150

100

50

0

Mill

isec

on

d re

spo

nse

tim

e

figure 2B

Withoptimizations(figure2B),whenscalingoutto170virtualusers,meanapplicationresponsiveness waslessthan300millisecondsperrequest,anoticeableimprovementoverthenon-optimizedscenario.


www.jboss.com 17

application performance tuning: a continuous process

Tuninganapplicationforoptimalperformancecanensureapositiveuserexperience,promotebusinessproductivity,andhelptooptimizeuseofhardwareandsoftwareresources.Performancetuningisnotaone-timetask,butanongoingprocessthatensuresawell-performingapplicationasbusinessconditionsandsystemtechnologychangeovertime.Applicationdevelopersandarchitectsshouldalwaysbepreparedtotunetheirapplicationsforperformancebothbeforeandaftertheyareputintoproduction.Asalways,themorebusiness-criticalanapplicationisandthehigherthevolumeoftransactionsitmustsupport,themoreimportant performance tuning will be to your business.

Whentunedinaccordancewiththecharacteristicsofyourapplication,JBossEAPcanprovidesuperiorapplicationperformance.Thispaperhasgivenyouanoverviewofbasicperformanceprinciplesaswellasan introduction to performance tuning best practices for JBoss EAP. Keep in mind that these techniques applytoanyoftheRedHatplatformsthatJBossEAPsupports:JBossEnterprisePortalPlatform,JBossEnterpriseSOAPlatform,andJBossEnterpriseBRMS.

ForadditionalinformationonJBossEAPperformancetuning,pleasereferto:

Managingapplicationperformance:JBossOperationsNetwork•

JBossEAP5AdministrationandConfigurationGuide•

JBossEAPOptimizationandTuningAssistanceServices•

JBossEAP5ProductDocumentation•


18 www.jboss.com

appendix: using large-page memory (linux-specific instructions)

Thisappendixcontainsaprocedureandexampledesignedtohelpyoutakeadvantageoflarge-pagememoryforyourapplications.Seethesection“Linux-specifictuning:Largememorypages”onpage11.

1. tell the Jvm to use large-page memory.

SunJVMandOpenJDKrequirethefollowingoption,passedonthecommandline,touselargepages:

-XX:+UseLargePages

TheSuninstructionsleaveitatthat,butifyoudonothingelseyouwillmostlikelygetthefollowingerror:Failed to reserve shared memory (error-no=12)

The following sections describe additional steps you should complete.

2. set kernel parameters.

Set three kernel parameters in /etc/sysctl.confasfollows:

kernel.shmmax = n • Where n is equal to the number of bytes of the maximum shared memory segment allowed on the system. YoushouldsetittoatleastthesizeofthelargestheapsizeyouwanttousefortheJVM.Alternatively,youcansetittothetotalamountofmemoryinthesystem,andyouwillneverhave torevisitit.

vm.nr_hugepages = n• Wherenisequaltothenumberoflargepages.Youwillneedtolookupthelargepagesizein /proc/meminfo.

vm.huge_tlb_shm_group = gid• WheregidistheIDofasharedgroupIDfortheusersyouwanttohaveaccesstothelargepages. This setting enables you to limit access to the large memory segment.

3. set lImIts.conf parameters.

Next,setthesememlocklimitsin/etc/security/limits.conf:

<username> soft memlock n <username> hard memlock n

Where <username>istheruntimeuseroftheJVM,andnisthenumberofpagesfromvm.nr_hugepages multiplied by the page size in KB from /proc/meminfo.

4. persIst your settIngs.

Enterthecommand:

sysctl –p

Thisensuresthatthesettingsyoucreatedinstep3willsurviveareboot.


www.jboss.com 19

5. reBoot.

WhentheOSallocatesthesepages,itmustfindcontiguousmemoryforthemortheoperationwillfail. Arebootwillpreventthisproblem.

6. confIrm page allocatIon.

Whenlargepagesareallocated,/proc/meminfo will display a non-zero number for HugePages_Total. Ifyoudonotseeanon-zeronumber,thenyouarenotusingthelargepages,andsomethingisconfiguredincorrectly.

WehavesometimesseenaproblemwithMySQLwheretheSELinuxpolicywaspreventingitfromaccessingthe large pages. Check /var/log/messages for avc_denied messages (error-no=13 “permission denied”) in the mysqld.log.

example: settIng large memory pages

Consideraserverwith8GBofmemory.Wewillallocate6GBtobesharedbytheJBossEAPJVMand aMySQLdatabase.

Thepagesizeis2MB,asshownin/proc/meminfo (Hugepagesize: 2048KB)

Configuration:/etc/sysctl.conf

Changemaximumsharedmemorysegmentsizeto8GB. •kernel.shmmax = 8589934592

Add the gid to the • hugetlb_shm_grouptogiveaccesstotheusers. vm.hugetlb_shm_group = 501

Add6GBin2MBpagestobesharedbetweentheJVMandMySQL. •vm.nr_hugepages = 3072

Calculations:

1024*1024*1024*8=8589934592

(1024*1024*1024*6)/(1024*1024*2)or6GB/2MB=3072pages

Configuration:/etc/security/limits.conf

AddthelimitsformemlocktoallowtheJVMandMySQLtoaccessthelarge-pagememory.•

jboss soft memlock 6291456 jboss hard memlock 6291456 mysql soft memlock 6291456 mysql hard memlock 6291456 root soft memlock 6291456 root hard memlock 6291456

Calculations:

3072largepages*2048KBpagesize–3072*2048=6291456

Configuration:/etc/group

AddJBossandMySQLuserstothe501(• hugetlb) group in /etc/grouptogiveuserspermission to attach to the shared memory segment.


Copyright©2010RedHat,Inc.RedHat,RedHatEnterpriseLinux,theShadowmanlogo,JBoss,MetaMatrix,andRHCEaretrademarksofRedHat,Inc.,registeredintheU.S.andothercountries.Linux® is the registered trademarkofLinusTorvaldsintheU.S.andothercountries.

JBoss sales and InquIrIes

north amerIca 1–888–REDHAT1 www.jboss.com

europe, mIddle east and afrIca 0080073342835 www.europe.redhat.com [email protected]

asIa pacIfIc +6564904200 www.apac.redhat.com [email protected]

latIn amerIca +541143416200 www.latam.redhat.com [email protected]

www.jboss.com #1622647_0310

aBout red hat

RedHatwasfoundedin1993andisheadquarteredinRaleigh,NC.Today,withmorethan60officesaroundtheworld,RedHatisthelargestpubliclytradedtechnologycompanyfullycommittedtoopensource.Thatcommitmenthaspaidoffovertime,forusandourcustomers,provingthevalueofopensourcesoftware andestablishingaviablebusinessmodelbuiltaroundtheopensourceway.RedHatprovideshigh-quality,affordabletechnologytotheenterprise.Oursolutionsaredeliveredviasubscriptionandrangefromoper-atingsystemsandplatformslikeRedHatEnterpriseLinuxandJBossEnterpriseMiddleware,toapplicationandmanagementtools,aswellasconsulting,training,andsupport.

best practices for performance tuning jboss enterprise ... · pdf filebest practices for...

Documents