![Page 1: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/1.jpg)
8th Sakai Conference
4-7 December 2007Newport Beach
Collaborative Performance TestingCollaborative Performance TestingHealthy Medicine for Robust Sakai CodeHealthy Medicine for Robust Sakai Code
![Page 2: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/2.jpg)
Collaborative Performance Collaborative Performance TestingTesting
• Using the example of Samigo, how performance testing can help developers find and fix performance defects before production.
![Page 3: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/3.jpg)
Who are we?Who are we?
• Lydia Li – Sakai Developer, Stanford University
• Chris Franz – President, Unicon
• Drew Zhu – Oracle DBA, University of Michigan
• Chris Kretler – Performance Tester, University of Michigan
![Page 4: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/4.jpg)
What is performance testing?What is performance testing?
• Replication of middleware calls• Tests have different goals:
– “Load”– “Stress”– “Benchmark”– “Capacity Planning”
• Response times, user capacity limits and infrastructure metrics
![Page 5: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/5.jpg)
What performance testing is notWhat performance testing is not
• A Simulation– Actual traffic is generated
• GUI test
• High-volume functional test– Scope is narrower
![Page 6: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/6.jpg)
When do we test?When do we test?
• All major releases – “Significant” patches
• Hardware Evaluations
• Configuration Changes
• Sakai performance testing
![Page 7: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/7.jpg)
Michigan Test EnvironmentMichigan Test Environment
• Dedicated testing environment:– Load Balancers (Netscaler RS9000s)– 8 application servers (Dell 2650s running RedHat)– 1 Database server (Sun T2000 running 10g)
• System Administrators (Jeff, Adi, Rick, Ken, Jenny)
• Database Administrator (Drew)• LoadRunner software
![Page 8: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/8.jpg)
Samigo Setup ProcessSamigo Setup Process
• Working Sakai build w/Samigo• Determined requirements and focus• Courses and students were created• Sample tests imported• Script creation
– Lydia’s help was invaluable here.
![Page 9: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/9.jpg)
Sample Script PortionSample Script PortionUnderstanding expected result is crucial to script creation.
![Page 10: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/10.jpg)
Load Test ResultsLoad Test Results
![Page 11: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/11.jpg)
DBA Tasks in Load Tests DBA Tasks in Load Tests
• Monitoring Database and DB Server
• Produce AWR reports
• Analyze AWR Reports and Identify Issues
• Communicate with Developers, Testers, and Sysadmins
• Query Details and Analyzing Execution Plans
• Provide suggestions
![Page 12: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/12.jpg)
Oracle OEM Screenshot -1Oracle OEM Screenshot -1
![Page 13: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/13.jpg)
Oracle OEM Screenshot -2Oracle OEM Screenshot -2
![Page 14: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/14.jpg)
Oracle OEM Screenshot -3Oracle OEM Screenshot -3
![Page 15: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/15.jpg)
Samigo Load Test AWRSamigo Load Test AWR• CPU Elapsed CPU per % Total• Time (s) Time (s) Executions Exec (s) DB Time SQL Id• ---------- ---------- ------------ ----------- ------- -------------• 3,184 3,296 68,972 0.05 43.8 38xh90r4gx5cz• select itemgradin0_.ITEMGRADINGID as ITEMGRAD1_139_, itemgradin0_.ASSESSMENTGRAD• INGID as ASSESSME2_139_, itemgradin0_.PUBLISHEDITEMID as PUBLISHE3_139_, itemgra• din0_.PUBLISHEDITEMTEXTID as PUBLISHE4_139_, itemgradin0_.AGENTID as AGENTID139_• , itemgradin0_.SUBMITTEDDATE as SUBMITTE6_139_, itemgradin0_.PUBLISHEDANSWERID a
• 858 858 254,115 0.00 11.4 0byxm85k79suc• select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS• WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE• RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as• TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A
• 834 980 79,456 0.01 13.0 3px7qs284b3xk• select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7• _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1• 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe• rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_
• 203 203 43,210 0.00 2.7 9xmw3p1px195k• select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from• SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i• n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and (• ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_
• 142 142 55,652 0.00 1.9 5n3rvywwp9y7u• select itemmetada0_.ITEMID as ITEMID1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1• _1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1_134_0_, itemmetada0_.ITEMID as ITE• MID134_0_, itemmetada0_.LABEL as LABEL134_0_, itemmetada0_.ENTRY as ENTRY134_0_• from SAM_PUBLISHEDITEMMETADATA_T itemmetada0_ where itemmetada0_.ITEMID=:1
![Page 16: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/16.jpg)
Samigo Load Test AWRSamigo Load Test AWR• CPU Elapsed CPU per % Total• Time (s) Time (s) Executions Exec (s) DB Time SQL Id• ---------- ---------- ------------ ----------- ------- -------------• 10,105 10,659 2,125,189 0.00 15.2 0byxm85k79suc• select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS• WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE• RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as• TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A
• 8,184 8,459 99,794 0.08 12.1 8wcadnu8rwd5b• select count(assessment0_.ASSESSMENTGRADINGID) as col_0_0_ from SAM_ASSESSMENTGR• ADING_T assessment0_ where assessment0_.FORGRADE=1 and assessment0_.AGENTID=:1 a• nd assessment0_.PUBLISHEDASSESSMENTID=:2
• 7,290 8,974 665,235 0.01 12.8 3px7qs284b3xk• select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7• _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1• 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe• rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_
• 5,979 6,189 77,580 0.08 8.8 3cswn2dndzcm0• select assessment0_.ASSESSMENTGRADINGID as ASSESSME1_140_, assessment0_.PUBLISHE• DASSESSMENTID as PUBLISHE2_140_, assessment0_.AGENTID as AGENTID140_, assessment• 0_.SUBMITTEDDATE as SUBMITTE4_140_, assessment0_.ISLATE as ISLATE140_, assessmen• t0_.FORGRADE as FORGRADE140_, assessment0_.TOTALAUTOSCORE as TOTALAUT7_140_, ass
• 2,174 2,232 317,661 0.01 3.2 9xmw3p1px195k• select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from• SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i• n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and (• ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_
• 1,950 2,026 22,183 0.09 2.9 8su2pcsy7fbq5• select assessment0_.ASSESSMENTGRADINGID as col_0_0_, publisheda1_.ID as col_1_0_• , publisheda1_.TITLE as col_2_0_, assessment0_.AGENTID as col_3_0_, assessment0_• .SUBMITTEDDATE as col_4_0_, assessment0_.ISLATE as col_5_0_, assessment0_.FORGRA• DE as col_6_0_, assessment0_.TOTALAUTOSCORE as col_7_0_, assessment0_.TOTALOVERR
![Page 17: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/17.jpg)
New Indexes for SamigoNew Indexes for Samigo• create index idx_ASSGRAD_AID_PUBASSEID on
SAM_ASSESSMENTGRADING_T(AGENTID,PUBLISHEDASSESSMENTID) tablespace ctools_indexes; create index IDX_ITEMGRADING_A_GRADINGID on SAM_ITEMGRADING_T(ASSESSMENTGRADINGID) tablespace ctools_indexes; create index IDX_PUBANS_ITEMTEXTID on SAM_PUBLISHEDANSWER_T(ITEMTEXTID) tablespace ctools_indexes; create index IDX_PUBMETDATA_ASSESSMENTID on SAM_PUBLISHEDMETADATA_T(ASSESSMENTID) tablespace ctools_indexes;
![Page 18: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/18.jpg)
Samigo Load Test AWRSamigo Load Test AWR• CPU Elapsed CPU per % Total• Time (s) Time (s) Executions Exec (s) DB Time SQL Id• ---------- ---------- ------------ ----------- ------- -------------• 4,840 5,046 1,007,039 0.00 21.0 0byxm85k79suc• select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS• WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE• RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as• TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A
• 3,489 4,245 315,353 0.01 17.7 3px7qs284b3xk• select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7• _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1• 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe• rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_
• 1,072 1,096 152,652 0.01 4.6 9xmw3p1px195k• select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from• SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i• n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and (• ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_
• 817 839 220,813 0.00 3.5 5n3rvywwp9y7u• select itemmetada0_.ITEMID as ITEMID1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1• _1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1_134_0_, itemmetada0_.ITEMID as ITE• MID134_0_, itemmetada0_.LABEL as LABEL134_0_, itemmetada0_.ENTRY as ENTRY134_0_• from SAM_PUBLISHEDITEMMETADATA_T itemmetada0_ where itemmetada0_.ITEMID=:1
• 623 643 15,350 0.04 2.7 c55kf9dj75jy4• select count(1) from SAKAI_REALM_RL_FN MAINTABLE LEFT JOIN SAKAI_REALM_RL• _GR GRANTED_ROLES ON (MAINTABLE.REALM_KEY = GRANTED_ROLES.REALM_KEY AND• MAINTABLE.ROLE_KEY = GRANTED_ROLES.ROLE_KEY), SAKAI_REALM REALMS, SAKAI_REAL• M_ROLE ROLES, SAKAI_REALM_FUNCTION FUNCTIONS where ( ROLES.ROLE_NAME in(
![Page 19: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/19.jpg)
Server Side MonitoringServer Side Monitoring• Ran Perl script to capture CPU utilization on
one of eight app servers as well as the DB server
• The amount of free space in the JVM on one of the application servers was also captured
![Page 20: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/20.jpg)
JAVA_OPTSJAVA_OPTSJAVA_OPTS="-server -Xms1500m -Xmx1500m"
JAVA_OPTS="$JAVA_OPTS -XX:NewSize=400m XX:MaxNewSize=400m"
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=192m -XX:MaxPermSize=192m"
JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -Dsakai.home=/usr/local/sakai/home -Dsakai.components.root=/usr/local/sakai/components"
JAVA_OPTS="$JAVA_OPTS -Dsakai.security=/usr/local/sakai/home"
JAVA_OPTS="$JAVA_OPTS -Dnetworkaddress.cache.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"
JAVA_OPTS="$JAVA_OPTS -Dsun.net.inetaddr.ttl=0 -Dsun.net.inetaddr.negative.ttl=0"
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
![Page 21: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/21.jpg)
CPU UtilizationCPU Utilization
• Both application server and database server utilization was spiky
• Application server was in the 20-60% range
• Database server was in the 60-100% range
![Page 22: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/22.jpg)
CPU Utilization GraphsCPU Utilization Graphs
![Page 23: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/23.jpg)
App Server Heap UsageApp Server Heap Usage
![Page 24: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/24.jpg)
Application ProfilingApplication Profiling
• If bottleneck was determined to be on application server side, we were prepared to use JProfiler to capture internal JVM statistics
• Due to its overhead, a profiler can generally
only capture statistics under lighter loads
![Page 25: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/25.jpg)
Purpose for the tripPurpose for the trip• Identify bottlenecks under load.
– no local performance test environment and expertise
• Establish a baseline for future regression testing.
• Gain confidence in our local deployment.– Samigo is an important tool for our users.
• Publish performance test report to the Sakai community.
![Page 26: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/26.jpg)
Prepare for Performance TestPrepare for Performance Test• What test case scenarios are we testing?
– Focus on Samigo’s delivery feature.
• What is the expected usage pattern– Our user support team provided the expected # of
concurrent users at Stanford– Emailed and collected other schools’ usage information
• Prepare for the test– Created sample assessments with question types. – Send Chris XMLs to be pre-loaded to the test
environment. – Assisted Chris with load test scripts.
![Page 27: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/27.jpg)
During the testDuring the test
• Monitored Oracle DB
• Monitored Application Servers,
• Monitored LoadRunner’s test results/graphs.
• Iterative performance tuning based on test results.
![Page 28: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/28.jpg)
What we learnedWhat we learned
• Database:– Added indexes upon Drew’s recommendation.
• Application:– Optimized application code to reduce excessive
DB calls (SAK-10153)
• No memory leaks found.
![Page 29: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/29.jpg)
Things we didn’t test in deliveryThings we didn’t test in delivery
• Focused on Samigo’s Delivery only– Did not test Audio and File Upload questions due
to limited time– Audio/File upload questions are heavily used at
Stanford. Please let us know if you have questions.
![Page 30: 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649f4a5503460f94c6c4d9/html5/thumbnails/30.jpg)
SummarySummary
• Performance test helps to identify bottlenecks under load.
• Establishes a baseline for future testing. – Identify performance improvement/degradation in future
releases.
• Load test should be an integral part of release.
• Performance test report can be found at http://bugs.sakaiproject.org/confluence/display/SAM/Home