performance tuning

PowerPoint Presentation

Enterprise Software Development

Eric Phan | Chief Architect @ SSWPerformance Tuning from the TrenchesJoin the Conversation #NetUG @EricPhan

Case StudyOptimizing .NETOptimizing SQLOptimizing Web Front EndSummary Lessons LearntFalse StartAgenda

B.E. - Software Engineering, Certified Scrum MasterEric Phan @EricPhanChief Architect @ SSWMCTS TFS, MCPLoves playing with the latest technology


Case Study A large retailerProblems with their existing online shopping siteOld, outdated, running on classic ASPWebsite crashed every time they had sales eventsLosing $$$$Volume starts increasing in the lead up to Xmas (from about late august)Physical shop + website closes after XmasMassive reopening sale mid January (this is usually when the website crashes)Join the Conversation #NetUG @EricPhan

Case Study A large retailerSSW was engaged to:Upgrade their site to a later version of their eCommerce platformDevelop a few new features (gift registry, gift cards, integration with their inventory tracking system)Ensure the site stays up during sales

Join the Conversation #NetUG @EricPhan

Dev dev devAs Jan approaches, get ready for go liveSprint just focussed on performanceRun MS web + load tests then optimize



Tweaking performanceRun .NET profilers (ANTS, JetBrains)Run SQL ProfilerAdd cachingJoin the Conversation #NetUG @EricPhan



Send it live!

Eric Phan (EP) - Eric Phan (EP) - Join the Conversation #NetUG @EricPhan

Crash and burnRoll back to old site (still crashes)Restart IIS constantlyNow that weve missed this date, theres more time to dig deeperJoin the Conversation #NetUG @EricPhan

Post MortemWe concluded that:The web tests didnt accurately reflect how users would use the siteThere were a lot more users hitting the site than the initial target (due to it being the big sale)The tests also didnt accurately represent the production environmentJoin the Conversation #NetUG @EricPhan


Then we embarked on a journeyFix tests to more accurately represent the loadIdentify bottle necksOptimize!Join the Conversation #NetUG @EricPhan

Fixing the testStart at 200Ramp up to 5000New goal was 4000 concurrent usersSee the fail!Join the Conversation #NetUG @EricPhan

Identify bottlenecksCode profilingSQL profilingHardwareJoin the Conversation #NetUG @EricPhan

No matter what we did


So we suspected something to do with the Hardware and HostingSimplified test to pull 1 image from the web server (no DB, no .NET)Same problem!Join the Conversation #NetUG @EricPhan

Checking the IIS serverLog onto the IIS server and monitor CPUStart Performance Monitor and add stats for:UsersSessionsThroughputWhen we hit the 5 minute fail markNo users, no session, no throughput logged in IISJoin the Conversation #NetUG @EricPhan

Cause?DDOS protectionJoin the Conversation #NetUG @EricPhan


Rerun Load TestsThis time, instead of failing within 5 mins, We are getting to 40mins before failingLog another ticket with the hostJoin the Conversation #NetUG @EricPhan


Progress!



What happened here?


Running the proper tests (not the single image)Now that the infrastructure issues have been sorted we can run the actually test scriptsSearch for productsAdd 10 to shopping cartGo through checkout processJoin the Conversation #NetUG @EricPhan


So this time..CPU + RAM are good on DB + IISBut page still arent being served?Join the Conversation #NetUG @EricPhan


Network saturationHitting the max capacity of the network cards between the web and SQL serverSwitching to gigabit fixed the issueBut its still a lot of chatter!Join the Conversation #NetUG @EricPhan

Identifying BottlenecksNow that we know our infrastructure is good.How do we find where our app is slow?Join the Conversation #NetUG @EricPhan

Identifying BottlenecksUser feedbackGlimpseIntelliTraceAPM toolsProfilingJoin the Conversation #NetUG @EricPhan

GlimpseInstall-Package GlimpseF5~/glimpse.axdgJoin the Conversation #NetUG @EricPhan





Intelli TraceGood to quickly eyeball whats going on under the coversJoin the Conversation #NetUG @EricPhan




Application Insightshttps://rules.ssw.com.au/rules-to-better-application-insightsJoin the Conversation #NetUG @EricPhan



Optimize SQLRun SQL ProfilerUse the Duration templateIdentify slow running queries



exec sp_executesql N'SELECT [Project1].[ObligationId] AS [ObligationId], [Project1].[ObligationGroupMemberId] AS [ObligationGroupMemberId], [Project1].[ObligationGroupId] AS [ObligationGroupId], [Project1].[OrganisationEntityId] AS [OrganisationEntityId], [Project1].[Code] AS [Code], [Project1].[Name] AS [Name], [Project1].[C1] AS [C1], [Project1].[C2] AS [C2], [Project1].[Id] AS [Id], [Project1].[Email] AS [Email], [Project1].[FullName] AS [FullName], [Project1].[Role] AS [Role], [Project1].[FinalizationDate] AS [FinalizationDate], [Project1].[Username] AS [Username] FROM ( SELECT [Extent1].[ObligationId] AS [ObligationId], [Join1].[ObligationGroupMemberId] AS [ObligationGroupMemberId], [Join1].[ObligationGroupId] AS [ObligationGroupId], [Join1].[OrganisationEntityId1] AS [OrganisationEntityId], [Join1].[Name] AS [Name], [Join1].[Code] AS [Code], cast(1 as bit) AS [C1], [Join3].[Id1] AS [Id], [Join3].[Username] AS [Username], [Join3].[FullName] AS [FullName], [Join3].[Email] AS [Email], [Join3].[Role] AS [Role], [Join3].[FinalizationDate] AS [FinalizationDate], CASE WHEN ([Join3].[Id1] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C2] FROM [dbo].[Obligation] AS [Extent1] INNER JOIN (SELECT [Extent2].[ObligationGroupMemberId] AS [ObligationGroupMemberId], [Extent2].[ObligationGroupId] AS [ObligationGroupId], [Extent3].[OrganisationEntityId] AS [OrganisationEntityId1], [Extent3].[Name] AS [Name], [Extent3].[Code] AS [Code] FROM [dbo].[ObligationGroupMember] AS [Extent2] INNER JOIN [dbo].[OrganisationEntity] AS [Extent3] ON [Extent2].[OrganisationEntityId] = [Extent3].[OrganisationEntityId] ) AS [Join1] ON [Extent1].[ObligationGroupId] = [Join1].[ObligationGroupId] LEFT OUTER JOIN (SELECT [Extent4].[Id] AS [Id1], [Extent4].[ObligationId] AS [ObligationId], [Extent4].[OrganisationEntityId] AS [OrganisationEntityId], [Extent4].[OrganisationHierarchyId] AS [OrganisationHierarchyId], [Extent5].[Username] AS [Username], [Extent5].[FullName] AS [FullName], [Extent5].[Email] AS [Email], [Extent5].[Role] AS [Role], [Extent5].[FinalizationDate] AS [FinalizationDate] FROM [dbo].[ObligationFinalizationByHierarchies] AS [Extent4] INNER JOIN [dbo].[ObligationFinalizationByHierarchyUsers] AS [Extent5] ON [Extent4].[Id] = [Extent5].[ObligationFinalizationByHierarchyId] ) AS [Join3] ON ([Join1].[OrganisationEntityId1] = [Join3].[OrganisationEntityId]) AND ([Join3].[ObligationId] = @p__linq__1) AND ([Join3].[OrganisationHierarchyId] IS NULL) WHERE [Extent1].[ObligationId] = @p__linq__0 ) AS [Project1] ORDER BY [Project1].[ObligationId] ASC, [Project1].[ObligationGroupMemberId] ASC, [Project1].[ObligationGroupId] ASC, [Project1].[OrganisationEntityId] ASC, [Project1].[C2] ASC',N'@p__linq__1 int,@p__linq__0 int',@p__linq__1=64,@p__linq__0=64

Analyse profile resultsLook for anything that takes over 100ms to executeDrill into each query and manually run it in SSMSUse the Execution Plans and Client statistics to helpSee if you can speed it up byAdding an indexRewriting the queryJoin the Conversation #NetUG @EricPhan

Remember this when tuning SQLCHECKPOINT; GO DBCC DROPCLEANBUFFERS; GOJoin the Conversation #NetUG @EricPhan

exec procCleanGhostTestResults


exec procCleanGhostTestResults

CREATE NONCLUSTERED INDEX IX_1ON [dbo].[TestGroupResult] ([Status])INCLUDE ([Id], [DateCreated])

CREATE NONCLUSTERED INDEX IX_2ON [dbo].[TestResultComment] ([TestResultId])INCLUDE ([Id])

CREATE NONCLUSTERED INDEX IX_3ON [dbo].[TestResult] ([TestType])INCLUDE ([Id],[TestId])

DROP INDEX IX_3 ON [dbo].[TestResult]DROP INDEX IX_2 ON [TestResultComment]DROP INDEX IX_1 ON [TestGroupResult]


SQL Execution PlansTells you where the bottlenecks areFat pipesScan vs SeekEstimated Rows vs Actual RowsWarnings


Execution Plan Resourceshttps://www.simple-talk.com/sql/performance/execution-plan-basics/https://www.simple-talk.com/sql/performance/why-developers-need-to-understand-execution-plans/ Join the Conversation #NetUG @EricPhan

Other things to look out forExcessive callsSQL profiler showed that to load the search page, it would make 300+ SQL callsStuff that doesnt make senseWhats worse is that the search triggered 1 insert and 1 updateThis was one of the causes of the high network traffic


Parameter Sniffingexec procGetComplicatedStuff 1exec procGetComplicatedStuff 15Execution plan is cachedProblem when there is uneven distribution of results based on the parameters

https://www.brentozar.com/blitzcache/parameter-sniffing/https://www.brentozar.com/archive/2013/11/why-parameter-sniffing-can-slow-down-queries-video/ https://www.brentozar.com/archive/2013/06/the-elephant-and-the-mouse-or-parameter-sniffing-in-sql-server/Join the Conversation #NetUG @EricPhan

Indexes and Index Usagehttps://www.simple-talk.com/sql/performance/tune-your-indexing-strategy-with-sql-server-dmvs/


Simulating live trafficUse profiler to capture a trace for a period of time that represents a typical usageRun the trace through the Database Tuning advisorRecommended indexesRecommended StatisticsJoin the Conversation #NetUG @EricPhan


DB Optimization SummaryFind bottlenecks:SQL ProfilerFix bottlenecks:Execution PlansDatabase Tuning AdvisorJoin the Conversation #NetUG @EricPhan

Optimize QueriesSELECT only the columns you needUse JOINs effectivelyAVOID cursors like the plagueWatch for Parameter SniffingKeep database statistics up to dateJoin the Conversation #NetUG @EricPhan

Other tipsDenomalize heavily accessed tablesSSDsMore RAMSeparate physical drives for data and logsJoin the Conversation #NetUG @EricPhan


Case StudyOptimizing .NETOptimizing SQLOptimizing Web Front EndSummary Lessons LearntFalse Start

Finding bottlenecksUse profiling toolsRedGate ANTZTelerik JustTraceVisual Studio Performance WizardUse APM tools on productionNewRelicApp Insights

Find the hotspots with Just TraceDrill into the code identified by the profiling toolJoin the Conversation #NetUG @EricPhan




Bad Exampleforeach (var o in Obligations) { foreach (var m in o.ObligationGroup.ObligationGroupMembers) { Console.WriteLine(o.ObligationGroup.GroupName + " - " + m.OrganisationEntityId); }}Join the Conversation #NetUG @EricPhan

Whats wrong with this code?



Watch out for LINQ QueriesEasy to writeGenerates some nasty SQLWatch out for loopsDoes a SELECT * if youre not projectingJoin the Conversation #NetUG @EricPhan

Fixedvar groupMembers = Obligations.Select(x => new { GroupName = x.ObligationGroup.GroupName, EntityId = x.ObligationGroup.ObligationGroupMembers .Select(m => m.OrganisationEntityId)});foreach (var m in groupMembers){ Console.WriteLine(m.GroupName + " - " + m.EntityId);}Join the Conversation #NetUG @EricPhan


Debug = falsehttps://www.ssw.com.au/ssw/HealthCheck Join the Conversation #NetUG @EricPhan

CachingCache data that doesnt change oftenE.g. Product descriptions, product categories, menusExpire the cache when the data changesWhy cache?Reduce calls to SQLReduce CPU used for complicated calculations that dont change often(Its like giving your application a cheat sheet with the answers)Join the Conversation #NetUG @EricPhan

Output Caching You can also cache entire pages if they are static by using the [OutputCache] attributehttp://www.asp.net/mvc/overview/older-versions-1/controllers-and-routing/improving-performance-with-output-caching-cs Join the Conversation #NetUG @EricPhan

Smarter data structuresIEnumerable products = GetProducts();Product match = null;foreach (var product in products) { if (product.Code == ABC) { match = product; break; }}Ormatch = products.FirstOrDefault(x => x.Code == ABC)

O(n)Join the Conversation #NetUG @EricPhan

Smarter Data StructuresIDictionary products = GetProducts() .ToDictionary(x => x.Code, x => x);Product match = products[ABC];

O(1)


Improving the searchSearching for keyword redCheck product name, description, attributes like category, brand and colourCan you imagine the SQL?Join the Conversation #NetUG @EricPhan

SELECT *FROM Product pLEFT OUTER JOIN ProductColour pc ON pc.ProductId = p.ProductIdLEFT OUTER JOIN ProductCategory c ON c.ProductCategoryId = p.ProductCategoryIdLEFT OUTER JOIN ProductBrand b ON b.ProductBrandId= p.ProductBrandId

WHERE p.Name LIKE %red% OR p.Description LIKE %red% OR c.Name LIKE %red% OR b.Name LIKE %red%

Join the Conversation #NetUG @EricPhanSELECT *FROM Product pLEFT OUTER JOIN ProductColour pc ON pc.ProductId = p.ProductIdLEFT OUTER JOIN ProductCategory c ON c.ProductCategoryId = p.ProductCategoryIdLEFT OUTER JOIN ProductBrand b ON b.ProductBrandId= p.ProductBrandId

WHERE p.Name LIKE %red% OR p.Description LIKE %red% OR c.Name LIKE %red% OR b.Name LIKE %red%

Improving the searchYou could use SQL Full Text Indexes


Join the Conversation #NetUG @EricPhanSELECT p.*FROM CONTAINSTABLE(Products, *, red') piINNER JOIN Products p on pi.[KEY] = p.ProductIdLEFT OUTER JOIN ProductColour pc ON pc.ProductId = p.ProductIdINNER JOIN CONTAINSTABLE(ProductColour, *, red') pci ON pci.[KEY] = pc.ProductColourIdLEFT OUTER JOIN ProductCategory c ON c.ProductCategoryId = p.ProductCategoryIdINNER JOIN CONTAINSTABLE(ProductCategory, *, red') ci ON ci.[KEY] = c.ProductCategoryIdLEFT OUTER JOIN ProductBrand b ON b.ProductBrandId= p.ProductBrandIdINNER JOIN CONTAINSTABLE(ProductBrand, *, red') bi ON bi.[KEY] = b. ProductBrandId

Improving the SearchWe opted to use Lucene.NEThttp://stackoverflow.com/questions/37059/lucene-net-and-sql-serverhttp://tilt.carr.no/Post/1/full-text-searching-with-lucene-net http://people.apache.org/~mikemccand/lucenebench/ As a bonus, you can add Solr to Lucene.NET and get a free REST API for querying your indexesJoin the Conversation #NetUG @EricPhan

Other coding tipsConsider using parallel task library for large data setshttps://msdn.microsoft.com/en-us/library/dd997392(v=vs.110).aspxhttp://www.codeproject.com/Articles/362996/Multi-core-programming-using-Task-Parallel-LibraryStart using the Async Await patternhttp://www.tugberkugurlu.com/archive/how-and-where-concurrent-asynchronous-io-with-asp-net-web-api Join the Conversation #NetUG @EricPhan

Keep up to dateMove From .NET 4.0 to .NET Framework 4.5 (30% faster startup, 30% lower memory usage)Move to ASP.NET Corehttp://web.ageofascent.com/asp-net-core-exeeds-1-15-million-requests-12-6-gbps/ Join the Conversation #NetUG @EricPhan

Optimizing .NETFor the shopping site, we added caching and improved the code behind heavily accessed pages like the shopping cart.Searching for products went from:300-500ms and 300 database reads, 1 insert, 1 update to, 28ms and 0 database callsJoin the Conversation #NetUG @EricPhan

Case StudyOptimizing .NETOptimizing SQLOptimizing Web Front EndSummary Lessons LearntFalse Start

Optimizing the front endAll about reducing the time from a user action to when they see something happenFirst loadNavigation

Key things to optimizeReduce # of requestsMove static images, javascript and CSS to a CDN (more concurrent requests)Reduce size of filesCache everything you canUse AJAX calls to fetch data and dynamically load them on a page instead of having to refreshJoin the Conversation #NetUG @EricPhan

Find bottlenecksGoogle Page Speed (built into chrome)YslowPingdomJoin the Conversation #NetUG @EricPhan





More infoMads Kristensen https://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/DEV-B418


Case StudyOptimizing .NETOptimizing SQLOptimizing Web Front EndSummary Lessons LearntFalse StartSummary

Where did we end up?



PerformanceNow SQL server was humming along during the load tests with about 10% CPU utilizationEasily hitting 4000 concurrent users Web server was now able to server way more requests, so the CPUs were starting run hot ~80%...Join the Conversation #NetUG @EricPhan

Hosting EnvironmentJoin the Conversation #NetUG @EricPhanSQL DBWeb ServerIISBusiness LogicFront EndInternetEnd User

Lessons learntKnow what youre target is (100 users? 1000 users?)Have Load tests setup and ready to go early in the development cycleEnsure the infrastructure is good before startingSometimes you dont get what you expect (100MB network vs 1000MB)Sometimes you cant see whats happening (resource contention from other VMs)


Lessons learntSometimes throwing more hardware at the problem is a quick and easy fixCPUs running @ 100% update CPU architecture and now 20%CPU architecture changes every 2 yearsCaching cache all the things!Session state and View State are badChatty SQL is bad, sometimes a stored procedure will do the trickJoin the Conversation #NetUG @EricPhan

Lessons LearntDont launch a new site when you know its going to be smashedGive it some breathing room to make adjustmentsConstantly have performance in mind when codingConstantly run load tests to know when performance changesJoin the Conversation #NetUG @EricPhan

Its important to constantly run load tests


After ShadowProtect was installed

Eric Phan (EP) - Optimizing DatabaseIndexesDenormaliseWatch out for parameter sniffing/snoopingMove complex logic from business layer into stored proceduresGood DB practicesJoin the Conversation #NetUG @EricPhan

Optimizing SoftwareData CachingCaution when using loopsParallelMove long running tasks to background/other threads or use asyncJoin the Conversation #NetUG @EricPhan

Optimizing Web UIsReduce SizeCompressionMinificationReduce Requests BundlingCaching

Optimizing HardwareLoad balancersMurphys LawMore RAMMore CPUsUpdate processor architectureNetwork connections between front end and dbJoin the Conversation #NetUG @EricPhan

Preventing this sort of thingMonitor & Be Proative!Run load tests regularly (daily, weekly, at the end of the sprint, as part of your CI)Load StormLoader.ioAzure Load Tests (20,000 free virtual user minutes per month = users x duration)Join the Conversation #NetUG @EricPhan

Azure Load Tests





Preventing this sort of thingUse a APM tool see performance in real time!New RelicStackify prefix.ioApp InsightsIn Visual Studio use the Intelli Trace Diagnostic Window while codingMake performance part of the DODLook at APM tools before scrum or during the sprint planning to see if theres an issue that needs to be added to the backlogGated check-ins that run load testsJoin the Conversation #NetUG @EricPhan

RememberPerformance tuning is a fine art that you can spend days on just to improve a few hundred milliseconds of performance. Always measure performance before and afterWeigh up the $$$ sometimes its cheaper to throw more hardware at the problemhttp://www.slideshare.net/EricPhan6/performance-tuning-61030451 Join the Conversation #NetUG @EricPhan

Stuff for next timeDeep Dive Optimizing the Front EndDeep Dive Optimizing SQL QueriesScaling in AzureJoin the Conversation #NetUG @EricPhan

Resourceshttps://www.red-gate.com/products/dotnet-development/ants-performance-profiler/#ebook Join the Conversation #NetUG @EricPhan

2 things...

@EricPhan #NetUG

https://www.slideshare.net/EricPhan6/performance-tuning-61030451 Tweet your favourite tip

Join the Conversation #DevOps @AdamCogan @DanijelMalik @EricPhan

Tweet your favourite video @SSWTV Check out tv.ssw.comJoin the Conversation #DevOps @AdamCogan @DanijelMalik @EricPhan

Thank you!

[email protected] | Melbourne | Brisbane | Adelaide