using server logs to your advantage
TRANSCRIPT
![Page 1: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/1.jpg)
Alex Johnson
alex (at) white.net@alex_cestrian
![Page 2: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/2.jpg)
USING SERVER LOGS TO YOUR ADVANTAGE
![Page 3: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/3.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
What are server logs?
![Page 4: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/4.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
A server log is a simple text file which records activity on a server.
![Page 5: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/5.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
![Page 6: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/6.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
So why bother looking at server logs?
![Page 7: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/7.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
There is only one resource that tells you what search engines are looking for on a domain…
These are web server logs.
including stuff they found 13 years ago.
![Page 8: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/8.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
How do we analyse all that data?
![Page 9: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/9.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
![Page 10: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/10.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
![Page 11: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/11.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
![Page 12: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/12.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
![Page 13: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/13.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
2 SCENARIOS
![Page 14: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/14.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
Scenario 1 IDENTIFY ORPHAN PAGES
![Page 15: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/15.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
An orphan is a page that is not linked to by another page on the site.
Homepage
Dresses Skirts Our offers
Summer 2016 offers
![Page 16: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/16.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
Summer 2016 Offers
![Page 17: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/17.jpg)
@alex_cestrian #OptimiseOxford
Why are orphan pages bad?
• There may be a lot of them, and they may be competing with your ‘live’ content
• They waste GoogleBot’s crawl budget for your domain
![Page 18: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/18.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
So how do we find orphan pages using log files?
![Page 19: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/19.jpg)
@alex_cestrian #OptimiseOxford
Upload a crawl of your website (from SF, DeepCrawl etc)
URLs that return a 200 status code✅ …that don’t appear in the crawl of your site
![Page 20: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/20.jpg)
@alex_cestrian #OptimiseOxford
Redundant content, off little value
404/410 status code
Relevant, valuable but out-of-date
301 redirect to relevant live page
Useful content that orphaned accidentally
Re-attach the page to the website
![Page 21: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/21.jpg)
@alex_cestrian #OptimiseOxford
If GoogleBot is wasting lots of time in a specific folder full of orphan pages that hold no value, block it via robots.txt
![Page 22: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/22.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
Scenario 2IMPROVING CRAWL EFFICIENCY
![Page 23: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/23.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
Find where GoogleBot is wasting time
Find parameter driven pages
![Page 24: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/24.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
![Page 25: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/25.jpg)
@alex_cestrian #OptimiseOxford
Block GoogleBot from crawling these URLs
![Page 26: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/26.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
Find infrequently visited pages Order by number of events: low to high
![Page 27: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/27.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
• Is this URL in the xml sitemap?
• Is the page too deep within the architecture?
• Is internal linking to this page optimal?
• Are links to this page travelling through multiple redirects?
• Can GoogleBot actually parse the links pointing to this page?
![Page 28: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/28.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
Look at all urls, and filter by average response time
Find slow loading pages
![Page 29: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/29.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
If time taken is consistently high, you need to look at how you can reduce the load of the page
![Page 30: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/30.jpg)
@alex_cestrian #OptimiseOxford#OptimiseOxford
“See what GoogleBot is actually consuming. Improve GoogleBot’s diet.”Oliver Mason at Brighton SEO 2016
![Page 31: Using server logs to your advantage](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587176ea1a28ab230b8b5293/html5/thumbnails/31.jpg)
THANK YOU
@alex_cestrianALEX JOHNSON
THANK YOU
ALEX