Site CrawlingWhat To Do & What To Look For

@katemorris

@katemorris

The “Oh Shit” Moment

@katemorris

The “WTF” Moment

@katemorris

WHEN IS THE

LAST TIME YOU

CRAWLED

YOUR SITE?

@katemorris

CRAWLING TIMELINESCRAWLING FUN

CRAWLING TOOLS CRAWLING DATA

@katemorris

CRAWLING

TOOLS

@katemorris

Moz

Best For: • In-House Marketers

• Beginning SEOs

Cost: $99/month+

User Score: 1-2

@katemorris

Raven

Best For: • In-House Marketers

• Small Agencies

Cost: $99/month+

User Score: 3

@katemorris

Xenu Link Sleuth

Best For: • Technical Experts

• Developers

Cost: Free

User Score: 4

@katemorris

Screaming Frog

Best For: • Technical Marketers

• SEOs

• Developers

Cost: $160/year

User Score: 5

@katemorris

Deep Crawl

Best For: • Expert SEOs

• Sites with Large

Footprints

Cost: $80/ month+

User Score: 6

@katemorris

Crawler User Scale

Unrelated to SEO/Marketing

Novice Level SEO

Knowledge,

Good SEO Knowledge,

Non-Technical

Good Technical,

Minimal SEO

1 2 3 4

5 6 7

Good SEO and Technical

Knowledge

Expert Level SEO Knowledge,

Minimal Technical

Expert Technical and SEO

Knowledge

@katemorris

Crawler User Scale

Unrelated to SEO/Marketing

Novice Level SEO

Knowledge,

Good SEO Knowledge,

Non-Technical

Good Technical,

Minimal SEO

Good SEO and Technical

Knowledge

Expert Level SEO Knowledge,

Minimal Technical

Expert Technical and SEO

Knowledge

BUILD YOUR OWN

@katemorris

THE

CRAW

LAll Examples Use Screaming

Frog or DeepCrawl

@katemorris

DeepCrawl Dashboard

@katemorris

4XX/5XX Errors

@katemorris

4XX/5XX Errors

@katemorris

Internal Redirects

@katemorris

External Redirects

@katemorris

Duplicate Titles

@katemorris

Duplicate Content

@katemorris

URL Issues

@katemorris

Meta Descriptions

@katemorris

Meta Descriptions

DeepCrawl breaks it down for you.

@katemorris

Canonicals

@katemorris

Image Alt Text

@katemorris

PDF Files

@katemorris

HOW OFTEN

DO I HAVE TO

DO THIS?

@katemorris

Every Quarter

@katemorris

Before and After All Major Site Updates

WEBSITE UPDATE WEBSITE UPDATE WEBSITE UPDATE WEBSITE UPDATE WEBSITE UP

@katemorris

Any Major Changes in Organic

Traffic

@katemorris

FUN CRAWLER

USES

@katemorris

Testing Sitemaps

@katemorris

Testing Sitemaps

@katemorris

Testing Backlinks

Issue: Backlink Analysis is time consuming,

especially checking backlink lists for live

links.

Solution: Make a text list of all the backlinks

and crawl. For Screaming Frog, set a

custom search for the client site.

@katemorris

Testing Backlinks

@katemorris

Testing Backlinks

@katemorris

Content Inventory

@katemorris

Confirm GA Implementation

@katemorris

Want more?http://www.seerinteractive.com/blog/screaming-frog-guide

@katemorris

Do This

Download and find your favorite crawler.

Make a calendar entry that reoccurs every three months to remind you to crawl your site.

At minimum, review:– Errors

– Redirects

– Duplicated Titles

– XML Sitemaps

Thanks

Icons provided by: Medialoot