serpact.bg · incorrect use of robots.txt + sitemap xml files capacity to respond #1 quickly time...
TRANSCRIPT
![Page 1: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/1.jpg)
![Page 2: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/2.jpg)
![Page 3: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/3.jpg)
![Page 4: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/4.jpg)
![Page 5: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/5.jpg)
Discovery
Where are the new URLs or new content on existing URLs?
Example of a source:- External links
Are there more to find behind this initial lead provided by discovery?
Bots will crawl the site based on the statement delivered by crawl scheduler
Crawling
Rendering
Let's analyze the web content and interpret it like a user
On the 2nd wave of indexing, bots render URLs by fetching scripts
Indexing
What are we going to show our users?
Upon successful discovery, crawl and rendering of URLs, they get to stored and categorized based on their context
![Page 6: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/6.jpg)
![Page 7: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/7.jpg)
●●
●●
![Page 8: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/8.jpg)
![Page 9: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/9.jpg)
●
●
●●
●
![Page 10: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/10.jpg)
Sites with > 10K URLs
YES!
Sites with < 10K URLs
Unless your new pages are crawled around the same day of publication
Should I really be concerned about my site's "crawl budget" ?
![Page 11: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/11.jpg)
![Page 12: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/12.jpg)
Freshness#5 ● Users tend to prefer fresh content● PageRank decays in time
Internal linking structure#4 ● Too complex & deep website architecture● # of internal & external links pointing to URLs
Quality of pages#3 ● Low quality, no added value or spam pages● Duplicated pages (index bloating)
Domain health#2● 50x / 40x / Soft 404 errors● Redirection chains & faulty redirects● Incorrect use of robots.txt + sitemap XML files
Capacity to respond quickly#1 ● Time to last byte - TTLB
● Average time the bot downloads a page
![Page 13: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/13.jpg)
![Page 14: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/14.jpg)
![Page 15: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/15.jpg)
![Page 16: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/16.jpg)
![Page 17: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/17.jpg)
![Page 18: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/18.jpg)
![Page 19: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/19.jpg)
![Page 20: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/20.jpg)
![Page 21: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/21.jpg)
![Page 22: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/22.jpg)
![Page 23: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/23.jpg)
![Page 24: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/24.jpg)
![Page 25: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/25.jpg)
![Page 26: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/26.jpg)
![Page 27: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/27.jpg)
●●●
●
●
●
●
![Page 28: serpact.bg · Incorrect use of robots.txt + sitemap XML files Capacity to respond #1 quickly Time to last byte - TTLB Average time the bot downloads a page](https://reader033.vdocuments.mx/reader033/viewer/2022042612/5f78cb5cb40ab4205c2ea520/html5/thumbnails/28.jpg)