Episode #137
Why and How You Should Control Your Crawl Budget
December 15, 2016

In Episode #137, Neil and Eric discuss the importance of knowing how to control your crawl budget. Tune in to discover how the quality of your pages affects your crawl budget and, in turn, can directly influence the level of traffic to your website. 

Time Stamped Show Notes:

  • 00:27 – Today’s topic: Why and How You Should Control Your Crawl Budget
  • 00:35 – The definition of a crawl budget
  • 01:09 – “Every site has a limited crawl budget”
  • 01:28 – Control what pages Google crawls and have them crawl your most important ones
  • 02:02 – Make sure the Google Search Console is installed
  • 02:50 – Consider removing low quality pages from your website or use robotstxt to block Google from crawling them
  • 03:21 – Deleting content can sometimes increase traffic
  • 03:35 – Look at impressions in your Google Search Console
  • 04:11 – Link and direct more people to high quality pages
  • 04:29 – WhitePages block off a lot of pages
  • 05:03 – Check the crawl error section in your Google Search Console
  • 06:20 – The faster your servers, the more pages are crawled
  • 07:05 – That’s it for today’s episode!

3 Key Points:

  1. A crawl budget is the maximum amount of pages Google crawls on your website.
  2. Keep your high quality pages up and running, then FOCUS your energy on linking people to those pages.
  3. Make sure you have a fast server so your website is crawled faster.

Leave some feedback:

  • What should we talk about next? Please let us know in the comments below.
  • Did you enjoy this episode? If so, please leave a short review.

Connect with us:

Discover Latest Episodes

Episode
ChatGPT Quietly Switches From Bing to Google
July 14, 2025

In this episode #3004, Eric Siu and Neil Patel discuss the shift from Bing to Google and the Microsoft-OpenAI partnership’s impact on businesses. They cover how marketing agencies are evolving amid ...

Episode
MrBeast’s SaaS Decision – A Mistake?, Pay per Crawl Launches for AI, Listicles Are Back, Why You Need Them for LLMs, How LLMs Will Rank You in the Future, AI Content Farms Are the New Arbitrage?
July 10, 2025

In this episode #3003, Eric Siu and Neil Patel discuss MrBeast's ViewStats and the AI thumbnail backlash, Cloudflare's pay-to-play model, and how LLMs are shaping the future of SEO. They also cover th...

Episode
The Collapse of Jaguar, How Meta Is Buying OpenAI Right in Front of Us, AI Mode vs. Classic Search Data, ChatGPT 5 Will Be INSANE
July 09, 2025

In this episode #3002, Eric Siu and Neil Patel discuss the collapse of Jaguar due to marketing missteps and the importance of understanding core audiences. They dive into Meta's AI acquisitions, compa...