Learning
SEO: How To Get Your 20 Minutes Worth Of SEO Done In 20 Minutes Lct.2
Learn SEO Free: 2. Check Robots.txt (Minutes 11-12)
2. Check Robots.txt (Minutes 11-12)
The Robots.txt file SEO is among the most important way to communicate to search engines where you want them to crawl and what pages you don’t want to be crawled.
Super important: The robots.txt file only controls the crawling of but not the indexing of pages. Some small sites have one or two lines in the file, while massive sites have incredibly complex setups.
Your average site will have just a few lines, and it rarely changes week to week. Despite the file rarely changing, it’s important to double-check that it’s still there and that nothing unintentional was added to it.
In the worst-case scenario, such as on a website relaunch or a new site update from your development team, the robots.txt file might get changed to “Disallow: /” to block search engines from crawling while the pages are under development on a staging server and then brought over to the live site with the disallow directive intact.
Make sure this is not on the live website:
User-agent: *
Disallow: /
But if it’s a normal week, there won’t be any changes, and it should only take a minute.
Follow for more Next Topic
Topic : 3. Review Page Speed In Google Analytics (Minutes 13-15)
Click on the provided link: NEXT PAGE
