The robots.txt file tells Google how to behave when crawling your website.
It is usually located at: domain.com/robots.txt
You will usually have to edit it via FTP or File Manager via your web hosting.
If Google sees a page as being "blocked" this means it's not allowed to crawl it, and when Google can't crawl something, it will often REMOVE IT FROM THE INDEX COMPLETELY.
Now this isn't such as problem for a filter page that you don't want indexed, but if it's your main category or landing page, then you've got a big problem.
The main reason why this error occurs, is because you have a page in your sitemap, that is being blocked, which means you're telling Google two conflicting things:
- Here is my Sitemap, Google please index all of these pages.
- Here is my robots.txt file, please don't crawl pages like these.
And if you have this error, it means one or more of your pages is showing up in both of these files.
How to Find and Fix The Error
Step 1 is to make sure these pages are supposed to be blocked.This error can often identify when someone has blocked something by accident, and can help you fix this before it causes you some big SEO headaches.
If the pages aren't supposed to be blocked, then analyse your robots.txt file rules, find the one blocking your main page / pages, and rewrite the rule so it doesn't block the pages (but still blocks what it was trying to block in the first place).
Step 2 - if the pages are blocked on purpose - remove them from the sitemap.The way to remove pages from the sitemap varies on different CMS, but here are some examples:
Need A Hand? My Audits Can Help
Analysing and fixing Google Search Console errors is a small part of my SEO Audit & Opportunity Report service, which will help you to identify any technical, on page, user experience, and off site SEO problems and potential missed opportunities.
Contact me via email for more information - [email protected]