.Google's John Mueller responded to a concern regarding why Google indexes web pages that are actually forbidden from creeping by robots.txt and also why the it's risk-free to neglect the relevant Search Console files about those creeps.Robot Website Traffic To Query Criterion URLs.The individual asking the concern recorded that crawlers were developing web links to non-existent question parameter Links (? q= xyz) to web pages with noindex meta tags that are additionally shut out in robots.txt. What urged the inquiry is actually that Google.com is crawling the web links to those webpages, acquiring blocked out by robots.txt (without noticing a noindex robots meta tag) at that point receiving turned up in Google Look Console as "Indexed, though blocked through robots.txt.".The person talked to the observing concern:." Yet below is actually the significant concern: why will Google.com mark web pages when they can't also see the web content? What's the advantage in that?".Google's John Mueller verified that if they can not creep the webpage they can't see the noindex meta tag. He likewise makes an exciting mention of the website: hunt driver, encouraging to disregard the end results given that the "typical" individuals won't observe those end results.He composed:." Yes, you are actually appropriate: if we can not creep the web page, our experts can't see the noindex. That claimed, if we can't crawl the pages, after that there's not a lot for us to index. Therefore while you might observe several of those web pages with a targeted web site:- inquiry, the normal individual won't view them, so I definitely would not bother it. Noindex is additionally fine (without robots.txt disallow), it simply indicates the Links will certainly end up being crawled (and also find yourself in the Look Console report for crawled/not catalogued-- neither of these statuses induce issues to the rest of the web site). The vital part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the restrictions in using the Site: hunt evolved search operator for diagnostic reasons. Some of those explanations is because it is actually certainly not connected to the regular search index, it's a distinct trait altogether.Google's John Mueller commented on the internet site search operator in 2021:." The brief response is actually that a web site: query is certainly not indicated to become total, neither utilized for diagnostics purposes.A web site query is a certain kind of hunt that limits the results to a particular web site. It's generally merely the word site, a bowel, and after that the website's domain.This question confines the outcomes to a particular web site. It's not indicated to become a comprehensive compilation of all the web pages coming from that web site.".2. Noindex tag without utilizing a robots.txt is actually great for these sort of circumstances where a robot is linking to non-existent web pages that are acquiring discovered through Googlebot.3. URLs along with the noindex tag will certainly produce a "crawled/not listed" item in Search Console which those won't have an adverse effect on the rest of the website.Read the question and also respond to on LinkedIn:.Why will Google.com mark webpages when they can't even find the content?Featured Photo by Shutterstock/Krakenimages. com.