.Google.com's John Mueller responded to an inquiry concerning why Google.com indexes webpages that are actually refused from crawling through robots.txt as well as why the it's risk-free to dismiss the related Browse Console files regarding those creeps.Bot Web Traffic To Inquiry Parameter URLs.The individual inquiring the concern chronicled that robots were actually generating links to non-existent inquiry criterion URLs (? q= xyz) to web pages with noindex meta tags that are actually additionally shut out in robots.txt. What prompted the question is actually that Google is crawling the web links to those web pages, getting blocked by robots.txt (without watching a noindex robots meta tag) then getting shown up in Google.com Browse Console as "Indexed, though shut out by robots.txt.".The person talked to the observing question:." Yet right here is actually the large inquiry: why would Google index web pages when they can not even observe the material? What's the perk in that?".Google's John Mueller validated that if they can't creep the web page they can't observe the noindex meta tag. He additionally creates an interesting mention of the internet site: hunt operator, advising to ignore the results since the "common" consumers will not find those outcomes.He created:." Yes, you are actually appropriate: if our experts can't creep the page, we can not observe the noindex. That said, if our experts can't crawl the webpages, after that there is actually not a whole lot for our company to index. Therefore while you may see some of those web pages with a targeted website:- concern, the normal individual will not view all of them, so I wouldn't fuss over it. Noindex is actually also fine (without robots.txt disallow), it only suggests the Links will certainly wind up being actually crawled (as well as wind up in the Explore Console file for crawled/not indexed-- neither of these statuses lead to concerns to the remainder of the site). The integral part is actually that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the constraints in using the Web site: search progressed hunt operator for diagnostic causes. Some of those main reasons is actually since it is actually certainly not linked to the normal hunt index, it's a separate point completely.Google.com's John Mueller commented on the internet site hunt operator in 2021:." The brief solution is that a website: query is not implied to be comprehensive, neither utilized for diagnostics purposes.A site concern is a details sort of search that confines the end results to a certain site. It is actually primarily just words web site, a colon, and afterwards the web site's domain name.This question limits the outcomes to a details site. It's not indicated to be a comprehensive assortment of all the web pages from that site.".2. Noindex tag without making use of a robots.txt is actually fine for these kinds of scenarios where a crawler is connecting to non-existent pages that are getting uncovered by Googlebot.3. Links along with the noindex tag will create a "crawled/not recorded" item in Search Console and also those will not have a damaging impact on the remainder of the web site.Check out the question as well as answer on LinkedIn:.Why will Google.com index pages when they can not even observe the information?Featured Image by Shutterstock/Krakenimages. com.