Answers to Can Search Engine read robots.txt if it's read access is restricted? ( 3 )

  1. 2016-08-03 15:08

    This file should be freely readable. Search engine are like visitors on your website. If a visitor can't see this file, then the search engine will not be able to see it either.

    There's absolutely no reason to try to hide this file.

  2. 2016-08-03 15:08

    Web crawlers need to be able to HTTP GET your robots.txt, or they will be unable to parse the file and respect your configuration.

  3. 2016-08-03 15:08

    The answer is no! But the simplest and safest too, is still to try:
    https://support.google.com/webmasters/answer/6062598?hl=en

    The robots.txt Tester tool shows you whether your robots.txt file blocks Google web crawlers from specific URLs on your site. For example, you can use this tool to test whether the Googlebot-Image crawler can crawl the URL of an image you wish to block from Google Image Search.

Leave a reply to - Can Search Engine read robots.txt if it's read access is restricted?

◀ Go back