disallow some image folders

Question

I am making my robots.txt file. But I am a little bit insecure about how to make disallow Googlebot-Image. I want to allow the Google bot to crawl my site, except for the disallow I have made below. This is what I made:

User-agent: Googlebot
Disallow:

Disallow: /courses/
Disallow: /portfolio/portfolio-template.php/

User-agent: Googlebot-Image
Disallow: 
/images/graphics/erhvervserfaring/
/images/graphics/uddannelse/

sitemap: http://www.example.com/sitemap.xml

Should the User-agent: Googlebot and User-agent: Googlebot-Image be written together, so it is this instead?:

User-agent: Googlebot-Image User-agent: Googlebot Disallow:

Disallow: /courses/
          /portfolio/portfolio-template.php/
          /images/graphics/erhvervserfaring/
          /images/graphics/uddannelse/

Show source
| seo   | web-crawler   | robots.txt   2016-12-30 18:12 1 Answers

Answers ( 1 )

  1. 2016-12-30 18:12

    No, you should write them separately along with the Disallow information. Also, you should copy the Disallow information too.

    User-agent: Googlebot-Image
    Disallow: /courses/
    Disallow: /portfolio/portfolio-template.php/
    Disallow: /images/graphics/erhvervserfaring/
    Disallow: /images/graphics/uddannelse/
    
    User-agent: Googlebot
    Disallow: /courses/
    Disallow: /portfolio/portfolio-template.php/
    Disallow: /images/graphics/erhvervserfaring/
    Disallow: /images/graphics/uddannelse/
    

    As a reference, you can see what Facebook and what apple did in their robots.txt.

◀ Go back