User-agent: dotbot Disallow: / User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /help/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ # note to site owner from server admins: # we noticed a large spike in MySQL traffic foryour site, and checking the # logs it seems that the DotBot spider is ignoring your robots.txt file and # spidering your complete site # # I've added the lines above, but you may want to think about using a # .htaccess file to clock their IP (208.115.111.251)