User-agent: * Disallow: /admin/ Disallow: /contrib/ Disallow: /doc/ Disallow: /lib/ Disallow: /modules/ Disallow: /plugins/ Disallow: /scripts/ Disallow: /tmp/ # Spammer User-agent: LinkWalker Disallow: / # archiver - no thanks #User-agent: ia_archiver #Disallow: / # spammer User-Agent: sitecheck.internetseer.com Disallow: / # company letting crawl the community: nono User-agent: grub-client Disallow: /