This allows all robots to crawl all files except the images file and the stats file. User-agent: * Disallow: /cgi-bin/ Disallow: /dbconn/ Disallow: /gfx/ Disallow: /images/ Disallow: /js/