# $Id: robots.txt,v 1.9.2.1 2008/12/10 20:12:19 goba Exp $## robots.txt## This file is to prevent the crawling and indexing of certain parts# of your site by web crawlers and spiders run by sites like Yahoo!# and Google. By telling these "robots" where not to go on your site,# you save bandwidth and server resources.## This file will be ignored unless it is at the root of your host:# Used:    http://example.com/robots.txt# Ignored: http://example.com/site/robots.txt## For more information about the robots.txt standard, see:# http://www.robotstxt.org/wc/robots.html## For syntax checking, see:# http://www.sxw.org.uk/computing/robots/check.htmlUser-agent: *Crawl-delay: 10# DirectoriesDisallow: /links/Disallow: /forum/Disallow: /cms2/Disallow: /cms/Disallow: /jtbits-com/Disallow: /design-jtbits-com/
Disallow: /backup-jtbits-com/Disallow: /tjbits-com/Disallow: /krondor-org/Disallow: /novindus-com/Disallow: /raymondfeistbooks-com/Disallow: /riftwarsaga-com/Disallow: /rillanon-com/Disallow: /triagia-com/Disallow: /includes/Disallow: /misc/Disallow: /modules/Disallow: /profiles/Disallow: /scripts/Disallow: /sites/Disallow: /themes/# FilesDisallow: /CHANGELOG.txtDisallow: /cron.phpDisallow: /INSTALL.mysql.txtDisallow: /INSTALL.pgsql.txtDisallow: /install.phpDisallow: /INSTALL.txtDisallow: /LICENSE.txtDisallow: /MAINTAINERS.txtDisallow: /update.phpDisallow: /UPGRADE.txtDisallow: /xmlrpc.php# Paths (clean URLs)Disallow: /admin/Disallow: /comment/reply/Disallow: /contact/Disallow: /logout/Disallow: /node/add/Disallow: /search/Disallow: /user/register/Disallow: /user/password/Disallow: /user/login/# Paths (no clean URLs)Disallow: /?q=admin/Disallow: /?q=comment/reply/Disallow: /?q=contact/Disallow: /?q=logout/Disallow: /?q=node/add/Disallow: /?q=search/Disallow: /?q=user/password/Disallow: /?q=user/register/Disallow: /?q=user/login/