« Not Pron Spoilers 1 to 5 | Main | What color is an octopus? »

Blocking Unwanted Robots

Home » Programming » Mod_rewrite
Home » Internet » SEO

Spam is a problem, posting entries to blogs with email address should be avoided. There are lots of automated programs used to collect email addresses; other than spam, bandwidth may also be an issue for these programs reads your entire website. If you only have a small bandwidth allocated to your site then you will be seeing that Bandwidth Limit Error in due time.

What I did? Blocking all unwanted robots out of my site using mod_rewrite by apache. First, you need to examine your access log file ang try to google on the robots that has visited your site if they are safe or just they are just scrapers. Just be carefull not to block those major search engine spiders like googlebot, inktomi slurp, msnbot or ask jeeves. Unless you don't want them crawl your website.

You need to modify your .htaccess file to block unwanted robots from scraping your website by:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Link[Ww]alker [OR]
RewriteRule ^.* - [F]
</IfModule mod_rewrite.c>

The above code tells the spiders Siphon and LinkWalker that they are not allowed on our website by returning a 403 Forbidden Error.

There are also good robots, most of them are used for link checking, so redirecting them to the proper areas would be a better solution.

<IfModule mod_rewrite.c>
RewriteCond %{HTTP_USER_AGENT} reciprocalman [OR]
RewriteCond %{HTTP_USER_AGENT} LinksManager.com_bot
RewriteRule ^$ /resources/
</IfModule mod_rewrite.c>

The code above tells the reciprocalman and the LinksManager.com_bot to go directly to the resources directory.

TrackBack

TrackBack URL for this entry:
http://www.tildemark.com/cgi-bin/mt/mt-tb.cgi/27

About

This page contains a single entry from the blog posted on September 13, 2006 3:00 PM .

The previous post in this blog was Not Pron Spoilers 1 to 5 .

The next post in this blog is What color is an octopus? .

Many more can be found on the main index page or by looking through the archives .

Creative Commons License
This weblog is licensed under a Creative Commons License .
Powered by
Movable Type 3.33