Block robots from accessing webpages using .htaccess

How to block

Use htaccess file.

this is run by server and ensured by the server that the rules written in it are enforced.

On the other hand, robots.txt file is just a guideline file. Robots are not bound to follow it.

To ensure denial of all requests for the restricted directory, put the following in the .htaccess file (keep it in the same directory),

deny from all

Allow access from one IP only and deny others,

order deny, allow
deny from all
allow from 255.255.12.34
ErrorDocument 403 errordoc.html

Allow access from a range of IPs (may be a LAN or a country),

order deny, allow
deny from all
allow from 255.255.0

Deny access from certain IP (e.g. a bot),

order allow, deny
deny from 255.255.123.456
allow from all

Block a bad bot

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^dirtybot
RewriteRule ^(.*)$ http://emptypage/

or use [F] to send Forbidden signal

RewriteRule .* – [F]

Block more that one bots

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^dirtybot [OR]
RewriteCond %{HTTP_USER_AGENT} ^badCrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^fakeOne
RewriteRule ^(.*)$ http://emptypage/

Deny more than one IPs or bot IPs

order allow,deny
deny from 255.255.123.456
deny from 123.255.123.456
deny from 456.255.123.456
allow from all

Deny more than one IPs or bot IPs ranges (may be countries)

order allow,deny
deny from 255.255.
deny from 123.456.
allow from all

Block particular ISPs,

order allow,deny
deny from this-bad-isp.com
deny from subdomain.bad-isp.com
allow from all

Block some referers or websites from accessing your content like images or css or js files

RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteRule .* - [F]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>