Block Referrer Spam (Updated)

[quote]If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. [/quote]

Both URLs just loaded the page without any additional message for me. Any idea why?

#1, Keith, United Kingdom, 7 January 2005. Reply to this.

Because I worded it badly, I think - you need to click on the relevant link on each page. So from the article, you click on the "swingers" link, then from the "swingers" page, you click the "spam" link again.

You may also not be sending a referrer header, which will stop it working.

#3, Dave Child, United Kingdom, 7 January 2005. Reply to this.

It would be really swell if you'd post the code for bad_referrer.php.

#5, Michael Ditto, United States, 1 February 2005. Reply to this.

Michael, the code for bad_refferer.php is pretty basic:

echo '<strong>Bad Referrer</strong><br><br>Unfortunately, the URL you have visited from appears to be blocked from referring visitors to this site. But don\'t panic! The chances are that if you are a real person this was a mistake by the filtering system. If you want to carry on to the page you were trying to visit, please <a href="http://' . $_SERVER["HTTP_HOST"].$_SERVER["REQUEST_URI"] . '">click here</a>.';

Thanks for the addition, Zed. That looks like a pretty useful rule to deal with that sort of thing, and could easily be changed to block those requests for "_vti_bin" urls, etc.

#7, Dave Child, United Kingdom, 1 February 2005. Reply to this.

Someone still has to pay for that bandwidth. And there's no reason to suspect they've not got a basic timeout written into the scripts that do the spamming.

Simply blocking them means they are invisible to me - they don't irritate me, and there's the minimum of inconvenience to the users. Which is exactly what I want ...

#9, Dave Child, United Kingdom, 8 February 2005. Reply to this.

I'm getting a 400 error too... ah well ill keep on trucking.

#11, chris, United Kingdom, 18 May 2005. Reply to this.

Thanks for the tips. Worked well. poker-4all.com keeps pingback spamming and refer spamming me...

#13, Tyler, United States, 6 July 2005. Reply to this.

I found several words that were very common in a large amount of my referer spam. I decided to eliminate a chunk of the individual sites by blocking on that word. There is the risk that I will block someone legitimate but I am willing to take the chance to save myself from the annoyance.
<code>
RewriteCond %{HTTP_REFERER} (poker) [NC,OR]
RewriteCond %{HTTP_REFERER} (casino) [NC,OR]
RewriteCond %{HTTP_REFERER} (pharmacy) [NC,OR]
RewriteCond %{HTTP_REFERER} (inkjet) [NC,OR]
RewriteCond %{HTTP_REFERER} (blackjack) [NC,OR]
RewriteCond %{HTTP_REFERER} (diet) [NC,OR]
RewriteCond %{HTTP_REFERER} (drugs) [NC,OR]
RewriteCond %{HTTP_REFERER} (holdem) [NC,OR]
RewriteCond %{HTTP_REFERER} (mortgage) [NC,OR]
RewriteCond %{HTTP_REFERER} (loan) [NC,OR]
</code>
I did attempt to be selective of what words I did this with. I don't think many sites have inkjet in their url. Could be wrong but again it's worth the chance.

#15, Tim, United States, 14 September 2005. Reply to this.

(that way it indeed is a load/CPU monster)

#17, Julius B. Thyssen, Netherlands, 24 September 2005. Reply to this.

It won't increase your bandwidth but will make your site a little slower.

#19, Dave Child, United Kingdom, 17 October 2005. Reply to this.

Struikel: It might work, however I've no idea if referrer spam bots are able to support 301 redirection (in fact, it's probably a good idea to test this - if they are unable to handle 301s, that would mean we could use that to filter bots from users.

If they did support 301s, redirecting the spam traffic back to the person responsible might well be a good idea.

#21, Dave Child, United Kingdom, 18 October 2005. Reply to this.

ILoveJackDaniel, it seems the "buy viagra online"-Guy has managed to bypass your checks. he he he :)

#23, Jimmy Flirten, Germany, 26 October 2005. Reply to this.

Man, I'm having some serious problems getting this to work. I've tried everything, and it just won't do it.

My current .htaccess looks like this:

-------------

DirectoryIndex index.html index.htm index.php index.shtml
AddType application/x-httpd-php .html .htm

RewriteEngine On
addhandler server-parsed html sssi page shtml htm

<limit GET POST PUT>
order deny,allow
deny from netwasgroup.com
deny from nic4u.com
deny from foto-porno-amatoriale.com
deny from video-porno-anale.com
deny from sborra-sopra-piedi.com
deny from piedi-feticismo.com
deny from sesso-vero-amatoriale.com
deny from sborrate-in-faccia.com
deny from lesbiche-sesso.net
deny from sesso-orale-gratis.net
deny from lesbiche-sesso.net
deny from goodcounter.net
deny from sborra-sopra-piedi.com
allow from all
</limit>

----------

But none of those sites listed are being blocked.

Can anyone help?

#25, Martin, United Kingdom, 14 November 2005. Reply to this.

Darrel: Yes, as long as you don't repeat the "RewriteEngine on" bit.

#27, Dave Child, United Kingdom, 23 November 2005. Reply to this.

Hi-
This all looks fine, and I've been working on this for days...to block these referer spammers.
But -- somehow, I'm not able to stop logging the fact that these spammers are getting "301", and my logs are filling as before.
(Trying to block a non-existent refer/index.php)
What am I missing?
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} (/refer/index.php) [NC,OR]
RewriteCond %{REQUEST_URI} (/refer) [NC]
RewriteCond %{HTTP_REFERER} !=""
RewriteRule ^(.*) %{HTTP_REFERER} [R=301,L]

#29, Rick, United States, 5 December 2005. Reply to this.

My daughter maintained her robotics blog on my site (http://synysys.com/roboblog) for several months until the project came to conclusion and was eventually taken off line. During it's life span, it fell to a bit of neglect and became the target of the referrer spam bots. Today our site still gets over a thousand hits per day looking for the old exploit.

I know this is a bit like closing the barn door after the cows have all run off and perhaps in this case even part of the barn burned down, but I thought I'd share the solution I hacked together today. I wasn't satisfied with a purely mod_rewrite solution since as others have noted, you still get a one line log entry in you access log. Essentially my solution is a two pronged approach. First it uses mod_rewrite to redirect the spammer back to their own machine. Second it puts a DROP entry in my firewall so that they won't be coming back to visit again any time soon. That way my logs aren't filling up with the same old rewrites over and over.

The entry in httpd.conf looks like this

RewriteEngine on
RewriteCond %{QUERY_STRING} disp=stats
RewriteMap referer-deny prg:/etc/httpd/refererdeny.pl
RewriteRule ^(.*)$ ${referer-deny:%
{REMOTE_ADDR}}/BITE_ME_SPAMMER? [R,L]

In my case it was a particular query string that typified the bulk of the spam traffic, but you can add other patterns to the above rewrite conditions to suit your own needs.

The PERL script looks like this

#!/usr/bin/perl
$| = 1; # Turn off buffering
while (<STDIN>) {
print "HTTP://",$_;
$b = ("/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s $_");
system ($b);
open (OUTFILE,">>/etc/httpd/referer.deny");
print OUTFILE ("$b");
close (OUTFILE);
}

The referer.deny "log" looks like this

/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s 219.93.21.20
/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s 220.165.140.8
/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s 83.100.149.29

So you could easily add #!/bin/sh to the head of it and run it as a shell script separately if you wanted. However, before you do that, you should probably sort the file and remove any duplicate entries that may have crept in. I have chosen to only block access to port 80. You could easily add port 25 or even remove the destination-port all together and completely block them from your site. Just be aware that some clever fool could forge your IP and potentially block you from your own site. Of course you could reduce the output to the log file by substituting $_ for $b and just end up with a list of blocked IPs.

I realize that not all site admins have root access to be able to run the firewall commands, so you might modify this to update a hosts.deny file that you've defined in your own .htaccess configuration. The point is you don't really want to have to manually enter every IP or host name if you are really getting bombarded. Again if you do this, you'll probably want to sort the file and remove dupes. I'd also recommend that us the DB utility to speed your lookups if you end up with a significant number of blocked hosts. You really don't want to bog down your site with lookups on account of these spammer fools.

Of course one of the problems that I alluded to earlier is that you may end up with unwanted blocks defined in your system. Most hosting environments offer CRON access. You might choose to flush the firewall rules over some period. Many of the spammers are running client based tools from dynamic IP pools on the ISP. Over time you could end up blocking a significant number of IPs that were only used once against your system. Since this system is automated, it's probably safer to clear it out periodically and let it repopulate itself with the bad apples that keep coming back.

I hope this helps someone. It seems to be working wonderfully for our site. My daughter's robotics project was archived as a PDF for those who are looking for it and the spammers trying to exploit the referrer logs aren't stealing my bandwidth or chewing up file space with senseless logs any longer.

Bruce

#31, Bruce MacKay, Canada, 23 January 2006. Reply to this.

I've been running a somewhat modified system to what I documented in comment #31. It's currently blocking 3208 IPs from people who have behaved badly on my system (mostly attempts at referrer spam, but that number also includes SSHD and misc script probes against my HTTPD) The firewall easily handles this many blocks and my web server is much happier with the reduction in load.

The system responds in real time to these attacks, gives them a custom 403 Error page and then blocks their IP. The custom 403 Error page is for non-script users who may be blocked inadvertently. It has a link to a recovery system which unblocks their IP and restores their access. Of course a bot doesn't follow the on-screen instructions and even if it did, it would just get blocked when it started behaving badly. All in all, it seems to be quite an effective solution.

For folks who don't have admin access to the firewall
on their server, the system is still quite effective, but you will continue to see all of the attempts in your httpd logs.

If anyone is interested in further details, I'd be more than happy to discuss this via chat or email. You can contact me on www.synysys.com. Anything that we can do to slow these idiots down is a step in the right direction.

Bruce

#33, Bruce, Canada, 4 March 2006. Reply to this.

Dammit, I really hate spam.

#35, CortalUX, United Kingdom, 7 May 2006. Reply to this.

I though some folks might like to take a look at this site for referrer spam:
http://unknowngenius.com/blog/wordpress/ref-karma/

he wrote a neat php script to automate updating referrer blocks.

#37, Rob, United States, 8 June 2006. Reply to this.

Sentimental and nostalgic. Great.

#39, Vasilii, Japan, 2 October 2006. Reply to this.

I've recently been getting a ton of referral spam. I've taken a bit of a different approach. I've implemented a little php, that does two things. The first, is check for a cookie. If that cookie doesn't exist, I set it. I then set another cookie with with a special value for reference, such as part of my sites domain name. The second check, is for the referrer itself. If its empty, I don't load the rest of the page and give them a chance to fix it, then click a link that will ensure the referrer came from my own site. If they spoof the referrer, they end up in an endless cycle of click here, etc.

The main problem I see with todays referrer spam, is SEO Blackhats don't care about getting it in your logs. They care about getting in into google analytics. This ensures them that google will harvest their links and search terms that came to your site. Asside from email and comment spamming, I see referral spamming as the next biggest threat to website.

With my php version, you can also do a count added to your cookies. If they reach a certain threshold, you can header location them off your site, either to the referral they try to spam at you, or wherever you want. ;)

#41, DigiP, 17 August 2011. Reply to this.