Skip Navigation

Block Referrer Spam (Updated)

Posted September 14, 2005 in Apache, Blogging, Spam

Log files are a useful tool for webmasters. It helps to know how people are finding your site, and what software they are using to view it, among other things. A strange decision by a small group of bloggers, though, has given unscrupulous marketers another window of opportunity to manipulate search engines to increase their traffic.

The decision made by these short-sighted bloggers was to display, on their site, a list of recent referrers to each page. I can't imagine any reason why a visitor might be in the least bit interested in seeing this, but nevertheless many sites now display referrers on every page.

As search engine spiders visit sites, they grab the contents of each page they visit. They use this snapshot in their index - meaning that although a page may change every minute or two, a search engine may be using a single copy of a page for several days, or even weeks.

So a referral URL that is on a page when the spiders come to visit can have quite a bit of value, if the search engine visiting uses link popularity in any way (Google uses link popularity, as do many others).

So marketers have started to use programs to visit pages using a fake referral header, to get their URLs listed on as many sites as possible, in the hopes that this will increase their traffic.

However, this renders log files almost completely useless. These fake visitors usually visit from search engines, having searched for a keyphrase relevant to their own site. They skew statistics relating to number of visitors received, the countries used to visit, the technology used to view the page, how users found the page, how long they spent on the site ... and so on.

A webmaster may find their search engine rankings dropping because of this, and they may find search engines have removed them completely. Many sites that use spam techniques are quickly identified and penalised, and penalties will often be applied to sites that link to them as well.

There are plenty of techniques available for blocking referrer spam, and everyone has their favourite. Personally, I use a combination of two techniques.

The first is fairly simple - my referrer log is not indexable. I don't display referrers on the pages of my site. My referral log is publicly available, but search engines are instructed to ignore it. This removes the main incentive for people to referrer-spam my site (the other reason for this type of spam - the hope that the site owner will themselves visit the spamming URL - is less common, because it has such a low response rate).

Second, I use an .htaccess file to block requests from whatever I've managed to identify as either a crawler designed to find URLs to spam or a spamming URL. This is a relatively simple blacklist, and though it cannot work as a long term solution to this problem, it keeps me happy for now.

To implement this technique on your own site, first make sure you are running Apache with mod_rewrite. If you are, create a file called ".htaccess" (just that, not .htaccess.txt or anything else) and paste the following into it:

Update: 14th September 2005

The list below has been expanded substantially over the last year, and now covers much more spam than before. As stated before, this is not a practical solution to the problem in the long term, as this list can only ever get longer and longer, and may become unmaintainable, or even (eventually) slow a site to a crawl as all the rules are processed. However, as of now, it is still a useful tool.

  1. RewriteEngine on
  2.  
  3. # Block Referrer Spam
  4.  
  5. # Drugs / Herbal
  6.  
  7. RewriteCond %{HTTP_REFERER} (sleep-?deprivation) [NC,OR]
  8. RewriteCond %{HTTP_REFERER} (sleep-?disorders) [NC,OR]
  9. RewriteCond %{HTTP_REFERER} (insomnia) [NC,OR]
  10. RewriteCond %{HTTP_REFERER} (phentermine) [NC,OR]
  11. RewriteCond %{HTTP_REFERER} (phentemine) [NC,OR]
  12. RewriteCond %{HTTP_REFERER} (vicodin) [NC,OR]
  13. RewriteCond %{HTTP_REFERER} (hydrocodone) [NC,OR]
  14. RewriteCond %{HTTP_REFERER} (levitra) [NC,OR]
  15. RewriteCond %{HTTP_REFERER} (hgh-) [NC,OR]
  16. RewriteCond %{HTTP_REFERER} (-hgh) [NC,OR]
  17. RewriteCond %{HTTP_REFERER} (ultram-) [NC,OR]
  18. RewriteCond %{HTTP_REFERER} (-ultram) [NC,OR]
  19. RewriteCond %{HTTP_REFERER} (cialis) [NC,OR]
  20. RewriteCond %{HTTP_REFERER} (soma-) [NC,OR]
  21. RewriteCond %{HTTP_REFERER} (-soma) [NC,OR]
  22. RewriteCond %{HTTP_REFERER} (diazepam) [NC,OR]
  23. RewriteCond %{HTTP_REFERER} (gabapentin) [NC,OR]
  24. RewriteCond %{HTTP_REFERER} (celebrex) [NC,OR]
  25. RewriteCond %{HTTP_REFERER} (viagra) [NC,OR]
  26. RewriteCond %{HTTP_REFERER} (fioricet) [NC,OR]
  27. RewriteCond %{HTTP_REFERER} (ambien) [NC,OR]
  28. RewriteCond %{HTTP_REFERER} (valium) [NC,OR]
  29. RewriteCond %{HTTP_REFERER} (zoloft) [NC,OR]
  30. RewriteCond %{HTTP_REFERER} (finasteride) [NC,OR]
  31. RewriteCond %{HTTP_REFERER} (lamisil) [NC,OR]
  32. RewriteCond %{HTTP_REFERER} (meridia) [NC,OR]
  33. RewriteCond %{HTTP_REFERER} (allegra) [NC,OR]
  34. RewriteCond %{HTTP_REFERER} (diflucan) [NC,OR]
  35. RewriteCond %{HTTP_REFERER} (zovirax) [NC,OR]
  36. RewriteCond %{HTTP_REFERER} (valtrex) [NC,OR]
  37. RewriteCond %{HTTP_REFERER} (lipitor) [NC,OR]
  38. RewriteCond %{HTTP_REFERER} (proscar) [NC,OR]
  39. RewriteCond %{HTTP_REFERER} (acyclovir) [NC,OR]
  40. RewriteCond %{HTTP_REFERER} (sildenafil) [NC,OR]
  41. RewriteCond %{HTTP_REFERER} (tadalafil) [NC,OR]
  42. RewriteCond %{HTTP_REFERER} (xenical) [NC,OR]
  43. RewriteCond %{HTTP_REFERER} (melatonin) [NC,OR]
  44. RewriteCond %{HTTP_REFERER} (xanax) [NC,OR]
  45. RewriteCond %{HTTP_REFERER} (herbal) [NC,OR]
  46. RewriteCond %{HTTP_REFERER} (drugs) [NC,OR]
  47. RewriteCond %{HTTP_REFERER} (lortab) [NC,OR]
  48. RewriteCond %{HTTP_REFERER} (adipex) [NC,OR]
  49. RewriteCond %{HTTP_REFERER} (propecia) [NC,OR]
  50. RewriteCond %{HTTP_REFERER} (carisoprodol) [NC,OR]
  51. RewriteCond %{HTTP_REFERER} (tramadol) [NC]
  52. RewriteRule .* - [F]
  53.  
  54. # Porn
  55.  
  56. RewriteCond %{HTTP_REFERER} (porno) [NC,OR]
  57. RewriteCond %{HTTP_REFERER} (shemale) [NC,OR]
  58. RewriteCond %{HTTP_REFERER} (gangbang) [NC,OR]
  59. RewriteCond %{HTTP_REFERER} (-cock) [NC,OR]
  60. RewriteCond %{HTTP_REFERER} (-anal) [NC,OR]
  61. RewriteCond %{HTTP_REFERER} (-orgy) [NC,OR]
  62. RewriteCond %{HTTP_REFERER} (cock-) [NC,OR]
  63. RewriteCond %{HTTP_REFERER} (anal-) [NC,OR]
  64. RewriteCond %{HTTP_REFERER} (orgy-) [NC,OR]
  65. RewriteCond %{HTTP_REFERER} (singles-?christian) [NC,OR]
  66. RewriteCond %{HTTP_REFERER} (dating-?christian) [NC,OR]
  67. RewriteCond %{HTTP_REFERER} (cumeating) [NC,OR]
  68. RewriteCond %{HTTP_REFERER} (cream-?pies) [NC,OR]
  69. RewriteCond %{HTTP_REFERER} (cumsucking) [NC,OR]
  70. RewriteCond %{HTTP_REFERER} (cumswapping) [NC,OR]
  71. RewriteCond %{HTTP_REFERER} (cumfilled) [NC,OR]
  72. RewriteCond %{HTTP_REFERER} (cumdripping) [NC,OR]
  73. RewriteCond %{HTTP_REFERER} (krankenversicherung) [NC,OR]
  74. RewriteCond %{HTTP_REFERER} (cumpussy) [NC,OR]
  75. RewriteCond %{HTTP_REFERER} (suckingcum) [NC,OR]
  76. RewriteCond %{HTTP_REFERER} (drippingcum) [NC,OR]
  77. RewriteCond %{HTTP_REFERER} (pussycum) [NC,OR]
  78. RewriteCond %{HTTP_REFERER} (swappingcum) [NC,OR]
  79. RewriteCond %{HTTP_REFERER} (eatingcum) [NC,OR]
  80. RewriteCond %{HTTP_REFERER} (cum-) [NC,OR]
  81. RewriteCond %{HTTP_REFERER} (-cum) [NC,OR]
  82. RewriteCond %{HTTP_REFERER} (sperm) [NC,OR]
  83. RewriteCond %{HTTP_REFERER} (christian-?dating) [NC,OR]
  84. RewriteCond %{HTTP_REFERER} (jewish-?singles) [NC,OR]
  85. RewriteCond %{HTTP_REFERER} (sex-?meetings) [NC,OR]
  86. RewriteCond %{HTTP_REFERER} (swinging) [NC,OR]
  87. RewriteCond %{HTTP_REFERER} (swingers) [NC,OR]
  88. RewriteCond %{HTTP_REFERER} (personals) [NC,OR]
  89. RewriteCond %{HTTP_REFERER} (sleeping) [NC,OR]
  90. RewriteCond %{HTTP_REFERER} (libido) [NC,OR]
  91. RewriteCond %{HTTP_REFERER} (grannies) [NC,OR]
  92. RewriteCond %{HTTP_REFERER} (mature) [NC,OR]
  93. RewriteCond %{HTTP_REFERER} (enhancement) [NC,OR]
  94. RewriteCond %{HTTP_REFERER} (sexual) [NC,OR]
  95. RewriteCond %{HTTP_REFERER} (gay-?teen) [NC,OR]
  96. RewriteCond %{HTTP_REFERER} (teen-?chat) [NC,OR]
  97. RewriteCond %{HTTP_REFERER} (gay-?chat) [NC,OR]
  98. RewriteCond %{HTTP_REFERER} (adult-?finder) [NC,OR]
  99. RewriteCond %{HTTP_REFERER} (adult-?friend) [NC,OR]
  100. RewriteCond %{HTTP_REFERER} (friend-?finder) [NC,OR]
  101. RewriteCond %{HTTP_REFERER} (friend-?adult) [NC,OR]
  102. RewriteCond %{HTTP_REFERER} (finder-?adult) [NC,OR]
  103. RewriteCond %{HTTP_REFERER} (finder-?friend) [NC,OR]
  104. RewriteCond %{HTTP_REFERER} (discrete-?encounters) [NC,OR]
  105. RewriteCond %{HTTP_REFERER} (cheating-?wives) [NC,OR]
  106. RewriteCond %{HTTP_REFERER} (housewives) [NC,OR]
  107. RewriteCond %{HTTP_REFERER} (\-sex\.) [NC,OR]
  108. RewriteCond %{HTTP_REFERER} (xxx) [NC,OR]
  109. RewriteCond %{HTTP_REFERER} (snowballing) [NC]
  110. RewriteRule .* - [F]
  111.  
  112. # Weight
  113.  
  114. RewriteCond %{HTTP_REFERER} (fat-) [NC,OR]
  115. RewriteCond %{HTTP_REFERER} (-fat) [NC,OR]
  116. RewriteCond %{HTTP_REFERER} (diet) [NC,OR]
  117. RewriteCond %{HTTP_REFERER} (pills) [NC,OR]
  118. RewriteCond %{HTTP_REFERER} (weight) [NC,OR]
  119. RewriteCond %{HTTP_REFERER} (supplement) [NC]
  120. RewriteRule .* - [F]
  121.  
  122. # Gambling
  123.  
  124. RewriteCond %{HTTP_REFERER} (texas-?hold-?em) [NC,OR]
  125. RewriteCond %{HTTP_REFERER} (poker) [NC,OR]
  126. RewriteCond %{HTTP_REFERER} (casino) [NC,OR]
  127. RewriteCond %{HTTP_REFERER} (blackjack) [NC]
  128. RewriteRule .* - [F]
  129.  
  130. # Loans / Finance
  131.  
  132. RewriteCond %{HTTP_REFERER} (mortgage) [NC,OR]
  133. RewriteCond %{HTTP_REFERER} (refinancing) [NC,OR]
  134. RewriteCond %{HTTP_REFERER} (cash-?advance) [NC,OR]
  135. RewriteCond %{HTTP_REFERER} (cash-?money) [NC,OR]
  136. RewriteCond %{HTTP_REFERER} (pay-?day) [NC]
  137. RewriteRule .* - [F]
  138.  
  139. # User Agents
  140.  
  141. RewriteCond %{HTTP_USER_AGENT} (Program\ Shareware|Fetch\ API\ Request) [NC,OR]
  142. RewriteCond %{HTTP_USER_AGENT} (Microsoft\ URL\ Control) [NC]
  143. RewriteRule .* - [F]
  144.  
  145. # Misc / Specific Sites
  146.  
  147. RewriteCond %{HTTP_REFERER} (netwasgroup\.com) [NC,OR]
  148. RewriteCond %{HTTP_REFERER} (nic4u\.com) [NC,OR]
  149. RewriteCond %{HTTP_REFERER} (wear4u\.com) [NC,OR]
  150. RewriteCond %{HTTP_REFERER} (foxmediasolutions\.com) [NC,OR]
  151. RewriteCond %{HTTP_REFERER} (liveplanets\.com) [NC,OR]
  152. RewriteCond %{HTTP_REFERER} (aeterna-tech\.com) [NC,OR]
  153. RewriteCond %{HTTP_REFERER} (continentaltirebowl\.com) [NC,OR]
  154. RewriteCond %{HTTP_REFERER} (chemsymphony\.com) [NC,OR]
  155. RewriteCond %{HTTP_REFERER} (infolibria\.com) [NC,OR]
  156. RewriteCond %{HTTP_REFERER} (globaleducationeurope\.net) [NC,OR]
  157. RewriteCond %{HTTP_REFERER} (soma\.125mb\.com) [NC,OR]
  158. RewriteCond %{HTTP_REFERER} (mitglied\.lycos\.de) [NC,OR]
  159. RewriteCond %{HTTP_REFERER} (foxmediasolutions\.com) [NC,OR]
  160. RewriteCond %{HTTP_REFERER} (jroundup\.com) [NC,OR]
  161. RewriteCond %{HTTP_REFERER} (feathersandfurvanlines\.com) [NC,OR]
  162. RewriteCond %{HTTP_REFERER} (conecrusher\.org) [NC,OR]
  163. RewriteCond %{HTTP_REFERER} (sbj-broadcasting\.com) [NC,OR]
  164. RewriteCond %{HTTP_REFERER} (edthompson\.com) [NC,OR]
  165. RewriteCond %{HTTP_REFERER} (codychesnutt\.com) [NC,OR]
  166. RewriteCond %{HTTP_REFERER} (artsmallforsenate\.com) [NC,OR]
  167. RewriteCond %{HTTP_REFERER} (axionfootwear\.com) [NC,OR]
  168. RewriteCond %{HTTP_REFERER} (protzonbeer\.com) [NC,OR]
  169. RewriteCond %{HTTP_REFERER} (candiria\.com) [NC,OR]
  170. RewriteCond %{HTTP_REFERER} (bigsitecity\.com) [NC,OR]
  171. RewriteCond %{HTTP_REFERER} (coresat\.com) [NC,OR]
  172. RewriteCond %{HTTP_REFERER} (istarthere\.com) [NC,OR]
  173. RewriteCond %{HTTP_REFERER} (amateurvoetbal\.net) [NC,OR]
  174. RewriteCond %{HTTP_REFERER} (alleghanyeda\.com) [NC,OR]
  175. RewriteCond %{HTTP_REFERER} (xadulthosting\.com) [NC,OR]
  176. RewriteCond %{HTTP_REFERER} (datashaping\.com) [NC,OR]
  177. RewriteCond %{HTTP_REFERER} (zick\.biz) [NC,OR]
  178. RewriteCond %{HTTP_REFERER} (newprinceton\.com) [NC,OR]
  179. RewriteCond %{HTTP_REFERER} (dvdsqueeze\.com) [NC,OR]
  180. RewriteCond %{HTTP_REFERER} (xopy\.com) [NC,OR]
  181. RewriteCond %{HTTP_REFERER} (webdevboard\.com) [NC,OR]
  182. RewriteCond %{HTTP_REFERER} (devaddict\.com) [NC,OR]
  183. RewriteCond %{HTTP_REFERER} (eaton-inc\.com) [NC,OR]
  184. RewriteCond %{HTTP_REFERER} (whiteguysgroup\.com) [NC,OR]
  185. RewriteCond %{HTTP_REFERER} (guestbookz\.com) [NC,OR]
  186. RewriteCond %{HTTP_REFERER} (webdevsquare\.com) [NC,OR]
  187. RewriteCond %{HTTP_REFERER} (indfx\.net) [NC,OR]
  188. RewriteCond %{HTTP_REFERER} (snap\.to) [NC,OR]
  189. RewriteCond %{HTTP_REFERER} (2y\.net) [NC,OR]
  190. RewriteCond %{HTTP_REFERER} (astromagia\.info) [NC,OR]
  191. RewriteCond %{HTTP_REFERER} (free-?sms) [NC]
  192. RewriteRule .* - [F]

The above will block just about all of the most common referral spam that I've seen so far. I'm adding to the list constantly (last addition: 14th September 2005) so do check back and see if there are updates if you're using it.

One potential problem with this technique, other than that it will, in time, become useless as too many URLs are added, is that there is always a possibility authentic visitors will be blocked. So, on this site, instead of the last line above, I've actually used something a little more user-friendly:

  1. RewriteRule .* bad_referrer.php [L]

Instead of a "Forbidden" message, this displays a quick note explaining why there has been an error and that the user can click on a link to proceed. If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. You should find you get a message explaining what has happened, and a URL to click if you want to proceed.

And there we have it. With minimum effort (for now), referral log spamming in my site has been almost entirely removed. Before adding this set of rules and scripts, I was seeing around 200 fake referrals per day in my log files. Now, I see about 3 or 4 a week. Hopefully, this will continue until I can devise a better way of protecting against this kind of problem - before blacklists become an impossibility to manage.

Block Referrer Spam (Updated) was posted September 14, 2005 in Apache, Blogging, Spam and has been tagged with spam, htaccess, apache, howto, referrer, admin, webdev, server and wordpress.

40 comments

Keep up to date with this discussion with RSS or Atom.
Keith
United Kingdom #1: January 7, 2005
[quote]If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. [/quote]

Both URLs just loaded the page without any additional message for me. Any idea why?
Keith
United Kingdom #2: January 7, 2005
[quote]If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. [/quote]

Both URLs just loaded the page without any additional message for me. Any idea why?
Because I worded it badly, I think - you need to click on the relevant link on each page. So from the article, you click on the "swingers" link, then from the "swingers" page, you click the "spam" link again.

You may also not be sending a referrer header, which will stop it working.
block requests that do not show a link to your page from the referrer page
It would be really swell if you'd post the code for bad_referrer.php.
I've noticed that 95% of the remaining referrer spam coming into my system had one thing in common: they were all looking for the file /adserver/campaign.php (which doesn't exist).

I've developed the following Apache mod_rewrite rule that seems to be quite effective, and just in time for the February log changeover:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} (adserver/campaign.php) [NC]
RewriteCond %{HTTP_REFERER} !=""
RewriteRule ^(.*) %{HTTP_REFERER} [R=301,L]

In plain English, what this says is that if the file requested contains "adserver/campaign.php", and that file doesn't exist on your server either as a file or a directory, and a referrer is set, redirect back to the referrer. Otherwise, proceed normally.
Michael, the code for bad_refferer.php is pretty basic:

echo '<strong>Bad Referrer</strong><br><br>Unfortunately, the URL you have visited from appears to be blocked from referring visitors to this site. But don\'t panic! The chances are that if you are a real person this was a mistake by the filtering system. If you want to carry on to the page you were trying to visit, please <a href="http://' . $_SERVER["HTTP_HOST"].$_SERVER["REQUEST_URI"] . '">click here</a>.';

Thanks for the addition, Zed. That looks like a pretty useful rule to deal with that sort of thing, and could easily be changed to block those requests for "_vti_bin" urls, etc.
Someone still has to pay for that bandwidth. And there's no reason to suspect they've not got a basic timeout written into the scripts that do the spamming.

Simply blocking them means they are invisible to me - they don't irritate me, and there's the minimum of inconvenience to the users. Which is exactly what I want ...
Somehow I can't get the bad_referrer.php to work. My server keeps giving me a "Bad request" 400 error.
I'm doing the blocking server-wide, so not from .htaccess, but from a separate block.conf in the conf.d directory of Apache 2.*
Do you have an idea on how to do this?
I tried an Alias, but the rewrite rules probably assume it can't be done just like that. Could you explain your code of bad-referrer.php a little more?
chris
United Kingdom #11: May 18, 2005
I'm getting a 400 error too... ah well ill keep on trucking.
what about not whowing the statistics altogether? put the referring page to a title or php-generate a GIF image with the referrer as a text in the image or simply password protect your stats.
Tyler
United States #13: July 6, 2005
Thanks for the tips. Worked well. poker-4all.com keeps pingback spamming and refer spamming me...
Thanks. One question: Does the speed suffer, when .htaccess is very large?
I found several words that were very common in a large amount of my referer spam. I decided to eliminate a chunk of the individual sites by blocking on that word. There is the risk that I will block someone legitimate but I am willing to take the chance to save myself from the annoyance.
<code>
RewriteCond %{HTTP_REFERER} (poker) [NC,OR]
RewriteCond %{HTTP_REFERER} (casino) [NC,OR]
RewriteCond %{HTTP_REFERER} (pharmacy) [NC,OR]
RewriteCond %{HTTP_REFERER} (inkjet) [NC,OR]
RewriteCond %{HTTP_REFERER} (blackjack) [NC,OR]
RewriteCond %{HTTP_REFERER} (diet) [NC,OR]
RewriteCond %{HTTP_REFERER} (drugs) [NC,OR]
RewriteCond %{HTTP_REFERER} (holdem) [NC,OR]
RewriteCond %{HTTP_REFERER} (mortgage) [NC,OR]
RewriteCond %{HTTP_REFERER} (loan) [NC,OR]
</code>
I did attempt to be selective of what words I did this with. I don't think many sites have inkjet in their url. Could be wrong but again it's worth the chance.
If anyone's interested, here's my listing;
http://jult.net/txt/blocks

I only use lists this size serverwide, not thru htaccess (that wat it's a load/CPU monster)
(that way it indeed is a load/CPU monster)
Does an excessive large .htaccess increase your bandwith usage? My site is down because of referrer-spam flooding. Does this solve thisproblem ?

Thanks for the works! great!
It won't increase your bandwidth but will make your site a little slower.
Thanks. It seems to work great. Is reflecting the spam to the referrer like this:

RewriteRule (.*) ^http://%{HTTP_REFERER}/$ [R=301,L]

A smart idea? So the referrers get the amount of data?
Struikel: It might work, however I've no idea if referrer spam bots are able to support 301 redirection (in fact, it's probably a good idea to test this - if they are unable to handle 301s, that would mean we could use that to filter bots from users.

If they did support 301s, redirecting the spam traffic back to the person responsible might well be a good idea.
Thanks for a good idea of blocking, but it works only partially for me. Even if i have words like "holdem" included in my blacklist, mod_rewrite passes through about one half of referrers. Shame...
ILoveJackDaniel, it seems the "buy viagra online"-Guy has managed to bypass your checks. he he he :)
Hi Jimmy.

The problem with posting techniques for dealing with spam is that you tend to become a target for it. I get the occasional piece of comment spam junk like the above (now deleted) and lots of referrer spam that's usually blocked.
Man, I'm having some serious problems getting this to work. I've tried everything, and it just won't do it.

My current .htaccess looks like this:

-------------

DirectoryIndex index.html index.htm index.php index.shtml
AddType application/x-httpd-php .html .htm

RewriteEngine On
addhandler server-parsed html sssi page shtml htm

<limit GET POST PUT>
order deny,allow
deny from netwasgroup.com
deny from nic4u.com
deny from foto-porno-amatoriale.com
deny from video-porno-anale.com
deny from sborra-sopra-piedi.com
deny from piedi-feticismo.com
deny from sesso-vero-amatoriale.com
deny from sborrate-in-faccia.com
deny from lesbiche-sesso.net
deny from sesso-orale-gratis.net
deny from lesbiche-sesso.net
deny from goodcounter.net
deny from sborra-sopra-piedi.com
allow from all
</limit>

----------

But none of those sites listed are being blocked.

Can anyone help?
Can this be added to a current htaccess list I have at the bottom of it. As I dont want it to affect the current IP's that i block.
Darrel: Yes, as long as you don't repeat the "RewriteEngine on" bit.
Jack
United States #28: December 1, 2005
Thanks for the rewrite rules!
Rick
United States #29: December 5, 2005
Hi-
This all looks fine, and I've been working on this for days...to block these referer spammers.
But -- somehow, I'm not able to stop logging the fact that these spammers are getting "301", and my logs are filling as before.
(Trying to block a non-existent refer/index.php)
What am I missing?
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} (/refer/index.php) [NC,OR]
RewriteCond %{REQUEST_URI} (/refer) [NC]
RewriteCond %{HTTP_REFERER} !=""
RewriteRule ^(.*) %{HTTP_REFERER} [R=301,L]
For the last month, our site has been getting pulverized with referral spam. We have a blog which is highly ranked in search engines. As a result, we were getting attacked by spammers to the point of over 7000 hits per day by each spammer. Nearly crippled our server. After doing some research and scouring the net, I found this site. THANK YOU SO MUCH! We are now using .htaccess to filter out the junk. It works like a champ! For the first time in several months our server is fast and we have much better control over referral spam. It took a while to get it to dial our .htaccess file in but well worth the time. Thanks again for your great resource.
Bruce MacKay
Canada #31: January 23, 2006
My daughter maintained her robotics blog on my site (http://synysys.com/roboblog) for several months until the project came to conclusion and was eventually taken off line. During it's life span, it fell to a bit of neglect and became the target of the referrer spam bots. Today our site still gets over a thousand hits per day looking for the old exploit.

I know this is a bit like closing the barn door after the cows have all run off and perhaps in this case even part of the barn burned down, but I thought I'd share the solution I hacked together today. I wasn't satisfied with a purely mod_rewrite solution since as others have noted, you still get a one line log entry in you access log. Essentially my solution is a two pronged approach. First it uses mod_rewrite to redirect the spammer back to their own machine. Second it puts a DROP entry in my firewall so that they won't be coming back to visit again any time soon. That way my logs aren't filling up with the same old rewrites over and over.

The entry in httpd.conf looks like this

RewriteEngine on
RewriteCond %{QUERY_STRING} disp=stats
RewriteMap referer-deny prg:/etc/httpd/refererdeny.pl
RewriteRule ^(.*)$ ${referer-deny:%
{REMOTE_ADDR}}/BITE_ME_SPAMMER? [R,L]

In my case it was a particular query string that typified the bulk of the spam traffic, but you can add other patterns to the above rewrite conditions to suit your own needs.

The PERL script looks like this

#!/usr/bin/perl
$| = 1; # Turn off buffering
while (<STDIN>) {
print "HTTP://",$_;
$b = ("/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s $_");
system ($b);
open (OUTFILE,">>/etc/httpd/referer.deny");
print OUTFILE ("$b");
close (OUTFILE);
}

The referer.deny "log" looks like this

/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s 219.93.21.20
/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s 220.165.140.8
/sbin/iptables -A INPUT -j DROP -p tcp --destination-port 80 -s 83.100.149.29

So you could easily add #!/bin/sh to the head of it and run it as a shell script separately if you wanted. However, before you do that, you should probably sort the file and remove any duplicate entries that may have crept in. I have chosen to only block access to port 80. You could easily add port 25 or even remove the destination-port all together and completely block them from your site. Just be aware that some clever fool could forge your IP and potentially block you from your own site. Of course you could reduce the output to the log file by substituting $_ for $b and just end up with a list of blocked IPs.

I realize that not all site admins have root access to be able to run the firewall commands, so you might modify this to update a hosts.deny file that you've defined in your own .htaccess configuration. The point is you don't really want to have to manually enter every IP or host name if you are really getting bombarded. Again if you do this, you'll probably want to sort the file and remove dupes. I'd also recommend that us the DB utility to speed your lookups if you end up with a significant number of blocked hosts. You really don't want to bog down your site with lookups on account of these spammer fools.

Of course one of the problems that I alluded to earlier is that you may end up with unwanted blocks defined in your system. Most hosting environments offer CRON access. You might choose to flush the firewall rules over some period. Many of the spammers are running client based tools from dynamic IP pools on the ISP. Over time you could end up blocking a significant number of IPs that were only used once against your system. Since this system is automated, it's probably safer to clear it out periodically and let it repopulate itself with the bad apples that keep coming back.

I hope this helps someone. It seems to be working wonderfully for our site. My daughter's robotics project was archived as a PDF for those who are looking for it and the spammers trying to exploit the referrer logs aren't stealing my bandwidth or chewing up file space with senseless logs any longer.

Bruce
Thx ! was really useful !!! i had like 100 referrr spam daily. !!!
I've been running a somewhat modified system to what I documented in comment #31. It's currently blocking 3208 IPs from people who have behaved badly on my system (mostly attempts at referrer spam, but that number also includes SSHD and misc script probes against my HTTPD) The firewall easily handles this many blocks and my web server is much happier with the reduction in load.

The system responds in real time to these attacks, gives them a custom 403 Error page and then blocks their IP. The custom 403 Error page is for non-script users who may be blocked inadvertently. It has a link to a recovery system which unblocks their IP and restores their access. Of course a bot doesn't follow the on-screen instructions and even if it did, it would just get blocked when it started behaving badly. All in all, it seems to be quite an effective solution.

For folks who don't have admin access to the firewall
on their server, the system is still quite effective, but you will continue to see all of the attempts in your httpd logs.

If anyone is interested in further details, I'd be more than happy to discuss this via chat or email. You can contact me on www.synysys.com. Anything that we can do to slow these idiots down is a step in the right direction.

Bruce
But still, how are you gonna catch ref.spam with just a list of descriptions? It's an endless road I'm travelling, and I'm getting quite fed up with these idiots doing this.

# the biggest losers ever ( they can't even spell: )
RewriteCond %{HTTP_REFERER} (-nude\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (abrianna\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (amanti\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (anali\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (burdizzo\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (bucetinhas\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (calcinha\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (cogidas\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (esibizioniste\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (fimosis\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (folladas\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (gotico\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (gozadas\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (hargitay\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (loredana\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (mamando\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (minifalda\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (plumprumps\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (porono\.) [NC,OR]
RewriteCond %{HTTP_REFERER} (ramalan\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (stretched\.org) [NC,OR]
RewriteCond %{HTTP_REFERER} (subsonica\.org) [NC,OR]
CortalUX
United Kingdom #35: May 7, 2006
Dammit, I really hate spam.
Rob
United States #36: May 27, 2006
Hey folks, sorry for the spam here but i'm wondering how you can tell if this is working? I found a couple sites referring to different ways to do what you're saying. One is a .conf file in /path/to/conf.d/ where it works w/ apache and other is .htaccess. What i'm seeing is a ton of referrer spam in our access logs which doesn't belong. It's forged as it's asking for sites that our apache server doesn't host. I would like to get this out of my log file and put the kabosh on the spamming offender by simply blocking their access (if possible). Any suggestions here?
Rob
United States #37: June 8, 2006
I though some folks might like to take a look at this site for referrer spam:
http://unknowngenius.com/blog/wordpress/ref-karma/

he wrote a neat php script to automate updating referrer blocks.
I had a wrong link coming in from a wrong place. This technique was the cure.
Sentimental and nostalgic. Great.
I geta TON of spams on my FAQ pages under comments. I have tried to implement this on my site but I can't seeing as how I use vhosts. Any way around this?

Comments Disabled

Sorry, but comments are no longer being accepted on this post. This is usually because a post has become out of date, or has become the target for an unusually high quantity of comment spam.