Tagged with "apache"

Block Referrer Spam (Updated)

Dave Child — Wed, 14 Sep 2005 11:36:00 +0100

Referrer spam is becoming increasingly common. At best, it will only render your log files useless. At worst, it can cause your site to be dropped by search engines and your running costs to skyrocket. Here's how to block spurious referrers.

Log files are a useful tool for webmasters. It helps to know how people are finding your site, and what software they are using to view it, among other things. A strange decision by a small group of bloggers, though, has given unscrupulous marketers another window of opportunity to manipulate search engines to increase their traffic.

The decision made by these short-sighted bloggers was to display, on their site, a list of recent referrers to each page. I can't imagine any reason why a visitor might be in the least bit interested in seeing this, but nevertheless many sites now display referrers on every page.

As search engine spiders visit sites, they grab the contents of each page they visit. They use this snapshot in their index - meaning that although a page may change every minute or two, a search engine may be using a single copy of a page for several days, or even weeks.

So a referral URL that is on a page when the spiders come to visit can have quite a bit of value, if the search engine visiting uses link popularity in any way (Google uses link popularity, as do many others).

So marketers have started to use programs to visit pages using a fake referral header, to get their URLs listed on as many sites as possible, in the hopes that this will increase their traffic.

However, this renders log files almost completely useless. These fake visitors usually visit from search engines, having searched for a keyphrase relevant to their own site. They skew statistics relating to number of visitors received, the countries used to visit, the technology used to view the page, how users found the page, how long they spent on the site ... and so on.

A webmaster may find their search engine rankings dropping because of this, and they may find search engines have removed them completely. Many sites that use spam techniques are quickly identified and penalised, and penalties will often be applied to sites that link to them as well.

There are plenty of techniques available for blocking referrer spam, and everyone has their favourite. Personally, I use a combination of two techniques.

The first is fairly simple - my referrer log is not indexable. I don't display referrers on the pages of my site. My referral log is publicly available, but search engines are instructed to ignore it. This removes the main incentive for people to referrer-spam my site (the other reason for this type of spam - the hope that the site owner will themselves visit the spamming URL - is less common, because it has such a low response rate).

Second, I use an .htaccess file to block requests from whatever I've managed to identify as either a crawler designed to find URLs to spam or a spamming URL. This is a relatively simple blacklist, and though it cannot work as a long term solution to this problem, it keeps me happy for now.

To implement this technique on your own site, first make sure you are running Apache with mod_rewrite. If you are, create a file called ".htaccess" (just that, not .htaccess.txt or anything else) and paste the following into it:

Update: 14th September 2005

The list below has been expanded substantially over the last year, and now covers much more spam than before. As stated before, this is not a practical solution to the problem in the long term, as this list can only ever get longer and longer, and may become unmaintainable, or even (eventually) slow a site to a crawl as all the rules are processed. However, as of now, it is still a useful tool.

RewriteEngine  on

# Block Referrer Spam

# Drugs / Herbal

    RewriteCond %{HTTP_REFERER} (sleep-?deprivation) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sleep-?disorders) [NC,OR]
    RewriteCond %{HTTP_REFERER} (insomnia) [NC,OR]
    RewriteCond %{HTTP_REFERER} (phentermine) [NC,OR]
    RewriteCond %{HTTP_REFERER} (phentemine) [NC,OR]
    RewriteCond %{HTTP_REFERER} (vicodin) [NC,OR]
    RewriteCond %{HTTP_REFERER} (hydrocodone) [NC,OR]
    RewriteCond %{HTTP_REFERER} (levitra) [NC,OR]
    RewriteCond %{HTTP_REFERER} (hgh-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-hgh) [NC,OR]
    RewriteCond %{HTTP_REFERER} (ultram-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-ultram) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cialis) [NC,OR]
    RewriteCond %{HTTP_REFERER} (soma-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-soma) [NC,OR]
    RewriteCond %{HTTP_REFERER} (diazepam) [NC,OR]	
    RewriteCond %{HTTP_REFERER} (gabapentin) [NC,OR]
    RewriteCond %{HTTP_REFERER} (celebrex) [NC,OR]
    RewriteCond %{HTTP_REFERER} (viagra) [NC,OR]
    RewriteCond %{HTTP_REFERER} (fioricet) [NC,OR]
    RewriteCond %{HTTP_REFERER} (ambien) [NC,OR]
    RewriteCond %{HTTP_REFERER} (valium) [NC,OR]
    RewriteCond %{HTTP_REFERER} (zoloft) [NC,OR]
    RewriteCond %{HTTP_REFERER} (finasteride) [NC,OR]
    RewriteCond %{HTTP_REFERER} (lamisil) [NC,OR]
    RewriteCond %{HTTP_REFERER} (meridia) [NC,OR]
    RewriteCond %{HTTP_REFERER} (allegra) [NC,OR]
    RewriteCond %{HTTP_REFERER} (diflucan) [NC,OR]
    RewriteCond %{HTTP_REFERER} (zovirax) [NC,OR]
    RewriteCond %{HTTP_REFERER} (valtrex) [NC,OR]
    RewriteCond %{HTTP_REFERER} (lipitor) [NC,OR]
    RewriteCond %{HTTP_REFERER} (proscar) [NC,OR]
    RewriteCond %{HTTP_REFERER} (acyclovir) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sildenafil) [NC,OR]
    RewriteCond %{HTTP_REFERER} (tadalafil) [NC,OR]
    RewriteCond %{HTTP_REFERER} (xenical) [NC,OR]
    RewriteCond %{HTTP_REFERER} (melatonin) [NC,OR]
    RewriteCond %{HTTP_REFERER} (xanax) [NC,OR]
    RewriteCond %{HTTP_REFERER} (herbal) [NC,OR]
    RewriteCond %{HTTP_REFERER} (drugs) [NC,OR]
    RewriteCond %{HTTP_REFERER} (lortab) [NC,OR]
    RewriteCond %{HTTP_REFERER} (adipex) [NC,OR]
    RewriteCond %{HTTP_REFERER} (propecia) [NC,OR]
    RewriteCond %{HTTP_REFERER} (carisoprodol) [NC,OR]
    RewriteCond %{HTTP_REFERER} (tramadol) [NC]
        RewriteRule .* - [F]

# Porn

    RewriteCond %{HTTP_REFERER} (porno) [NC,OR]
    RewriteCond %{HTTP_REFERER} (shemale) [NC,OR]
    RewriteCond %{HTTP_REFERER} (gangbang) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-cock) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-anal) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-orgy) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cock-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (anal-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (orgy-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (singles-?christian) [NC,OR]
    RewriteCond %{HTTP_REFERER} (dating-?christian) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cumeating) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cream-?pies) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cumsucking) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cumswapping) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cumfilled) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cumdripping) [NC,OR]
    RewriteCond %{HTTP_REFERER} (krankenversicherung) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cumpussy) [NC,OR]
    RewriteCond %{HTTP_REFERER} (suckingcum) [NC,OR]
    RewriteCond %{HTTP_REFERER} (drippingcum) [NC,OR]
    RewriteCond %{HTTP_REFERER} (pussycum) [NC,OR]
    RewriteCond %{HTTP_REFERER} (swappingcum) [NC,OR]
    RewriteCond %{HTTP_REFERER} (eatingcum) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cum-) [NC,OR]
    RewriteCond %{HTTP_REFERER} (-cum) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sperm) [NC,OR]
    RewriteCond %{HTTP_REFERER} (christian-?dating) [NC,OR]
    RewriteCond %{HTTP_REFERER} (jewish-?singles) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sex-?meetings) [NC,OR]
    RewriteCond %{HTTP_REFERER} (swinging) [NC,OR]
    RewriteCond %{HTTP_REFERER} (swingers) [NC,OR]
    RewriteCond %{HTTP_REFERER} (personals) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sleeping) [NC,OR]
    RewriteCond %{HTTP_REFERER} (libido) [NC,OR]
    RewriteCond %{HTTP_REFERER} (grannies) [NC,OR]
    RewriteCond %{HTTP_REFERER} (mature) [NC,OR]
    RewriteCond %{HTTP_REFERER} (enhancement) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sexual) [NC,OR]
    RewriteCond %{HTTP_REFERER} (gay-?teen) [NC,OR]
    RewriteCond %{HTTP_REFERER} (teen-?chat) [NC,OR]
    RewriteCond %{HTTP_REFERER} (gay-?chat) [NC,OR]
    RewriteCond %{HTTP_REFERER} (adult-?finder) [NC,OR]
    RewriteCond %{HTTP_REFERER} (adult-?friend) [NC,OR]
    RewriteCond %{HTTP_REFERER} (friend-?finder) [NC,OR]
    RewriteCond %{HTTP_REFERER} (friend-?adult) [NC,OR]
    RewriteCond %{HTTP_REFERER} (finder-?adult) [NC,OR]
    RewriteCond %{HTTP_REFERER} (finder-?friend) [NC,OR]
    RewriteCond %{HTTP_REFERER} (discrete-?encounters) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cheating-?wives) [NC,OR]
    RewriteCond %{HTTP_REFERER} (housewives) [NC,OR]
    RewriteCond %{HTTP_REFERER} (\-sex\.) [NC,OR]
    RewriteCond %{HTTP_REFERER} (xxx) [NC,OR]
    RewriteCond %{HTTP_REFERER} (snowballing) [NC]		
        RewriteRule .* - [F]

# Weight

    RewriteCond %{HTTP_REFERER} (fat-) [NC,OR]	
    RewriteCond %{HTTP_REFERER} (-fat) [NC,OR]	
    RewriteCond %{HTTP_REFERER} (diet) [NC,OR]
    RewriteCond %{HTTP_REFERER} (pills) [NC,OR]
    RewriteCond %{HTTP_REFERER} (weight) [NC,OR]
    RewriteCond %{HTTP_REFERER} (supplement) [NC]		
        RewriteRule .* - [F]	

# Gambling

    RewriteCond %{HTTP_REFERER} (texas-?hold-?em) [NC,OR]
    RewriteCond %{HTTP_REFERER} (poker) [NC,OR]
    RewriteCond %{HTTP_REFERER} (casino) [NC,OR]
    RewriteCond %{HTTP_REFERER} (blackjack) [NC]		
        RewriteRule .* - [F]

# Loans / Finance

    RewriteCond %{HTTP_REFERER} (mortgage) [NC,OR]
    RewriteCond %{HTTP_REFERER} (refinancing) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cash-?advance) [NC,OR]
    RewriteCond %{HTTP_REFERER} (cash-?money) [NC,OR]
    RewriteCond %{HTTP_REFERER} (pay-?day) [NC]		
        RewriteRule .* - [F]

# User Agents

    RewriteCond %{HTTP_USER_AGENT} (Program\ Shareware|Fetch\ API\ Request) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (Microsoft\ URL\ Control) [NC]
        RewriteRule .* - [F]

# Misc / Specific Sites

    RewriteCond %{HTTP_REFERER} (netwasgroup\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (nic4u\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (wear4u\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (foxmediasolutions\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (liveplanets\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (aeterna-tech\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (continentaltirebowl\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (chemsymphony\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (infolibria\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (globaleducationeurope\.net) [NC,OR]
    RewriteCond %{HTTP_REFERER} (soma\.125mb\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (mitglied\.lycos\.de) [NC,OR]
    RewriteCond %{HTTP_REFERER} (foxmediasolutions\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (jroundup\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (feathersandfurvanlines\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (conecrusher\.org) [NC,OR]
    RewriteCond %{HTTP_REFERER} (sbj-broadcasting\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (edthompson\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (codychesnutt\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (artsmallforsenate\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (axionfootwear\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (protzonbeer\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (candiria\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (bigsitecity\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (coresat\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (istarthere\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (amateurvoetbal\.net) [NC,OR]
    RewriteCond %{HTTP_REFERER} (alleghanyeda\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (xadulthosting\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (datashaping\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (zick\.biz) [NC,OR]
    RewriteCond %{HTTP_REFERER} (newprinceton\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (dvdsqueeze\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (xopy\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (webdevboard\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (devaddict\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (eaton-inc\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (whiteguysgroup\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (guestbookz\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (webdevsquare\.com) [NC,OR]
    RewriteCond %{HTTP_REFERER} (indfx\.net) [NC,OR]
    RewriteCond %{HTTP_REFERER} (snap\.to) [NC,OR]
    RewriteCond %{HTTP_REFERER} (2y\.net) [NC,OR]
    RewriteCond %{HTTP_REFERER} (astromagia\.info) [NC,OR]
    RewriteCond %{HTTP_REFERER} (free-?sms) [NC]       
        RewriteRule .* - [F]

The above will block just about all of the most common referral spam that I've seen so far. I'm adding to the list constantly (last addition: 14th September 2005) so do check back and see if there are updates if you're using it.

One potential problem with this technique, other than that it will, in time, become useless as too many URLs are added, is that there is always a possibility authentic visitors will be blocked. So, on this site, instead of the last line above, I've actually used something a little more user-friendly:

RewriteRule .* bad_referrer.php [L]

Instead of a "Forbidden" message, this displays a quick note explaining why there has been an error and that the user can click on a link to proceed. If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. You should find you get a message explaining what has happened, and a URL to click if you want to proceed.

And there we have it. With minimum effort (for now), referral log spamming in my site has been almost entirely removed. Before adding this set of rules and scripts, I was seeing around 200 fake referrals per day in my log files. Now, I see about 3 or 4 a week. Hopefully, this will continue until I can devise a better way of protecting against this kind of problem - before blacklists become an impossibility to manage.

Ignore Directories in mod_rewrite

Dave Child — Thu, 08 Sep 2005 12:00:00 +0100

A quick piece of code for you. If you are using mod_rewrite and creating RewriteRules for a website that emulate a directory structure, you might happen across the same problem I've had. If you have actual, real folders on the site as well, you don't want requests for items in those folders to be rewritten. You need a way to prevent the RewriteRule(s) matching the real folders. The easiest way to do this is (I think) by adding a RewriteRule for each of the real folders, like the below. This rule will match any request to those folders and prevent it being rewritten later in the set of rules.

RewriteRule ^folder_name/.*$ - [PT]

Password Protect a Directory with .htaccess

Dave Child — Tue, 15 Mar 2005 09:58:46 +0000

A tutorial explaining how to retrict access to a directory on a web server using .htaccess.

Password protecting a directory can be done several ways. Many people use PHP or ASP to verify users, but if you want to protect a directory of files or images (for example), that often isn't practical. Fortunately, Apache has a built-in method for protecting directories from prying eyes, using the .htaccess file.

In order to protect your chosen directory, you will first need to create an .htaccess file. This is the file that the server will check before allowing access to anything in the same directory. That's right, the .htaccess file belongs in the directory you are protecting, and you can have one in each of as many directories as you like.

You'll need first to define a few parameters for the .htaccess file. It needs to know where to find certain information, for example a list of valid usernames and passwords. This is a sample of the few lines required in an .htaccess file to begin with, telling it where the usernames and passwords can be found, amongst other things.

AuthUserFile /full/path/to/.htpasswd
AuthName "Please Log In"
AuthType Basic

You've now defined a few basic parameters for Apache to manage the authorisation process. First, you've defined the location of the .htpasswd file. This is the file that contains all the usernams and encrypted passwords for your site. We'll cover adding information to this file shortly. It's extremely important that you place this file outside of the web root. You should only be able to access it by FTP, not over the web.

The AuthName parameter basically just defines the title of the password entry box when the user logs in. It's not exactly the most important part of the file, but should be defined. The AuthType tells the server what sort of processing is in use, and "Basic" is the most common and perfectly adequate for almost any purpose.

We've told apache where to find files, but we've not told it who, of those people defined in the .htpasswd file, can access the directory. For that reason, we still have another line to define.

If we want to grant access to everyone in the .htpasswd file, we can add this line ("valid-user" is like a keyword, telling apache any user will do):

require valid-user

If we want to just grant access to a single user, we can use "user" and their username instead of "valid-user":

require user dave

A normal and complete .htaccess file might look like this:

AuthUserFile /home/dave/.htpasswd
AuthName "Dave's Login Area"
AuthType Basic

require user dave

Now we have almost everything defined, but we are still missing an .htpasswd file. Without that, the server won't know what usernames and passwords are ok.

An .htpasswd file is made up of a series of lines, one for each valid user. Each line looks like this, with a username, then colon, then encrypted password:

username:encryptedpassword

The password encryption is the same as you'll find in PHP's crypt() function. It is not reversible, so you can't find out a password from the encrypted version. (Please note that on page 2 of this article is a tool to help you generate an .htpasswd file, that will help you encrypt passwords).

A user of "dave" and password of "dave" might be added with the following line:

dave:XO5UAT7ceqPvc

Each time you run an encryption function like "crypt", you will almost certainly get a different result. This is down to something called "salt", which in the above case was "XO" (first two letters of encrypted password). Different salt will give different encrypted values, and if not explicitly specified will be randomly generated. Don't worry though, the server is quite capable of understanding all this - if you come up with a different value for the encrypted password and replace it, everything would still work fine, as long as the password was the same.

Once you've created your .htpasswd file, you need to upload it to a safe location on your server, and check you've set the .htaccess file to point to it correctly. Then, upload the .htaccess file to the directory you want to protect and you'll be all set. Simply visit the directory to check it is all working.

.htpasswd Generator

The .htpasswd file needs encrypted passwords, which can be a problem for anyone without experience with a programming language. For that reason, I've created this simple tool, which, if you enter the username and password you wish to use, will generate the appropriate line to add to your .htpasswd file.

[!htpasswd!]

HTTP Status Codes for Beginners

Dave Child — Tue, 11 May 2004 15:33:55 +0100

All valid HTTP 1.1 Status Codes simply explained.

HTTP, Hypertext Transfer Protocol, is the method by which clients (i.e. you) and servers communicate. When someone clicks a link, types in a URL or submits out a form, their browser sends a request to a server for information. It might be asking for a page, or sending data, but either way, that is called an HTTP Request. When a server receives that request, it sends back an HTTP Response, with information for the client. Usually, this is invisible, though I'm sure you've seen one of the very common Response codes - 404, indicating a page was not found. There are a fair few more status codes sent by servers, and the following is a list of the current ones in HTTP 1.1, along with an explanation of their meanings.

A more technical breakdown of HTTP 1.1 status codes and their meanings is available at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. There are several versions of HTTP, but currently HTTP 1.1 is the most widely used.

Informational

100 - Continue
A status code of 100 indicates that (usually the first) part of a request has been received without any problems, and that the rest of the request should now be sent.
101 - Switching Protocols
HTTP 1.1 is just one type of protocol for transferring data on the web, and a status code of 101 indicates that the server is changing to the protocol it defines in the "Upgrade" header it returns to the client. For example, when requesting a page, a browser might receive a statis code of 101, followed by an "Upgrade" header showing that the server is changing to a different version of HTTP.

Successful

200 - OK
The 200 status code is by far the most common returned. It means, simply, that the request was received and understood and is being processed.
201 - Created
A 201 status code indicates that a request was successful and as a result, a resource has been created (for example a new page).
202 - Accepted
The status code 202 indicates that server has received and understood the request, and that it has been accepted for processing, although it may not be processed immediately.
203 - Non-Authoritative Information
A 203 status code means that the request was received and understood, and that information sent back about the response is from a third party, rather than the original server. This is virtually identical in meaning to a 200 status code.
204 - No Content
The 204 status code means that the request was received and understood, but that there is no need to send any data back.
205 - Reset Content
The 205 status code is a request from the server to the client to reset the document from which the original request was sent. For example, if a user fills out a form, and submits it, a status code of 205 means the server is asking the browser to clear the form.
206 - Partial Content
A status code of 206 is a response to a request for part of a document. This is used by advanced caching tools, when a user agent requests only a small part of a page, and just that section is returned.

Redirection

300 - Multiple Choices
The 300 status code indicates that a resource has moved. The response will also include a list of locations from which the user agent can select the most appropriate.
301 - Moved Permanently
A status code of 301 tells a client that the resource they asked for has permanently moved to a new location. The response should also include this location. It tells the client to use the new URL the next time it wants to fetch the same resource.
302 - Found
A status code of 302 tells a client that the resource they asked for has temporarily moved to a new location. The response should also include this location. It tells the client that it should carry on using the same URL to access this resource.
303 - See Other
A 303 status code indicates that the response to the request can be found at the specified URL, and should be retrieved from there. It does not mean that something has moved - it is simply specifying the address at which the response to the request can be found.
304 - Not Modified
The 304 status code is sent in response to a request (for a document) that asked for the document only if it was newer than the one the client already had. Normally, when a document is cached, the date it was cached is stored. The next time the document is viewed, the client asks the server if the document has changed. If not, the client just reloads the document from the cache.
305 - Use Proxy
A 305 status code tells the client that the requested resource has to be reached through a proxy, which will be specified in the response.
307 - Temporary Redirect
307 is the status code that is sent when a document is temporarily available at a different URL, which is also returned. There is very little difference between a 302 status code and a 307 status code. 307 was created as another, less ambiguous, version of the 302 status code.

Client Error

400 - Bad Request
A status code of 400 indicates that the server did not understand the request due to bad syntax.
401 - Unauthorized
A 401 status code indicates that before a resource can be accessed, the client must be authorised by the server.
402 - Payment Required
The 402 status code is not currently in use, being listed as "reserved for future use".
403 - Forbidden
A 403 status code indicates that the client cannot access the requested resource. That might mean that the wrong username and password were sent in the request, or that the permissions on the server do not allow what was being asked.
404 - Not Found
The best known of them all, the 404 status code indicates that the requested resource was not found at the URL given, and the server has no idea how long for.
405 - Method Not Allowed
A 405 status code is returned when the client has tried to use a request method that the server does not allow. Request methods that are allowed should be sent with the response (common request methods are POST and GET).
406 - Not Acceptable
The 406 status code means that, although the server understood and processed the request, the response is of a form the client cannot understand. A client sends, as part of a request, headers indicating what types of data it can use, and a 406 error is returned when the response is of a type not i that list.
407 - Proxy Authentication Required
The 407 status code is very similar to the 401 status code, and means that the client must be authorised by the proxy before the request can proceed.
408 - Request Timeout
A 408 status code means that the client did not produce a request quickly enough. A server is set to only wait a certain amount of time for responses from clients, and a 408 status code indicates that time has passed.
409 - Conflict
A 409 status code indicates that the server was unable to complete the request, often because a file would need to be editted, created or deleted, and that file cannot be editted, created or deleted.
410 - Gone
A 410 status code is the 404's lesser known cousin. It indicates that a resource has permanently gone (a 404 status code gives no indication if a resource has gine permanently or temporarily), and no new address is known for it.
411 - Length Required
The 411 status code occurs when a server refuses to process a request because a content length was not specified.
412 - Precondition Failed
A 412 status code indicates that one of the conditions the request was made under has failed.
413 - Request Entity Too Large
The 413 status code indicates that the request was larger than the server is able to handle, either due to physical constraints or to settings. Usually, this occurs when a file is sent using the POST method from a form, and the file is larger than the maximum size allowed in the server settings.
414 - Request-URI Too Long
The 414 status code indicates the the URL requested by the client was longer than it can process.
415 - Unsupported Media Type
A 415 status code is returned by a server to indicate that part of the request was in an unsupported format.
416 - Requested Range Not Satisfiable
A 416 status code indicates that the server was unable to fulfill the request. This may be, for example, because the client asked for the 800th-900th bytes of a document, but the document was only 200 bytes long.
417 - Expectation Failed
The 417 status code means that the server was unable to properly complete the request. One of the headers sent to the server, the "Expect" header, indicated an expectation the server could not meet.

Server Error

500 - Internal Server Error
A 500 status code (all too often seen by Perl programmers) indicates that the server encountered something it didn't expect and was unable to complete the request.
501 - Not Implemented
The 501 status code indicates that the server does not support all that is needed for the request to be completed.
502 - Bad Gateway
A 502 status code indicates that a server, while acting as a proxy, received a response from a server further upstream that it judged invalid.
503 - Service Unavailable
A 503 status code is most often seen on extremely busy servers, and it indicates that the server was unable to complete the request due to a server overload.
504 - Gateway Timeout
A 504 status code is returned when a server acting as a proxy has waited too long for a response from a server further upstream.
505 - HTTP Version Not Supported
A 505 status code is returned when the HTTP version indicated in the request is no supported. The response should indicate which HTTP versions are supported.