Your conversion rate is a measure of the number of potential customers that go on to buy. In the context of a website, it is usually the percentage of visitors that make a purchase. Many websites concentrate solely on increasing the number of visitors they have, when often they have fairly simple problems with their site that, if solved, would have a huge effect on their conversion rate and improve their site's bottom line at minimal expense.

Improving a website conversion rate can be relatively simple. Here are 10 techniques for doing just that:

10. Make The User's Life Easy

Let's start with something that sounds simple, but apparently is too complex for many companies to get right. The more difficult you make your web site to use, the less people will buy from you.

A well designed website should aim to prevent nobody from buying - to allow 100% of the people who want to buy to do so. So where do they go wrong?

• Accessibility
  Making a site accessible is a legal obligation in many countries. Despite that, inaccessible websites are still being created. That can affect your sales, depending on how inaccessible you are, as visitors find the site impossible to use and go elsewhere (and end up recommending one of your competitors to their friends as well). A fairly typical inaccessible site could be losing 5% of potential sales because of this. (A really inaccessible website could even prevent search engines indexing it, giving a far higher amount of potential lost sales.)  
• Browsers
  Many designers only pay attention to Internet Explorer. The justification for this is usually that 99% of the site's users use IE. It never seems to occur to the designers that perhaps the reason they have so few visitors with other browsers is that their site is fundamentally broken - it doesn't work in anything else. Percentages of people not using IE varies from site to site - over 60% of visitors to this site use an alternative browser, for example. The number most often quoted though, is that 80-85% of web users are using IE on Windows, which means that an average site that doesn't work in anything else could easily be losing 15-20% of sales.  
• Be Bold!
  What happens when a user decides to buy a product? They add it to a shopping basket. How do they add it? They click a button or link (usually a button). What happens when they can't see the button? They go elsewhere. There are some users who are still uncomfortable scrolling. Having things above the fold is still important. And yet there are still plenty of sites out there with buttons that are too subtle, or don't say the right thing, or are hidden away at the bottom of the page. "Add" is rubbish button text. "Buy" is ok. "Add xxx To Your Basket" is great. "Add xxx to Your Basket" in big letters on a big, bright button, near the top of the page, is even better. Calls to action, like this, don't have to be gaudy or tasteless, but they do have to be obvious and clear. Sites I have worked on where just the call to action was changed have reported anything from a 1% to 30% increase in sales as a result.  
• Usability
  If your potential customers want to find out more before they buy, can they? Is it obvious to the user where to go to find the technical specs on your products? Are they online at all? Are they in PDF format? Can users even find your products in the first place? This is probably the most common mistake I see on any website - a complete failure to think of what the user wants and needs, and how they might use a site. Plenty of sites have product pages with a photo and some sales patter - and nothing else. Anything from 1% to 99% of potential sales can be lost through poor usability.

When you combine all of the problems above, it becomes fairly clear how easy it is to have a site perform poorly. Make your site accessible, make sure it is usable, make sure it works in common browsers, and make your calls to action clear and unambiguous, and you should be in a position to start converting the people who want to buy.

9. Be Clear, Open and Honest

If you have a product out of stock, say so. Few things annoy users as much as reading all about a product they are after, adding it to a cart, and starting the checkout process - only to find out the product isn't actually available.

The same applies to pricing - a user might spend $100 on a product, but when they find out the shipping is $100 on top of that, they are unlikely to continue the sale. Showing delivery pricing is tricky business, but not impossible. An Ip to Country database will allow you to work out where a user is from and show them a likely delivery cost, for example. If you can't do that, show delivery prices for the countries most appropriate to you - where your products are most often delivered, or for major world regions.

8. Don't Waste Time

One of the biggest mistakes sites make is asking for too much information. Your conversion process may be sale, or it may be a request for information. Either way, don't waste the user's time asking for things you don't need to know. This is, of course, doubly important when it comes to asking for information the user deems private, and that they don't want to give out without good reason.

You don't need to demand the user's email address before letting them download a PDF. You don't need their phone number when they fill out an email enquiry form. A user may not want to buy from you twice - so why make them create an account so they can buy again later before processing their first order? You can give the user the option to do all of these things by all means, but make sure it's not compulsory.

7. Help The User Trust You

Most people are still cautious when buying online, and rightly so. There are plenty of people you really shouldn't give your credit card information to! It's important to give the potential customer every reason to trust you.

An address - bricks and mortar, not a P.O. Box - is a good start. A phone number, with people answering the phone, also helps. Showing a privacy policy and explaining shipping procedures clearly can also help the user to trust you. If you have a SSL certificate, show the "VeriSign Secured" logo to the user.

Design and content also play a part in trust. A poor design gives off an unprofessional feeling. If a company can't afford a decent website, or won't spend the money on it, how can a user be sure their order will be treated with the importance it deserves? If content is inaccurate or badly written, the same applies - show that you take pride in what you do.

6. Have a Clear Returns Policy

Returns on the web are, and are likely to remain, a major issue for consumers. With a bricks and mortar shop, the customer knows where the shop is and that to return the product they simply have to go back there and explain the problem. With the web, this is more of an issue. This is especially true for clothing (where people cannot try things on before buying).

Users are impressed with sites with a good returns policy and are more likely to buy from them. Have people phone for returns - they can then explain the problem to a real person, which is always a good first step. Free return shipping is usually a good option, if commercially viable. People don't like to pay to return things, especially if it is a mistake by the retailer. Finally, give the user plenty of time to return things. 28 days is fairly common, but if it takes you that long to deliver a product, what use is the return policy? 28 days from the date of delivery is better.

5. Keep the User Informed

When somebody buys something online, they want to know when it's going to arrive at their door. People are impatient, after all. Giving them an estimated delivery date during the checkout process is a good start. Emailing them when their product is dispatched is great. Giving them a tracking number if using a delivery service that supports online tracking is even better. Keep the user informed at every step of the process, before and after sale, about as much as you can.

How will this improve your conversion rate? Leaving the customer happy once they have made a sale means they are more likely to speak favourably about you later. They may even recommend you to their friends and within online communities. They are also far more likely to buy from you again.

Think about it like this - if a salesman is doing their absolute best to help you, and to make your life easy, and answering your questions, you might buy what they were selling. If they completely ignored you after you'd bought from them, how would you feel about them? They might well have undone all the good work they put in, because once you'd completed your purchase they see no immediate value in you. A company that shows it cares about their customers, even after they've finished shopping, will make a user far happier and far more likely to return.

4. Offer Different Payment Options

It might sound obvious, but you should offer the user a reasonable selection of methods of payment. Not everybody has a credit card, and those that do don't always want to use them. You don't have to accept cheques, but when deciding on payment methods, consider alternatives to the usual methods. Make the user's life easy and give them what they want.

3. Improve the Value of Visitors

People that buy from you are doing so because they like what it is they see. If a user adds a product to a basket, show them other things they might like as well. If they are viewing a product, the same applies - show them similar items. While they might not buy the product they first saw, other similar ones may not have issues that put them off the first. Upselling and cross-selling are tried and tested sales techniques, and there is no reason not to use them on the web.

2. Be Memorable

A good site will include information. A poor one is just an online catalogue. Information (articles, advice, reviews and so on) all help the user early in their buying process. Users start with research online, just as they do offline. If you can make contact with the user at that stage of their process, and give a favourable impression, there is a good chance that they will come back and buy from you when they finally decide to make a purchase.

Being memorable, and making sure you stick in the user's mind, is dependant on a lot of factors. You must have a USP (see the next point), and branding is important (no good if your visitors remember why you are great but don't remember your name), as well as the quality of your site and information.

1. Know Your USP

Finally, the most important point of all - your Unique Selling Point (USP). Your USP is what sets you apart from your competition. If a visitor goes to several sites looking for a product, why would they decide to buy from you instead of somewhere else?

Many companies do not know their USP. Almost all companies have one, but not all of them are aware of it. If you are a family run business, that's a potential USP. Great customer service, low prices, products that can't be bought elsewhere, free delivery, great support - all of these are USPs. Tell your users what yours is. Shout it from the proverbial rooftops.

Part 2

In March 2009, part 2 of this series was added: Nine More Tips for Improving Your Website Conversion Rate.

Bonus!

One excellent (and practical) way to increase your website conversion rate is to add consumer reviews to your store. They are a proven way to increase sales, and they have an excellent positive effect on your search engine optimisation work. A service like FeedbackFair will give your reviews extra credibility.

It is amazing how many people hire online marketers without the faintest idea of what online marketers actually do. Search engine optimisation (SEO) is fairly simple - SEOs will try and improve your site's performance, usually by trying to leverage their knowledge of how search engines work and tricks they can use to make sites seem more relevant than they actually are to specific keywords.

Marketing online, though, need not have anything to do with search engines. Search engines are irrelevant - good positions and traffic are a by-product of effective online marketing.

Unfortunately, after educating a client on what online marketing is, they usually assume that if they pay you a few hundred pounds, you can make their site compete with the very best out there.

• Client:
  'Ok, I see. Great positions aren't necessarily worth much unless there are customers searching for those keywords.'
• Me:
  'Right. We want high traffic, but not if it's not going to be bad for your bottom line. Traffic that doesn't convert to sales just costs you money. Same applies for phrases people never search for. No point being number one for the phrase "fish banana druid" - it's likely to get you as many customers as peeing on people that walk past your shop will.'
• Client:
  'Ok, so if I pay you, say, £300, how long before I'm at number one for this list of relevant phrases?'
• Me:
  'You wouldn't get in a boxing ring with Joe Calzhaghe after jogging a couple of miles and doing a few push-ups, would you?'
• Client:
  'Well, no.'
• Me:
  'Exactly. To compete with the big dogs, you need to think bigger. Your site is a 10 stone weakling at the moment, and the aim is to turn it into a champion. It needs to be Rocky Balboa. You won't get the top spots quickly - this takes time and hard work. And it's not cheap.'

People are obsessed with money. Absolutely obsessed. Even more so in a company environment. The chances are the most of the time, the person you are talking to at a client (or potential client) company is not the top dog. They have to justify their decisions, and they certainly have to justify what they spend.

The problem is that the way most people look at SEO (and they are thinking SEO, not marketing - it's up to you to show them the difference) is that they're going to pay a certain amount of money for the top spots for certain keywords. You can guarantee they've been told another company will guarantee 10 number 1 positions for $50.

This is where ROI comes into play. ROI stands for "Return on Investment". Paying $50 for a $0 return is a bad idea - but people do it all the time, because it's cheap. Paying $5,000 for a $50,000 return is a great idea - but people gasp at the very idea they could spend that much in the beginning, despite the potential.

In order to measure a return, you need to use tracking. If you're focussed on natural search, measure natural search traffic. See how many people come to the site, and where from. See where they go in the site. See if they view products, add them to a basket, and complete sales. See if they view products then come back weeks later to buy them. Measure that over time and you can tell a client exactly what effect your marketing campaign is having - and you will be able to show them what they are getting for their money. Usually, telling a client you are going to do this will also put their mind at ease - much easier to spend money on someone when that person tells you how they're going to measure their success. Most companies involved in SEO and online marketing focus on positions, not results.

• Client:
  'That's good to know. If I can see what's going on, I can give hard numbers to my boss. I'd rather tell him we have 10% more visitors and 20% more sales than tell him we're in top positions for our target phrases but traffic has gone down.'
• Me:
  'Woohoo! You've taken your first step into a larger world.'

The other thing to bear in mind with money conversations is that most companies think of their site like a brochure. They think of it as a print-like cost, where they pay a fixed sum and that's it. They put the site up, leave it, and expect results. They should be thinking of a site like a salesman. A salesman that never sleeps, rarely gets ill, and can handle virtually unlimited enquiries. As such, they should be thinking of the money they spend more like a wage.

• Client:
  'We're spending $200 a month on our site now for hosting. Are you saying we should be spending a lot more?'
• Me:
  'What would you pay a salesman with the figures your site has, ignoring PPC?'
• Client:
  'Probably $3000 a month.'
• Me:
  'Then that's what you should be spending on the site. As the figures get better, spend a little more. Remember that that needs to include redesigns, hosting and other costs.'

(Note: PPC is something of a difficult subject to bring in to a monthly spend on a site. You should have a monthly spend on PPC, but it should be managed as a separate entity.)

The same traffic you are monitoring to see where site visitors are coming from and what they are doing when they reach the site can also give you some good places to start making changes. Break the traffic down by area, by language, by time of day (user time of day, not server time of day), and track who converts to a sale and who doesn't. Track people through the sales process, and watch which links they click to navigate and buy products.

This will tell you a huge amount about the current users of the site. It will show you quick wins, opportunities, and highlight problems. Forget search - if on your first day marketing a website you can spot that there is a problem with the site checkout process and get it fixed, you could double sales from existing users. That's a good start to any campaign.

Look at language and area closely as well. If a site is getting traffic from the US, but only sells to the UK, look at similar companies only serving the US and strike a deal with them. You direct US traffic to them, they direct UK traffic to you, and you both do slightly better.

Check browser usage stats, especially if the site is a tables-based dinosaur. The chances are that it is an inaccessible mess. Get it cleaned up! Semantic markup is key - it allows user agents (browsers, search engine spiders, screen readers) to attach specific meaning to different areas of a page. Unlike with tables, semantic markup allows you to differentiate between a header and normal content, or to identify an address. Accessible coding is likely to draw attention, and should help you retain a higher percentage of your visitors, and should help reduce the running costs of your website (lower bandwidth bills and quicker turnarounds on redesigns, for example, both save you money).

• Client:
  'I don't care about different browsers though - they only make up 1% of my traffic. Everyone else uses Internet Explorer'
• Me:
  'Perhaps it is because your site doesn't work in other browsers that that number is so low. Even if you do have 99% of users on the same system, the other 1% is still important. Techies use different browsers and operating systems. Techies are also often the people who are asked by their families if they know a good site to buy something from. Many directory editors are in the same boat, and techies can create links to your site.'
• Client:
  'Ok, techies are important. But do I need to care about blind users and all that accessibility stuff?'
• Me:
  'Yes, of course. It's a legal obligation for one thing, but users with sight problems make up a far larger proportion of your audience than you might think. They have a voice too - and it's far harder to undo the damage some adverse publicity can do than it is to make a site work properly in the first place. Finally, search engine spiders are blind users with no JavaScript support.'

Dynamic sites are slightly trickier to improve. Most of the time, they are restricted, with the original authors not allowing access to the website code. Even if access to the code is allowed, changes may be overwritten later or worse cause immediate problems on the site. That said, making a site easier to use is important, and often dynamic sites are not easy to use.

Look at the pages users visit in the site, and how they get there. Look at the products they buy and spot themes. Use that information to make the important sections and products easier to find and organise. For example, if listing products, don't make people click through 4 levels of navigation to find them - improve the product navigation. Once they get there, allow them to reorder the page according to what they consider important, be that name, price, manufacturer - whatever is possible.

Remember also that people like to tell other people about things they find. If a user likes something on your site, they may email the address of the page they are on to a friend. Most people use forms to set the ordering criteria of a page. That means that the user will be sending a friend a URL that will show that friend something different to what the user currently sees. Make life easy for your users - use URLs, not forms, wherever possible in a site.

• Client:
  'I am curious about one thing. We're already really well ranked for the name of our main product, and lots of people search for it. The people that visit our site tend to buy the product. But I can't help feeling that there should be more people coming from the engines. Any ideas?'
• Me:
  'Yes. The Overture Search Term Suggestion Tool', capitalising my speech for no good reason, 'shows that millions of people search for that phrase. I can see you have a top spot. And your traffic is surprisingly low, but converting well.'
• Client:
  'So I'm not imagining it then - we have a problem?'
• Me:
  'Yes, we do. When a user looks at search results, they scan the first two or three words of each link. Your link says "Arthur Jackson Ltd. Sheds and other garden products." That comes from your page title.'
• Client:
  'And that's bad?'
• Me:
  'Most people will only glance at "Arthur Jackson Ltd". You need to show them, in the first two or three words of your page title, that you have what they are looking for. And you're not doing that. The user has no reason to click on your link ahead of all the others they see.'

Titles are tricky. They're important to the user, they provide the text for bookmarks, they appear in search results, and search engines use them as part of ranking algorithms. You need for fit branding into a title, and describe a product, ideally also incorporating a call to action. Tricky stuff. But not impossible.

First, consider the brand. Most companies think their company name should be the first thing in a page title, even if the rest is unique for each page (as it should be). However, unless the company has a household brand name, the company name is irrelevant to the searcher. They're looking for a product (or the answer to a question), so show them you have it.

Next, remember that as titles are used as the text for bookmarks, links and appear in search engines, they should, when taken out of context, by themselves, leave no doubt what a page is about.

A good example of a title is:

• "Norwegian Blue Parrot - Buy Norwegian Blue Parrots from Mr. Praline's Pet Shop".

You've included the all-important product name twice in the title, along with a call to action, a hefty dose of branding, and not added irrelevant information. It's a title that tells the user straight away what the page is about. No messing around.

• Client:
  'Ok, the titles need sorting, but what about the content of the site? I keep hearing that "Content is King".'
• Me:
  'Content is, ultimately, King. Sites with lots of great content will, over a decent time period, far outperform sites with no original content. But content doesn't just have to be on site ...'

Product is important. The object you sell though is only half of the picture. A user will want support from you. They will want information. They may want news. All of this is part and parcel of the package a company offers. Your site needs good, visible support (including a phone number), as well as plenty of good, original information. Guides to products, online manuals, FAQs, advice - there are always areas, in any industry, where content can be added.

Content need not be solely posted on the website either. Big news should be released as a press release, and there are plenty of services that will distribute press releases for you. These will be reproduced all over the web, allowing more and more people to hear of the company. Most press release services will allow you to embed a link to a site in a press release, generating more direct traffic as well.

When writing content, or advising on the writing of content, remember that it is not about keywords. Sure, keywords are important, but there is more to it than simply stuffing as many keywords into text as possible. Content needs to answer questions - to provide information. It needs to give a user what they are looking for, and they need to feel that it has done that. Content that is written for SEO can read very badly with too many keywords in, and can mean that although more people see an article, most of them leave the site straight away to find a better one.

A good way to add content to a site is a blog, or a news section. Aside from adding plenty of information, this gives a great opportunity to connect with the user. Consumers are constantly being targeted, from every angle, by companies anxious to take their money. Sometimes they get trodden on. When adding content to your site, stay on the side of the average consumer. Recently, in the UK, the energy companies all raised their prices dramatically. Sites that allow users to compare fuel prices almost all missed a great opportunity to have themselves noticed - not one of them posted a decent news item denouncing the changes as unnecessary or over the top. They all simply commented on the change factually.

While on the one hand, some of these companies may be unable to comment in this fashion (and many companies have strict policies regarding neutrality and customer perception), at least one should have been able to stand out by taking a clear, customer-supporting position on the issue. That is the kind of thing that gets companies noticed and remembered, and spotting opportunities like that is key to a good marketing strategy.

Not all content need be inflammatory of course. It does need to be unique in some way, however. It can be controversial, but it could also be definitive - the ultimate and complete guide to a topic. Controversial content is interesting to the user, and definitive content is just plain useful - either makes for good content for any website.

Users go through different stages when buying products, and one of the early ones is a research stage. There is always a good chance that a user will come back to the same place that helped them or impressed them when they were doing research to buy what they were looking for. This is branding - associating specific ideas and feelings with your company. You want your users, when they revisit the web to make a purchase, to think of your company first.

Which brings us nicely to our last, and most important point. Why would a customer think of any company first, ahead of any other. Content will help, yes. A nice design might even make a difference. More than anything else, though, customers pay attention to the company that stands out from the crowd - the company that is different, that offers them something nobody else does. Often known as a Unique Selling Point, or USP, this is the thing that makes you memorable, or if ignored helps you blend into the crowd.

• Client:
  'But we don't have a USP. How do we get one?'
• Me:
  'Well, hang on one minute. You say you don't have a USP, but is there nothing about your product that makes it better than the alternatives?'
• Client:
  'Well, we sell Norwegian Blue Parrots. They're all the same, really. Although a rather large proportion of our competitors appear to sell mostly dead ones.'
• Me:
  'There you go then. Your USP is that your product is, in fact, not dead.'
• Client:
  'By that reasoning, a USP could be almost anything, when put in the right light. And when did we turn into a Monty Python sketch?'
• Me:
  'Be quiet about the Monty Python thing. Yes, though, a USP can be virtually anything. It can be quicker delivery than competitors, better products, better customer service, a freephone enquiries number, or simply the people that run the business. Almost every business has a USP - although most of them don't know what it is.'

Many businesses don't know their own USP. They can't tell you, when you ask, what makes them different. Many of them will just say "because we're better than the others", but can't explain why. Usually, however, a quick chat will reveal what makes them stand out. Whatever the USP is, it needs to be clear and obvious on the website. The customer can't miss it, because if they don't know what makes one business different from another, they're not going to remember it.

• Client:
  'What about search? You've not told me how to get my site to the top of the search engines!'
• Me:
  'Let's review, shall we. You've changed your site substantially, so that it meets current standards and you can sell to more of your users. You're showing your clients why you are better than your competition. You've started releasing press releases, and adding content to your site. You're championing the cause of the common man, increasing link numbers and getting people talking about your business. And you know how your users find your site, and what they do when they get there.'
• Client:
  'And?'
• Me:
  'You're positioning yourself as a great resource for your market. Your search engine rankings will come as a direct result of everything else you are doing. You're going to perform well in search, as a direct result of good marketing.'
• Client:
  'I'll get my chequebook.' (Hah. As if.)

For hobby webmasters, this is generally not a problem. For anyone making money online, though, it can be extremely nervewracking. For those whose livlihoods depend on their websites, losing all search engine traffic can be a devastating blow.

Unfortunately, a great deal of the threads and dicussions on this topic often result in a large amount of misinformation. For example, as a result of one recent Google update, many sites had lost significant rankings. Some forums were claiming that the specific industries had specifically been targetted and sites in that industry had been penalised in some way. Some claimed that Google had "lost" a serious amount of data, or had re-added old data, and that was what caused the change. There are as many explanations for loss of traffic as there are sites that have dropped out there.

Unfortunately, with all of the wild ideas and crazy theories being bandied around, the average site owner has a very hard time working out first what has happened, and second what to do about it.

The very first thing to consider when looking at the effect of a shift in algorithms is that a change rarely affects all ranking criteria at once. They rarely, if ever, target a specific industry, even though the effect of a change on a specific market may be far greater than in others (this is especially true in ultra-competitive arenas, such as real estate, finance and the adult industry, where those at the top are often precariously balanced, and a tiny change in algorithms can mean major changes to the SERPs).

Fixing The Problem

Before anything else, it is important to make sure there actually is a problem. The forums usually first fill with these types of posts during an update. However, while the update is going on the SERPs are in a state of flux. Sites can appear all over the place during an update, so save the panic until the update is over. Updates can last days, and it is a good idea to watch a few of the SEO forums to find out when an update has finished.

If the update has finished and a site has definitely dropped, it is rare that it will be able to regain the exact same (or better) traffic within a short space of time. If an algorithm change has caused a site to be dropped, the chances are that one specific thing that was making that site rank well (for example, rented links) has been devalued. If the only thing that was making a site rank well has become less important, there are probably no quick fixes.

Is It a Penalty?

The first thought to cross most peoples' minds when sites lose traffic and drop down the SERPs is that there must be a penalty applied to their site. Penalties are very real, yes, but there is no reason to suspect you have had a penalty applied unless one of the following is true:

• You have been doing bad things. If you've been using cloaking, hidden text, doorway pages, keyword stuffing or link farms etc, expect to be penalised.

• You can't find your site - at all - in the search engine you suspect has penalised you. In the case of Google, search for "site:addedbytes.com" (replacing addedbytes.com with your domain name, of course). If no results are returned, Google will show you something like this:

  Your Search - site:addedbytes.com - did not match any documents.

  Suggestions:

  • Make sure all words are spelled correctly.
  • Try different keywords.
  • Try more general keywords.

If you have been penalised, then you'll get no sympathy from me - be more careful in future! SEO is not about getting top rankings for two weeks before vanishing forever from results, it's about a sustained and long-term effort to get top spots. It's not a sprint, it's a marathon. It's not worth taking the kind of risks that will get you penalised unless you have no choice. (</lecture>)

To come back from a penalty is not a quick process, but it is relatively simple. First, remove all remotely-fishy stuff from your site. Before a site is reincluded, the chances are it will be checked, and if you've not corrected what you were doing wrong, you will not be reincluded. Be over-cautious at this stage - better to remove absolutely anything that a search engine might dislike than remove the obvious things and be refused reinclusion because you've assumed that the search engine penalised you for something specific and that fixing that alone is enough. Once you are certain there is nothing left on your domain that can be considered dodgy by the whitest of white hats, then file a reinclusion request with the engine you are having trouble with.

Then wait - and it may be many months before you are reincluded, if at all. Don't pressure the engine and don't file the request every day or week. File it once and wait. You're the kid in the corner with the large hat with a D on it. THe search engine doesn't like you - you tried to manipulate it (even if it wasn't your work, it's your site and your responsibility). Have patience and work on building links to your site and building content - at least then when you are reincluded you should have better traffic.

What Next?

If you've not been penalised, then you should look at why you have dropped. Actually, let me rephrase - you should look at why your competitors have risen up the SERPs - that is a more accurate way to look at it. Check out the top sites in your field - what are they getting traffic for? What do their sites have that yours doesn't? Link quantity? Link quality? Content? Meta tags? A title in a specific shade of blue? Look for themes in the top ranking sites - if you can find out why they are top now, and you are not, you know what to work on.

The chances are that if you're not been penalised and your site is not performing as well, you need to look at improving or updating your online marketing tactics.

If your site is lacking in normal, organic links for example (you have previously paid for all of your links), then start adding things to your site people will want to link to to add to your normal organic links. Add a blog and post controversial or funny (but always unique) items on there. The web is a conversation, and you are not as prominent as you once were because the search engines are getting better - to get yourself noticed, you need to be talked about. [The same applies in all area - if the people doing better than you all have very content-heavy sites, hire some copywriters and get them writing some interesting and engaging content; if the people doing better than you have sites built with good quality, semantic markup, and you don't, have your site rebuilt.]

The most important thing is to treat a perceived drop in rankings for what it is: a temporary glitch in your grand plan. Put in a bit of hard work and a little investment in your online marketing and you should see improvements. You were ranking well before, so the chances are good that you will rank well again.

The decision made by these short-sighted bloggers was to display, on their site, a list of recent referrers to each page. I can't imagine any reason why a visitor might be in the least bit interested in seeing this, but nevertheless many sites now display referrers on every page.

As search engine spiders visit sites, they grab the contents of each page they visit. They use this snapshot in their index - meaning that although a page may change every minute or two, a search engine may be using a single copy of a page for several days, or even weeks.

So a referral URL that is on a page when the spiders come to visit can have quite a bit of value, if the search engine visiting uses link popularity in any way (Google uses link popularity, as do many others).

So marketers have started to use programs to visit pages using a fake referral header, to get their URLs listed on as many sites as possible, in the hopes that this will increase their traffic.

However, this renders log files almost completely useless. These fake visitors usually visit from search engines, having searched for a keyphrase relevant to their own site. They skew statistics relating to number of visitors received, the countries used to visit, the technology used to view the page, how users found the page, how long they spent on the site ... and so on.

A webmaster may find their search engine rankings dropping because of this, and they may find search engines have removed them completely. Many sites that use spam techniques are quickly identified and penalised, and penalties will often be applied to sites that link to them as well.

There are plenty of techniques available for blocking referrer spam, and everyone has their favourite. Personally, I use a combination of two techniques.

The first is fairly simple - my referrer log is not indexable. I don't display referrers on the pages of my site. My referral log is publicly available, but search engines are instructed to ignore it. This removes the main incentive for people to referrer-spam my site (the other reason for this type of spam - the hope that the site owner will themselves visit the spamming URL - is less common, because it has such a low response rate).

Second, I use an .htaccess file to block requests from whatever I've managed to identify as either a crawler designed to find URLs to spam or a spamming URL. This is a relatively simple blacklist, and though it cannot work as a long term solution to this problem, it keeps me happy for now.

To implement this technique on your own site, first make sure you are running Apache with mod_rewrite. If you are, create a file called ".htaccess" (just that, not .htaccess.txt or anything else) and paste the following into it:

Update: 14th September 2005

The list below has been expanded substantially over the last year, and now covers much more spam than before. As stated before, this is not a practical solution to the problem in the long term, as this list can only ever get longer and longer, and may become unmaintainable, or even (eventually) slow a site to a crawl as all the rules are processed. However, as of now, it is still a useful tool.

The above will block just about all of the most common referral spam that I've seen so far. I'm adding to the list constantly (last addition: 14th September 2005) so do check back and see if there are updates if you're using it.

One potential problem with this technique, other than that it will, in time, become useless as too many URLs are added, is that there is always a possibility authentic visitors will be blocked. So, on this site, instead of the last line above, I've actually used something a little more user-friendly:

Instead of a "Forbidden" message, this displays a quick note explaining why there has been an error and that the user can click on a link to proceed. If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. You should find you get a message explaining what has happened, and a URL to click if you want to proceed.

And there we have it. With minimum effort (for now), referral log spamming in my site has been almost entirely removed. Before adding this set of rules and scripts, I was seeing around 200 fake referrals per day in my log files. Now, I see about 3 or 4 a week. Hopefully, this will continue until I can devise a better way of protecting against this kind of problem - before blacklists become an impossibility to manage.

In order to protect your chosen directory, you will first need to create an .htaccess file. This is the file that the server will check before allowing access to anything in the same directory. That's right, the .htaccess file belongs in the directory you are protecting, and you can have one in each of as many directories as you like.

You'll need first to define a few parameters for the .htaccess file. It needs to know where to find certain information, for example a list of valid usernames and passwords. This is a sample of the few lines required in an .htaccess file to begin with, telling it where the usernames and passwords can be found, amongst other things.

You've now defined a few basic parameters for Apache to manage the authorisation process. First, you've defined the location of the .htpasswd file. This is the file that contains all the usernams and encrypted passwords for your site. We'll cover adding information to this file shortly. It's extremely important that you place this file outside of the web root. You should only be able to access it by FTP, not over the web.

The AuthName parameter basically just defines the title of the password entry box when the user logs in. It's not exactly the most important part of the file, but should be defined. The AuthType tells the server what sort of processing is in use, and "Basic" is the most common and perfectly adequate for almost any purpose.

We've told apache where to find files, but we've not told it who, of those people defined in the .htpasswd file, can access the directory. For that reason, we still have another line to define.

If we want to grant access to everyone in the .htpasswd file, we can add this line ("valid-user" is like a keyword, telling apache any user will do):

If we want to just grant access to a single user, we can use "user" and their username instead of "valid-user":

A normal and complete .htaccess file might look like this:

Now we have almost everything defined, but we are still missing an .htpasswd file. Without that, the server won't know what usernames and passwords are ok.

An .htpasswd file is made up of a series of lines, one for each valid user. Each line looks like this, with a username, then colon, then encrypted password:

The password encryption is the same as you'll find in PHP's crypt() function. It is not reversible, so you can't find out a password from the encrypted version. (Please note that on page 2 of this article is a tool to help you generate an .htpasswd file, that will help you encrypt passwords).

A user of "dave" and password of "dave" might be added with the following line:

Each time you run an encryption function like "crypt", you will almost certainly get a different result. This is down to something called "salt", which in the above case was "XO" (first two letters of encrypted password). Different salt will give different encrypted values, and if not explicitly specified will be randomly generated. Don't worry though, the server is quite capable of understanding all this - if you come up with a different value for the encrypted password and replace it, everything would still work fine, as long as the password was the same.

Once you've created your .htpasswd file, you need to upload it to a safe location on your server, and check you've set the .htaccess file to point to it correctly. Then, upload the .htaccess file to the directory you want to protect and you'll be all set. Simply visit the directory to check it is all working.

.htpasswd Generator

The .htpasswd file needs encrypted passwords, which can be a problem for anyone without experience with a programming language. For that reason, I've created this simple tool, which, if you enter the username and password you wish to use, will generate the appropriate line to add to your .htpasswd file.

One potential problem with image rollovers, though, is that in order for an image to be displayed, it must be downloaded. Consequently, for rollovers to work smoothly and quickly, all the necessary images must be already available on the user's PC. Otherwise, the rollovers will behave badly, like in this example using large images.

Until recently, rollover effects were achieved through use of JavaScript, and as a result, a plethora of solutions to the preloading image problem in JavaScript are available. However, using JavaScript to preload images, though not a bad idea when using JavaScript to control rollovers, becomes less bright when it is CSS that's controlling them. A user could very easily (and this is becoming more common) have a CSS-capable browser without JavaScript support or with JavaScript turned off.

So there's a clear need for a way to use CSS to preload images or find another way to avoid the problem. Which gives us two relatively simple solutions to our problem.

The first solution is to create a single background image for your element that actually contains both the rollover and non-rollover images, and then position is using the background-position CSS property. Instead of changing the image when the mouse moves over the element, you can simply change the background-position to reveal the previously hidden rollover image. There's a more detailed explanation of this technique over at WellStyled.com.

The other option available to you is to trick the browser into downloading the image before it is required for the rollover. This can be done by applying the image as a background to an element, and then hiding it using the background-position property. The image will then be downloaded but will not be displayed. Then, when the rollover is activated, it will operate smoothly and instantly.

First, you need to select an element that doesn't currently have a background image. If so select an element that does have a background image, you will either end up not preloading the image you are after, or you will prevent the element's normal background displaying. Neither is ideal.

Once you have picked an element to use for this purpose, you need to add the background image. The following CSS can be applied to the element and will place the background image outside the viewable area of the image:

Your rollover image will then be loaded when the page itself is initially loaded, along with the other images. When a rollover is then activated, the image will already be available to the browser and the effect will be instant.