Conversion Rates

Once you've built your website, the temptation is to think of it as finished. Maybe build another one. Certainly aim to drag more visitors, kicking and screaming if necessary, to your shop. After all, if you're making 10 sales for every 1000 people to visit, then getting 2000 people to visit will mean 20 sales! Right?

Not necessarily. Were that true, however, how much will it cost to double your visitor count? For more sites, plenty. What if you, instead of doubling your visitor count, worked to double your sales from your existing customers? How about bumping it to 5%? 10%? Higher?

Think a 5% conversion rate is aiming high? The top online converter in December 2008 was turning 31.1% of visits to sales. Amazon was at 23.7% for the same period. Yes, it was Christmas, but looking at the same report for July 2008 you can see top sellers with conversion rates of 15% to 25%.

9. Know your audience.

You are not Amazon. I repeat, for those of you in the cheap seats: You Are Not Amazon. Amazon is a global household brand with different stock to you. And a different market to you. Something that works for Amazon might work for you. But just because something works for Amazon does not guarantee that it will work for you.

Be aware of what your audience wants. The odds are they are telling you what they want. Do you have a customer support line? Listen to it. If all your calls start with "I tried to order online, but ..." then you have a problem with your site that your customers are actually taking the time to tell you about. How many potential customers who had the same problem simply went to a competitor instead of phoning?

Even if you don't have website problems (and if your conversion rate is 1% to 2%, I'd be shocked if you didn't), that phone line is a window into the requirements of your customers. If they are phoning to find out what it costs to have your blue widget delivered, or to tell you that your site is too slow, or to ask whether the green trousers come with a belt, they are telling you what they want your site to do for them.

8. Don't be afraid to make, and admit to, mistakes.

Most people are terrified of failure, or more specifically of being seen to have failed. Often, even when there is a great potential reward, people will turn down a reasonable idea - just because they are afraid of how people's perceptions of them will change if they are wrong.

I have known website owners who have refused to experiment with ideas because they didn't want to risk reducing their conversion rate (of 1%). I've known others who've refused to experiment with one part of a site because last time it was changed the conversion rate improved (therefore, presumably, changing that same bit in any way could only result in a decrease in conversions).

In the same vein, the sunk cost fallacy is a common cause of low conversion rates. Once someone has made the decision to do something with a site, you will find that person resistive to change. This is especially true where they have paid for the implementation of the change. It is even true when it can be demonstrated that the change in question is directly responsible for a drop in the site conversion rate.

Most people are remarkably blinkered to these sorts of issues - you may even be suffering from one of these blind spots yourself.

Learn to experiment, and learn to let things go - you learn from your mistakes, not your successes. And don't be afraid to try the same reasonable ideas that once failed again at a later date. You will rarely know precisely why an idea failed to improve a conversion rate, but an idea that doesn't work one year may be spectacularly successful the next.

7. Don't jump to conclusions.

It is extremely difficult to predict accurately what changes will have a significant positive effect. So many factors work together, and there are so many differences between users of one site compared to another, that experience can often work against you when you're trying to improve a conversion rate.

Let me take you on a journey through Site A (an online shop I had the pleasure of advising). The owner of Site A had changed his product listing. He used to have ten items per page, and he increased it to 20. The extra products listed on each page gave the users more choice, but his conversion rate didn't change. Next, he added larger product imagery to the listing - unfortunately this slowed the site down considerably, and his conversion rate went down 10%. Finally, he split his categories up, as they were getting a bit crowded and unwieldy. After this last change, his conversion rate increased back to its original level.

After three rounds of changes, the owner of Site A had the same conversion rate for his site, and viewed the only positive change as the last one, change 3. He attributed it to better organisation of products. And here we have the problem - he was wrong. The positive change was actually change 2.

I'm going to run through this slowly, as this is counter-intuitive - after all, how can the change with the measured negative effect actually be positive? The reason the change was actually positive is that these things work together - they are not independent.

The first change to Site A was neutral. When the second change was made, 20 extra images per page were being loaded. These images slowed the site to a crawl. When the third change was made, several categories contained fewer products than before - around half of his categories had 10 products or less - and these new, smaller category pages loaded faster. Much faster. These smaller category pages were much faster to load, and the extra imagery did a good job of helping to sell to the users. The increase in sales came from these pages, after the third change, came from these smaller categories.

After change 3, Site A still has the same conversion rate as it did before starting. But once the changes were correctly identified as positive (change 2), negative (change 1) and neutral (change 3), it is a simple matter of reversing the negative. So, the new categories were kept, the images were kept, and the number of items per page was reduced back down to 10. The final result of all of this was a positive change to the conversion rate.

And the moral of the story - it is very easy, in hindsight, to attribute positive and negative effects to changes, but it is extremely difficult to do so accurately and correctly. Examine the effects of changes and, if you can, isolate and test individual effects.

6. Track everything.

Many people make the mistake of only tracking small pieces of information. Or tracking everything but only looking at a small part of the data. Data is your friend, and you need to understand what the different metrics available to you are actually measures of if you want to have any clue what effect your work is actually having.

Most important, of course, when looking at the conversion rate, is the volume of sales. Some people only look at that number. But other numbers can tell you about how useful your site is throughout the shopping process.

The percentage of repeat visitors tells you something, about whether you are engaging visitors early in the purchasing process and bringing them back for the sale. A high bounce rate indicates pages that are failing to deliver on their promises. Add to cart rates, cart abandonment rates, login vs registration vs abandonment rates, product removal (from cart) rates - all of these will identify areas of your checkout and purchase process that are underperforming, or improving.

5. Get On With It!

There's no better way to ensure an unchanging conversion rate than doing nothing. Don't wait - start today. The more you experiment, the more you will learn and the more opportunities you give yourself to improve your numbers. Get in the habit of testing and tracking constantly. There's always room for improvement.

4. Make the most of your space. But avoid clutter.

How many sites have a nice piece of prime real estate at the top of the page simply saying "your cart is empty"? Too many. The user usually knows their cart is empty. They've not added anything to it. Why occupy a prime sales spot with worthless information? You could hide the cart, or link to a "getting started" article - an introduction to your products, or recommendations.

A cart is designed to catch the eye - but before something is added to the cart, this is an unwelcome distraction from other, more relevant, calls to action. You want your users clicking through to products and then adding those products to their baskets - and you don't want to distract them in the meantime with an empty cart space or a useless "checkout" button.

3. Unobstructive convenience is key.

When I worked with Juno Records in London, one of my most startling discoveries was that far from being obsessed with speed, some of the users of the site would spend a day - or more - building a cart of products.

There was a good reason for this - the shopping page of the site at the time was several megabytes - but users would happily load the page, and browse the mammoth list of products. They could listen to snippets of tracks, and they would browse the entire list - top to bottom - listening to snippets and adding things to their basket.

One of the first versions of the new Juno site had a cart on a separate page to the products (as is common for many ecommerce sites). When you added a new item to your cart, it took you away from the page you were on. Given that users had spent time at this point browsing through the product list, and by moving them to a new page you lost their place on that product list, this was not a usability improvement.

Likewise, some users would leave their cart half-built and return to it the next day. Or several days later. The default session timeout on one early version "broke" the site for those users.

One trick to avoiding problems like this is first to know your users. Their habits may not be what you expect.

But more importantly than that, think "unobtrusive convenience". Make the user's life easier, but don't make them work unnecessarily for that convenience.

For example, save the user's cart for them, but don't force them to create an account to do so - cookies are your friend. The user should be able to leave, and then come back later and continue shopping. Check that moving users to a new page when they add an item to their cart doesn't disrupt their shopping. Don't force users to create accounts when they want to buy - offer them the option but don't let it get in their way if they're not interested.

2. Be smart with discount codes and coupons.

Discount codes and coupons can be a great way to drive sales. They can also backfire, making customers who would otherwise buy feel like they're not getting a good deal, or missing out on a saving. This isn't good for business.

There are several solutions to this, but ideally you should look to downplay discount code redemption in the checkout process. One good trick is to ensure everyone has a discount, even if it's not a great one. Another is to move code redemption into the account system instead of the checkout system. Yet another is to identify users with discounts (if you can) when they log in and then offer to apply the code if they have one available.

1. Be clear about your pricing strategy.

Pricing is tricky for almost every industry. Price yourself too high and you'll lose out to cheaper competitors. Price yourself too low and you'll end up appearing cheap. Price some products high and others low and you'll look like you're pushing some brands over others, or worse that you're expensive for some products (which tends to make people think you're expensive for all products, even if you're not).

It's important to be clear with your pricing - if you're aiming to be the cheapest then be the cheapest. Offer price comparisons or price matching. Tout your prices above all else. If you're going for higher prices and selling quality of service, don't shout about your expensive prices.

This also applies to shipping. Delivery costs are almost always an additional expense incurred during the checkout process. If you add delivery costs during the checkout process, then exactly when you don't want people reconsidering their purchase, you're giving them a reason to. Make delivery free or flat rate and add it from the start of the shopping process. There's no harm in reducing delivery cost - and telling the customer you've done so - during the checkout process.

Bonus!

One excellent (and practical) way to increase your website conversion rate is to add consumer reviews to your store. They are a proven way to increase sales, and they have an excellent positive effect on your search engine optimisation work. A service like FeedbackFair will give your reviews extra credibility.

Your conversion rate is a measure of the number of potential customers that go on to buy. In the context of a website, it is usually the percentage of visitors that make a purchase. Many websites concentrate solely on increasing the number of visitors they have, when often they have fairly simple problems with their site that, if solved, would have a huge effect on their conversion rate and improve their site's bottom line at minimal expense.

Improving a website conversion rate can be relatively simple. Here are 10 techniques for doing just that:

10. Make The User's Life Easy

Let's start with something that sounds simple, but apparently is too complex for many companies to get right. The more difficult you make your web site to use, the less people will buy from you.

A well designed website should aim to prevent nobody from buying - to allow 100% of the people who want to buy to do so. So where do they go wrong?

• Accessibility
  Making a site accessible is a legal obligation in many countries. Despite that, inaccessible websites are still being created. That can affect your sales, depending on how inaccessible you are, as visitors find the site impossible to use and go elsewhere (and end up recommending one of your competitors to their friends as well). A fairly typical inaccessible site could be losing 5% of potential sales because of this. (A really inaccessible website could even prevent search engines indexing it, giving a far higher amount of potential lost sales.)  
• Browsers
  Many designers only pay attention to Internet Explorer. The justification for this is usually that 99% of the site's users use IE. It never seems to occur to the designers that perhaps the reason they have so few visitors with other browsers is that their site is fundamentally broken - it doesn't work in anything else. Percentages of people not using IE varies from site to site - over 60% of visitors to this site use an alternative browser, for example. The number most often quoted though, is that 80-85% of web users are using IE on Windows, which means that an average site that doesn't work in anything else could easily be losing 15-20% of sales.  
• Be Bold!
  What happens when a user decides to buy a product? They add it to a shopping basket. How do they add it? They click a button or link (usually a button). What happens when they can't see the button? They go elsewhere. There are some users who are still uncomfortable scrolling. Having things above the fold is still important. And yet there are still plenty of sites out there with buttons that are too subtle, or don't say the right thing, or are hidden away at the bottom of the page. "Add" is rubbish button text. "Buy" is ok. "Add xxx To Your Basket" is great. "Add xxx to Your Basket" in big letters on a big, bright button, near the top of the page, is even better. Calls to action, like this, don't have to be gaudy or tasteless, but they do have to be obvious and clear. Sites I have worked on where just the call to action was changed have reported anything from a 1% to 30% increase in sales as a result.  
• Usability
  If your potential customers want to find out more before they buy, can they? Is it obvious to the user where to go to find the technical specs on your products? Are they online at all? Are they in PDF format? Can users even find your products in the first place? This is probably the most common mistake I see on any website - a complete failure to think of what the user wants and needs, and how they might use a site. Plenty of sites have product pages with a photo and some sales patter - and nothing else. Anything from 1% to 99% of potential sales can be lost through poor usability.

When you combine all of the problems above, it becomes fairly clear how easy it is to have a site perform poorly. Make your site accessible, make sure it is usable, make sure it works in common browsers, and make your calls to action clear and unambiguous, and you should be in a position to start converting the people who want to buy.

9. Be Clear, Open and Honest

If you have a product out of stock, say so. Few things annoy users as much as reading all about a product they are after, adding it to a cart, and starting the checkout process - only to find out the product isn't actually available.

The same applies to pricing - a user might spend $100 on a product, but when they find out the shipping is $100 on top of that, they are unlikely to continue the sale. Showing delivery pricing is tricky business, but not impossible. An Ip to Country database will allow you to work out where a user is from and show them a likely delivery cost, for example. If you can't do that, show delivery prices for the countries most appropriate to you - where your products are most often delivered, or for major world regions.

8. Don't Waste Time

One of the biggest mistakes sites make is asking for too much information. Your conversion process may be sale, or it may be a request for information. Either way, don't waste the user's time asking for things you don't need to know. This is, of course, doubly important when it comes to asking for information the user deems private, and that they don't want to give out without good reason.

You don't need to demand the user's email address before letting them download a PDF. You don't need their phone number when they fill out an email enquiry form. A user may not want to buy from you twice - so why make them create an account so they can buy again later before processing their first order? You can give the user the option to do all of these things by all means, but make sure it's not compulsory.

7. Help The User Trust You

Most people are still cautious when buying online, and rightly so. There are plenty of people you really shouldn't give your credit card information to! It's important to give the potential customer every reason to trust you.

An address - bricks and mortar, not a P.O. Box - is a good start. A phone number, with people answering the phone, also helps. Showing a privacy policy and explaining shipping procedures clearly can also help the user to trust you. If you have a SSL certificate, show the "VeriSign Secured" logo to the user.

Design and content also play a part in trust. A poor design gives off an unprofessional feeling. If a company can't afford a decent website, or won't spend the money on it, how can a user be sure their order will be treated with the importance it deserves? If content is inaccurate or badly written, the same applies - show that you take pride in what you do.

6. Have a Clear Returns Policy

Returns on the web are, and are likely to remain, a major issue for consumers. With a bricks and mortar shop, the customer knows where the shop is and that to return the product they simply have to go back there and explain the problem. With the web, this is more of an issue. This is especially true for clothing (where people cannot try things on before buying).

Users are impressed with sites with a good returns policy and are more likely to buy from them. Have people phone for returns - they can then explain the problem to a real person, which is always a good first step. Free return shipping is usually a good option, if commercially viable. People don't like to pay to return things, especially if it is a mistake by the retailer. Finally, give the user plenty of time to return things. 28 days is fairly common, but if it takes you that long to deliver a product, what use is the return policy? 28 days from the date of delivery is better.

5. Keep the User Informed

When somebody buys something online, they want to know when it's going to arrive at their door. People are impatient, after all. Giving them an estimated delivery date during the checkout process is a good start. Emailing them when their product is dispatched is great. Giving them a tracking number if using a delivery service that supports online tracking is even better. Keep the user informed at every step of the process, before and after sale, about as much as you can.

How will this improve your conversion rate? Leaving the customer happy once they have made a sale means they are more likely to speak favourably about you later. They may even recommend you to their friends and within online communities. They are also far more likely to buy from you again.

Think about it like this - if a salesman is doing their absolute best to help you, and to make your life easy, and answering your questions, you might buy what they were selling. If they completely ignored you after you'd bought from them, how would you feel about them? They might well have undone all the good work they put in, because once you'd completed your purchase they see no immediate value in you. A company that shows it cares about their customers, even after they've finished shopping, will make a user far happier and far more likely to return.

4. Offer Different Payment Options

It might sound obvious, but you should offer the user a reasonable selection of methods of payment. Not everybody has a credit card, and those that do don't always want to use them. You don't have to accept cheques, but when deciding on payment methods, consider alternatives to the usual methods. Make the user's life easy and give them what they want.

3. Improve the Value of Visitors

People that buy from you are doing so because they like what it is they see. If a user adds a product to a basket, show them other things they might like as well. If they are viewing a product, the same applies - show them similar items. While they might not buy the product they first saw, other similar ones may not have issues that put them off the first. Upselling and cross-selling are tried and tested sales techniques, and there is no reason not to use them on the web.

2. Be Memorable

A good site will include information. A poor one is just an online catalogue. Information (articles, advice, reviews and so on) all help the user early in their buying process. Users start with research online, just as they do offline. If you can make contact with the user at that stage of their process, and give a favourable impression, there is a good chance that they will come back and buy from you when they finally decide to make a purchase.

Being memorable, and making sure you stick in the user's mind, is dependant on a lot of factors. You must have a USP (see the next point), and branding is important (no good if your visitors remember why you are great but don't remember your name), as well as the quality of your site and information.

1. Know Your USP

Finally, the most important point of all - your Unique Selling Point (USP). Your USP is what sets you apart from your competition. If a visitor goes to several sites looking for a product, why would they decide to buy from you instead of somewhere else?

Many companies do not know their USP. Almost all companies have one, but not all of them are aware of it. If you are a family run business, that's a potential USP. Great customer service, low prices, products that can't be bought elsewhere, free delivery, great support - all of these are USPs. Tell your users what yours is. Shout it from the proverbial rooftops.

Part 2

In March 2009, part 2 of this series was added: Nine More Tips for Improving Your Website Conversion Rate.

Bonus!

One excellent (and practical) way to increase your website conversion rate is to add consumer reviews to your store. They are a proven way to increase sales, and they have an excellent positive effect on your search engine optimisation work. A service like FeedbackFair will give your reviews extra credibility.

It is amazing how many people hire online marketers without the faintest idea of what online marketers actually do. Search engine optimisation (SEO) is fairly simple - SEOs will try and improve your site's performance, usually by trying to leverage their knowledge of how search engines work and tricks they can use to make sites seem more relevant than they actually are to specific keywords.

Marketing online, though, need not have anything to do with search engines. Search engines are irrelevant - good positions and traffic are a by-product of effective online marketing.

Unfortunately, after educating a client on what online marketing is, they usually assume that if they pay you a few hundred pounds, you can make their site compete with the very best out there.

• Client:
  'Ok, I see. Great positions aren't necessarily worth much unless there are customers searching for those keywords.'
• Me:
  'Right. We want high traffic, but not if it's not going to be bad for your bottom line. Traffic that doesn't convert to sales just costs you money. Same applies for phrases people never search for. No point being number one for the phrase "fish banana druid" - it's likely to get you as many customers as peeing on people that walk past your shop will.'
• Client:
  'Ok, so if I pay you, say, £300, how long before I'm at number one for this list of relevant phrases?'
• Me:
  'You wouldn't get in a boxing ring with Joe Calzhaghe after jogging a couple of miles and doing a few push-ups, would you?'
• Client:
  'Well, no.'
• Me:
  'Exactly. To compete with the big dogs, you need to think bigger. Your site is a 10 stone weakling at the moment, and the aim is to turn it into a champion. It needs to be Rocky Balboa. You won't get the top spots quickly - this takes time and hard work. And it's not cheap.'

People are obsessed with money. Absolutely obsessed. Even more so in a company environment. The chances are the most of the time, the person you are talking to at a client (or potential client) company is not the top dog. They have to justify their decisions, and they certainly have to justify what they spend.

The problem is that the way most people look at SEO (and they are thinking SEO, not marketing - it's up to you to show them the difference) is that they're going to pay a certain amount of money for the top spots for certain keywords. You can guarantee they've been told another company will guarantee 10 number 1 positions for $50.

This is where ROI comes into play. ROI stands for "Return on Investment". Paying $50 for a $0 return is a bad idea - but people do it all the time, because it's cheap. Paying $5,000 for a $50,000 return is a great idea - but people gasp at the very idea they could spend that much in the beginning, despite the potential.

In order to measure a return, you need to use tracking. If you're focussed on natural search, measure natural search traffic. See how many people come to the site, and where from. See where they go in the site. See if they view products, add them to a basket, and complete sales. See if they view products then come back weeks later to buy them. Measure that over time and you can tell a client exactly what effect your marketing campaign is having - and you will be able to show them what they are getting for their money. Usually, telling a client you are going to do this will also put their mind at ease - much easier to spend money on someone when that person tells you how they're going to measure their success. Most companies involved in SEO and online marketing focus on positions, not results.

• Client:
  'That's good to know. If I can see what's going on, I can give hard numbers to my boss. I'd rather tell him we have 10% more visitors and 20% more sales than tell him we're in top positions for our target phrases but traffic has gone down.'
• Me:
  'Woohoo! You've taken your first step into a larger world.'

The other thing to bear in mind with money conversations is that most companies think of their site like a brochure. They think of it as a print-like cost, where they pay a fixed sum and that's it. They put the site up, leave it, and expect results. They should be thinking of a site like a salesman. A salesman that never sleeps, rarely gets ill, and can handle virtually unlimited enquiries. As such, they should be thinking of the money they spend more like a wage.

• Client:
  'We're spending $200 a month on our site now for hosting. Are you saying we should be spending a lot more?'
• Me:
  'What would you pay a salesman with the figures your site has, ignoring PPC?'
• Client:
  'Probably $3000 a month.'
• Me:
  'Then that's what you should be spending on the site. As the figures get better, spend a little more. Remember that that needs to include redesigns, hosting and other costs.'

(Note: PPC is something of a difficult subject to bring in to a monthly spend on a site. You should have a monthly spend on PPC, but it should be managed as a separate entity.)

The same traffic you are monitoring to see where site visitors are coming from and what they are doing when they reach the site can also give you some good places to start making changes. Break the traffic down by area, by language, by time of day (user time of day, not server time of day), and track who converts to a sale and who doesn't. Track people through the sales process, and watch which links they click to navigate and buy products.

This will tell you a huge amount about the current users of the site. It will show you quick wins, opportunities, and highlight problems. Forget search - if on your first day marketing a website you can spot that there is a problem with the site checkout process and get it fixed, you could double sales from existing users. That's a good start to any campaign.

Look at language and area closely as well. If a site is getting traffic from the US, but only sells to the UK, look at similar companies only serving the US and strike a deal with them. You direct US traffic to them, they direct UK traffic to you, and you both do slightly better.

Check browser usage stats, especially if the site is a tables-based dinosaur. The chances are that it is an inaccessible mess. Get it cleaned up! Semantic markup is key - it allows user agents (browsers, search engine spiders, screen readers) to attach specific meaning to different areas of a page. Unlike with tables, semantic markup allows you to differentiate between a header and normal content, or to identify an address. Accessible coding is likely to draw attention, and should help you retain a higher percentage of your visitors, and should help reduce the running costs of your website (lower bandwidth bills and quicker turnarounds on redesigns, for example, both save you money).

• Client:
  'I don't care about different browsers though - they only make up 1% of my traffic. Everyone else uses Internet Explorer'
• Me:
  'Perhaps it is because your site doesn't work in other browsers that that number is so low. Even if you do have 99% of users on the same system, the other 1% is still important. Techies use different browsers and operating systems. Techies are also often the people who are asked by their families if they know a good site to buy something from. Many directory editors are in the same boat, and techies can create links to your site.'
• Client:
  'Ok, techies are important. But do I need to care about blind users and all that accessibility stuff?'
• Me:
  'Yes, of course. It's a legal obligation for one thing, but users with sight problems make up a far larger proportion of your audience than you might think. They have a voice too - and it's far harder to undo the damage some adverse publicity can do than it is to make a site work properly in the first place. Finally, search engine spiders are blind users with no JavaScript support.'

Dynamic sites are slightly trickier to improve. Most of the time, they are restricted, with the original authors not allowing access to the website code. Even if access to the code is allowed, changes may be overwritten later or worse cause immediate problems on the site. That said, making a site easier to use is important, and often dynamic sites are not easy to use.

Look at the pages users visit in the site, and how they get there. Look at the products they buy and spot themes. Use that information to make the important sections and products easier to find and organise. For example, if listing products, don't make people click through 4 levels of navigation to find them - improve the product navigation. Once they get there, allow them to reorder the page according to what they consider important, be that name, price, manufacturer - whatever is possible.

Remember also that people like to tell other people about things they find. If a user likes something on your site, they may email the address of the page they are on to a friend. Most people use forms to set the ordering criteria of a page. That means that the user will be sending a friend a URL that will show that friend something different to what the user currently sees. Make life easy for your users - use URLs, not forms, wherever possible in a site.

• Client:
  'I am curious about one thing. We're already really well ranked for the name of our main product, and lots of people search for it. The people that visit our site tend to buy the product. But I can't help feeling that there should be more people coming from the engines. Any ideas?'
• Me:
  'Yes. The Overture Search Term Suggestion Tool', capitalising my speech for no good reason, 'shows that millions of people search for that phrase. I can see you have a top spot. And your traffic is surprisingly low, but converting well.'
• Client:
  'So I'm not imagining it then - we have a problem?'
• Me:
  'Yes, we do. When a user looks at search results, they scan the first two or three words of each link. Your link says "Arthur Jackson Ltd. Sheds and other garden products." That comes from your page title.'
• Client:
  'And that's bad?'
• Me:
  'Most people will only glance at "Arthur Jackson Ltd". You need to show them, in the first two or three words of your page title, that you have what they are looking for. And you're not doing that. The user has no reason to click on your link ahead of all the others they see.'

Titles are tricky. They're important to the user, they provide the text for bookmarks, they appear in search results, and search engines use them as part of ranking algorithms. You need for fit branding into a title, and describe a product, ideally also incorporating a call to action. Tricky stuff. But not impossible.

First, consider the brand. Most companies think their company name should be the first thing in a page title, even if the rest is unique for each page (as it should be). However, unless the company has a household brand name, the company name is irrelevant to the searcher. They're looking for a product (or the answer to a question), so show them you have it.

Next, remember that as titles are used as the text for bookmarks, links and appear in search engines, they should, when taken out of context, by themselves, leave no doubt what a page is about.

A good example of a title is:

• "Norwegian Blue Parrot - Buy Norwegian Blue Parrots from Mr. Praline's Pet Shop".

You've included the all-important product name twice in the title, along with a call to action, a hefty dose of branding, and not added irrelevant information. It's a title that tells the user straight away what the page is about. No messing around.

• Client:
  'Ok, the titles need sorting, but what about the content of the site? I keep hearing that "Content is King".'
• Me:
  'Content is, ultimately, King. Sites with lots of great content will, over a decent time period, far outperform sites with no original content. But content doesn't just have to be on site ...'

Product is important. The object you sell though is only half of the picture. A user will want support from you. They will want information. They may want news. All of this is part and parcel of the package a company offers. Your site needs good, visible support (including a phone number), as well as plenty of good, original information. Guides to products, online manuals, FAQs, advice - there are always areas, in any industry, where content can be added.

Content need not be solely posted on the website either. Big news should be released as a press release, and there are plenty of services that will distribute press releases for you. These will be reproduced all over the web, allowing more and more people to hear of the company. Most press release services will allow you to embed a link to a site in a press release, generating more direct traffic as well.

When writing content, or advising on the writing of content, remember that it is not about keywords. Sure, keywords are important, but there is more to it than simply stuffing as many keywords into text as possible. Content needs to answer questions - to provide information. It needs to give a user what they are looking for, and they need to feel that it has done that. Content that is written for SEO can read very badly with too many keywords in, and can mean that although more people see an article, most of them leave the site straight away to find a better one.

A good way to add content to a site is a blog, or a news section. Aside from adding plenty of information, this gives a great opportunity to connect with the user. Consumers are constantly being targeted, from every angle, by companies anxious to take their money. Sometimes they get trodden on. When adding content to your site, stay on the side of the average consumer. Recently, in the UK, the energy companies all raised their prices dramatically. Sites that allow users to compare fuel prices almost all missed a great opportunity to have themselves noticed - not one of them posted a decent news item denouncing the changes as unnecessary or over the top. They all simply commented on the change factually.

While on the one hand, some of these companies may be unable to comment in this fashion (and many companies have strict policies regarding neutrality and customer perception), at least one should have been able to stand out by taking a clear, customer-supporting position on the issue. That is the kind of thing that gets companies noticed and remembered, and spotting opportunities like that is key to a good marketing strategy.

Not all content need be inflammatory of course. It does need to be unique in some way, however. It can be controversial, but it could also be definitive - the ultimate and complete guide to a topic. Controversial content is interesting to the user, and definitive content is just plain useful - either makes for good content for any website.

Users go through different stages when buying products, and one of the early ones is a research stage. There is always a good chance that a user will come back to the same place that helped them or impressed them when they were doing research to buy what they were looking for. This is branding - associating specific ideas and feelings with your company. You want your users, when they revisit the web to make a purchase, to think of your company first.

Which brings us nicely to our last, and most important point. Why would a customer think of any company first, ahead of any other. Content will help, yes. A nice design might even make a difference. More than anything else, though, customers pay attention to the company that stands out from the crowd - the company that is different, that offers them something nobody else does. Often known as a Unique Selling Point, or USP, this is the thing that makes you memorable, or if ignored helps you blend into the crowd.

• Client:
  'But we don't have a USP. How do we get one?'
• Me:
  'Well, hang on one minute. You say you don't have a USP, but is there nothing about your product that makes it better than the alternatives?'
• Client:
  'Well, we sell Norwegian Blue Parrots. They're all the same, really. Although a rather large proportion of our competitors appear to sell mostly dead ones.'
• Me:
  'There you go then. Your USP is that your product is, in fact, not dead.'
• Client:
  'By that reasoning, a USP could be almost anything, when put in the right light. And when did we turn into a Monty Python sketch?'
• Me:
  'Be quiet about the Monty Python thing. Yes, though, a USP can be virtually anything. It can be quicker delivery than competitors, better products, better customer service, a freephone enquiries number, or simply the people that run the business. Almost every business has a USP - although most of them don't know what it is.'

Many businesses don't know their own USP. They can't tell you, when you ask, what makes them different. Many of them will just say "because we're better than the others", but can't explain why. Usually, however, a quick chat will reveal what makes them stand out. Whatever the USP is, it needs to be clear and obvious on the website. The customer can't miss it, because if they don't know what makes one business different from another, they're not going to remember it.

• Client:
  'What about search? You've not told me how to get my site to the top of the search engines!'
• Me:
  'Let's review, shall we. You've changed your site substantially, so that it meets current standards and you can sell to more of your users. You're showing your clients why you are better than your competition. You've started releasing press releases, and adding content to your site. You're championing the cause of the common man, increasing link numbers and getting people talking about your business. And you know how your users find your site, and what they do when they get there.'
• Client:
  'And?'
• Me:
  'You're positioning yourself as a great resource for your market. Your search engine rankings will come as a direct result of everything else you are doing. You're going to perform well in search, as a direct result of good marketing.'
• Client:
  'I'll get my chequebook.' (Hah. As if.)

Anchor text is the text used to link to another site. In this example - Google Web Search - the anchor text is "Google Web Search".

Atom

Atom is a file format used for web feeds. It is a type of XML document, and is used in syndication.

Black Hat

Black Hat is the term used to describe techniques used by some search marketers to promote websites. These techniques are those that go against guidelines published by search engines, and in many cases their use can result in a site being penalised or removed from search engine listings. Black Hat is the opposite of White Hat.

ccTLD

A ccTLD is a country-code top level domain. .uk, for example, is a ccTLD, as are .au (Australia), .de (Germany), .fr (France), .ca (Canada) and .nz (New Zealand).

Click-through Rate

See CTR.

Cloaking

Cloaking is a technique used to show content to a search engine and different content to a user. The content shown to the engine is usually designed to help a page rank very well for a certain phrase or word, and the content shown to the user usually designed to maximise the conversions from that page. Search engines dislike this technique and many sites are banned for using it. It is a Black Hat technique.

Conversion

A conversion is when a website user completes a specific goal. With some sites that can be to complete a sale; with others, to sign up to a newsletter; and with others to make an enquiry.

Cookie

A cookie is a small text file stored on a website user's computer. It identifies a repeat visitor to a site, often with a unique code, allowing people to shop online and removing the need to log in to sites repeatedly. Cookies are often considered dangerous by less experienced web users. You can find out more about cookies in Are Cookies Dangerous?

CPA

CPA stands for "Cost-Per-Action", and is a form of advertising model. The idea is that an advertiser pays a specific amount for each successful conversion, be that a sale or a signup.

CPC

CPC stands for "Cost-Per-Click", and is a form of advertising model. The idea is that an advertiser pays a specific amount for each visitor referred to their website, regardless of whether that user converts to a sale or not.

CPM

CPM stands for "Cost-Per-Mille", and is a form of advertising model. The idea is that an advertiser pays a specific amount for every thousand times his advert is seen on a site, regardless of how many of the users who see the advert click on it and visit the advertiser's site.

Crawler

See Spider.

CTR

CTR stands for "Click-through Rate". It is an indicator of the percentage of people who see an advert who actually click on it. For example, if one out of every hundred people who view an advert click on it, the advert with have a CTR of 1%.

Directory

A directory is different to a search engine in that it organises the sites it lists in categories. Sites are usually added by hand, rather than found using a spider, and often a small fee is charged for this addition.

Data Center

A data center is a large collection of computers that hold information for a search engine. Major search engines have several of these around the world. Their purposes is to process search queries.

Doorway Page

A doorway page is a page designed specifically to rank well in search engines. Often a visitor to a doorway page will not notice they have visited one, as they will be sent straight on to the target page instantly. Use of doorways is a Black Hat technique.

Feed

A feed is a file that users can download that contains information about recent updates and additions to a website. Often these feeds are used for syndication purposes. Using feeds and programs designed to use feeds, users can often keep up to date with many hundreds of websites.

FFA

FFA stands for "Free-For All". It is usually used in conjunction with links pages that allow anyone and everyone to add a link to the page.

Google Dance

The Google Dance is the name for the process Google used to go through very regularly when it updated an algorithm. As various data centres around the world were progrssively updated, people would be able to make the same search several times in succession and see different results each time. The Google Dance does not happen as often now, but can still be seen when major changes are made to the Google infrastructure or algorithms.

Hit

A "hit" can mean one of two things.

• When searching the web, a hit can be a result found by a search engines that matches the search criteria.
• In analytics, a hit is when a file is requested by a server. Some people have used hits as a measure of website traffic, however hits to a server include images and repeat visitors, and are a poor indicator of traffic. One thousand hits very rarely equals one thousand visits.

IBL

IBL stands for "Inbound Link", and refers to a link pointing to a website from a separate website (unlike an internal link, which refers to a link within one website pointing to somewhere else within the same site).

Impression

Impression is the word used to describe a single viewing of something. A page impression would mean a single view of a web page. In advertising, one impression is a single view of the advert.

Keyword

A keyword is simply a word used to describe a page. It can also be a word used by someone trying to find a site, entered into a search engine.

Keyphrase

A keyphrase is very similar to a keyword, except that it is a phrase made up of several words.

Keyword Stuffing

Keyword stuffing is the practice of repeating a keyword (or keywords) far too many times throughout a page. It may be that the keyword is repeated so many times in the text that as a result the text reads badly. It may be that it is repeated lots of times in meta tags, or elsewhere in code, or it may be a combination of these things. Common practice in the late 90s, this is now considered a technique that may harm a site more than help it.

Link Building

Link Building is the process used to increase the number of links to a website. This can include submitting a website to directories, creating more content for a website, link rental, and many more techniques. Most search engines now use link data extensively in their algorithms, and so link building has become far more common.

Meta Data / Meta Tag

Meta Data is information held about a page or document. It is usually held invisibly within the page, and may include a description of the page, a list of relevant keywords, or the name of the author. For a full explanation of common meta tags, and how to work out which ones are worth using, please read Meta Tags.

Page Title

A page title is an important part of a page - it is usually the part of the page that appears as a link in search results. It is usually visible in the title bar of your browser while you are viewing a page.

PageRank / PR

PageRank is an algorithm, developer by Larry Page and Sergey Brin, founders of Google. It allows you to find the "best" pages of a group of pages by looking at how the pages link to each other. The more links a page has, the better it is considered, and the more important its links, in turn, are considered. PageRank is named after Larry Page.

Pay Per Action

Pay Per Action advertising is the same advertising model as CPA, in that an advertiser will pay every time a user completes a specific action.

Pay Per Call

Pay Per Call advertising is a subset of Pay Per Action, and is the same advertising model as CPA, in that an advertiser will pay every time a user calls a specific number.

Pay Per Click

Pay Per Click advertising is the same advertising model as CPC, in that an advertiser will pay every time a user clicks on their advert.

PFI

PFI stands for "Pay For Inclusion". Some engines will charge sites to be listed at all in their results (notably Yahoo for many years). Prices vary greatly, and some engines charge annually, where others charge a one-off fee. This is a far more common feature of directories than search engines.

PPC

See Pay Per Click.

Robots.txt

A robots.txt file is a simple text file that contains instructions for search engine spiders. It can tell specific spiders to slow down, or not to index specific area of a site. For more information, please read robots.txt.

ROI

ROI stands for "Return on Investment". It is a measure of the success of any marketing campaign. A marketing campaign that cost ?10,000 but made ?3,000 would obviously have a low ROI. A marketing campaign that cost ?10,000 but made ?100,000 would have a high ROI.

RSS

RSS is a type of XML file, and is the most commonly used file format for website feeds.

SEM

SEM is an acronym of "Search Engine Marketing". SEM is a broader topic than SEO, and can include, for example, an online PR campaign or PPC (and other forms of) advertising.

SEO

SEO is an acronym of "Search Engine Optimisation", and is the art of altering a website to improve a site's performance in search engines (note: an improvement in performance does not equal an increase in traffic!).

SERPs

SERPs is an acronym for "Search Engine Result Pages".

SEs

SE is an abbreviation of "Search Engine".

Site Map

A site map is a page, or set of pages, on a website, designed to help users and search engines find their way around a site.

Spam

Spam has many different meanings on the web. The most common meaning is related to email, where spam describes unwanted email, often commercial in nature, and often sent out indiscriminately to millions of people at once. In a search engine context, spam refers to pages that are listed out of place. This can mean pages that are found for keywords unrelated to their content. It can also mean pages appearing unnaturally high in search engines. These pages are often promoted using Black Hat techniques, especially cloaking and doorway pages.

Spider

A spider, also often called a "crawler", is a program created by a search engine to index pages on the web. It visits pages on the web, collects their content, and finds links within that page. It then adds the links found on that page to those it intends to crawl.

Splash Page

A splash page is an introduction page to a website, often created using flash. They are much derided, as they slow down access to a website and often provide no useful information to the user.

Stop Word

A stop word is a word that is ignored by the search engines. It is a word that appears so often on the web as to be useless to a search engine. Examples include "a", "and", "I", "you" and "it".

Syndication

Syndication is where a website makes information available for others to use. In the majority of cases, the information available is a list of the content most recently added to the site (a feed), to allows visitors to keep up to date easily with new content added to many sites.

Text Link Ad

A text link ad is a type of advert on a website, placed in return for a simple monthly fee. These types of advert can have a positive effect on a website's SEO campaign, and can directly generate traffic to websites.

TLD

A TLD is an acronym for "Top Level Domain". .com, .org, .net, .biz, .info, .name and .pro are all examples of TLDs. They are usually global TLDs, unlike ccTLDs, which are country-code domains.

URL / URI

A URL (Uniform Resource Locator), sometimes (more correctly) referred to as a URI (Uniform Resource Identifier), is in basic terms a web address. For example, "http://www.addedbytes.com" is a URI.

Visit

A visit is different from a Hit or an Impression, in that it indicates a single person's visit to a website. A visit may include many page impressions, and many hits.

White Hat

White Hat is the term used to describe techniques used by some search marketers to promote websites. These techniques are those that adhere to the guidelines published by search engines. White Hat is the opposite of Black Hat.

XML

XML is a file format designed to create files that are easy to share and understand.

For hobby webmasters, this is generally not a problem. For anyone making money online, though, it can be extremely nervewracking. For those whose livlihoods depend on their websites, losing all search engine traffic can be a devastating blow.

Unfortunately, a great deal of the threads and dicussions on this topic often result in a large amount of misinformation. For example, as a result of one recent Google update, many sites had lost significant rankings. Some forums were claiming that the specific industries had specifically been targetted and sites in that industry had been penalised in some way. Some claimed that Google had "lost" a serious amount of data, or had re-added old data, and that was what caused the change. There are as many explanations for loss of traffic as there are sites that have dropped out there.

Unfortunately, with all of the wild ideas and crazy theories being bandied around, the average site owner has a very hard time working out first what has happened, and second what to do about it.

The very first thing to consider when looking at the effect of a shift in algorithms is that a change rarely affects all ranking criteria at once. They rarely, if ever, target a specific industry, even though the effect of a change on a specific market may be far greater than in others (this is especially true in ultra-competitive arenas, such as real estate, finance and the adult industry, where those at the top are often precariously balanced, and a tiny change in algorithms can mean major changes to the SERPs).

Fixing The Problem

Before anything else, it is important to make sure there actually is a problem. The forums usually first fill with these types of posts during an update. However, while the update is going on the SERPs are in a state of flux. Sites can appear all over the place during an update, so save the panic until the update is over. Updates can last days, and it is a good idea to watch a few of the SEO forums to find out when an update has finished.

If the update has finished and a site has definitely dropped, it is rare that it will be able to regain the exact same (or better) traffic within a short space of time. If an algorithm change has caused a site to be dropped, the chances are that one specific thing that was making that site rank well (for example, rented links) has been devalued. If the only thing that was making a site rank well has become less important, there are probably no quick fixes.

Is It a Penalty?

The first thought to cross most peoples' minds when sites lose traffic and drop down the SERPs is that there must be a penalty applied to their site. Penalties are very real, yes, but there is no reason to suspect you have had a penalty applied unless one of the following is true:

• You have been doing bad things. If you've been using cloaking, hidden text, doorway pages, keyword stuffing or link farms etc, expect to be penalised.

• You can't find your site - at all - in the search engine you suspect has penalised you. In the case of Google, search for "site:addedbytes.com" (replacing addedbytes.com with your domain name, of course). If no results are returned, Google will show you something like this:

  Your Search - site:addedbytes.com - did not match any documents.

  Suggestions:

  • Make sure all words are spelled correctly.
  • Try different keywords.
  • Try more general keywords.

If you have been penalised, then you'll get no sympathy from me - be more careful in future! SEO is not about getting top rankings for two weeks before vanishing forever from results, it's about a sustained and long-term effort to get top spots. It's not a sprint, it's a marathon. It's not worth taking the kind of risks that will get you penalised unless you have no choice. (</lecture>)

To come back from a penalty is not a quick process, but it is relatively simple. First, remove all remotely-fishy stuff from your site. Before a site is reincluded, the chances are it will be checked, and if you've not corrected what you were doing wrong, you will not be reincluded. Be over-cautious at this stage - better to remove absolutely anything that a search engine might dislike than remove the obvious things and be refused reinclusion because you've assumed that the search engine penalised you for something specific and that fixing that alone is enough. Once you are certain there is nothing left on your domain that can be considered dodgy by the whitest of white hats, then file a reinclusion request with the engine you are having trouble with.

Then wait - and it may be many months before you are reincluded, if at all. Don't pressure the engine and don't file the request every day or week. File it once and wait. You're the kid in the corner with the large hat with a D on it. THe search engine doesn't like you - you tried to manipulate it (even if it wasn't your work, it's your site and your responsibility). Have patience and work on building links to your site and building content - at least then when you are reincluded you should have better traffic.

What Next?

If you've not been penalised, then you should look at why you have dropped. Actually, let me rephrase - you should look at why your competitors have risen up the SERPs - that is a more accurate way to look at it. Check out the top sites in your field - what are they getting traffic for? What do their sites have that yours doesn't? Link quantity? Link quality? Content? Meta tags? A title in a specific shade of blue? Look for themes in the top ranking sites - if you can find out why they are top now, and you are not, you know what to work on.

The chances are that if you're not been penalised and your site is not performing as well, you need to look at improving or updating your online marketing tactics.

If your site is lacking in normal, organic links for example (you have previously paid for all of your links), then start adding things to your site people will want to link to to add to your normal organic links. Add a blog and post controversial or funny (but always unique) items on there. The web is a conversation, and you are not as prominent as you once were because the search engines are getting better - to get yourself noticed, you need to be talked about. [The same applies in all area - if the people doing better than you all have very content-heavy sites, hire some copywriters and get them writing some interesting and engaging content; if the people doing better than you have sites built with good quality, semantic markup, and you don't, have your site rebuilt.]

The most important thing is to treat a perceived drop in rankings for what it is: a temporary glitch in your grand plan. Put in a bit of hard work and a little investment in your online marketing and you should see improvements. You were ranking well before, so the chances are good that you will rank well again.

A robots.txt file (a document detailing the robots.txt exclusion standard is available) is always stored in the root of your site, and is always named in lower case. For example, if a website at http://www.addedbytes.com/ had a robots.txt file it would be found at http://www.addedbytes.com/robots.txt - and only there. Spiders will always search for it in the root of a domain, and will never ever look for it elsewhere. You cannot specify a different name or location for a robots.txt file.

A robots.txt file should be viewed like a list of recommendations. By including one, you are asking the spiders that visit your site to ignore certain things that you would prefer not to be indexed, but they are not obliged to pay attention to that. If you really do not want things indexed, it is far better to disallow access with server-side programming than a robots.txt file.

Writing a robots.txt File

A robots.txt file is a list of instructions. Each instruction is divided into two parts. The first, "User-agent" (case-sensitive), tells robots reading the file which robots should pay attention to the instructions that follow. Usually, this will be a "*", which is a wild card meaning "all robots". The wild card character can only be used in this context, except in the case of Googlebot, which does support it in other places (see User-Agent Specific Commands).

Following this line specifying a user agent are the rules themselves. The rules that apply to a defined user agent must be defined on the lines following the "User-agent" instruction. There can be no blank lines within each set of instructions, and there must be at least one blank line seperating sets of instructions. The instructions are usually of the format: "Disallow: /folder/" or "Disallow: /file.htm". There can only be one instruction per line, and you should really avoid putting spaces before the instructions (though this isn't specifically allowed or disallowed, it is probably best to avoid taking a risk).

Anything following a hash symbol "#" is considered a comment and ignored. At least, according to the standards. Rumours abound, though, that in the past some engines have ignored a line with a hash symbol on it wherever it is placed, so you may want to place each comment on a line by itself.

For example, the following robots.txt file is technically valid:

If you want to prevent robots from indexing anything at all on your site, you could add the following to your robots.txt file:

If you want to prevent all robots, except for a particular one or two, from accessing a folder, you could write a file like this, which will allow GoogleBot to index everything on your site, but prevent all other robots from accessing the folder called, imaginatively, "folder":

Please note: Many people believe that it is necessary to define the robot-specific rules before the general rules. This is not necessary according to the robots.txt exclusion standard, however there is no evidence of it causing problems, so may be worth doing, if there is a small chance it will help things to work as you intend.

Once you have written a robots.txt file, it is often a good idea to run it through a validator to check for errors, as they may do considerable harm if they prevent your site from being indexed. SearchEngineWorld's robots.txt validator is the most proficient of those available, or if you prefer, there is a validator that understands more unusual commands like Crawl-delay available as well.

Example Files

This is the robots.txt file for AddedBytes.com. As you can see, I have disallowed the indexing of a few files, but not many. Specifically, I have asked Google not to index "404.php", which is the page a user is redirected to if a page is not found, and "friend.php", which is linked to from every page, but is there to allow users to refer friends to the site, and so should not really be indexed.

This file, from eBay, is again quite short, and simply specifies a few folders that should not be indexed:

As you can see, Google will still list pages excluded by robots.txt, as Google is still aware they exist. However, Google will not index the content of the page and the page will not show up in searches except where a search includes the address of the excluded page.

Blank robots.txt files

It may be that you do not want to prevent spiders from indexing anything on your site. If that is the case, you should still add a robot.txt file, but an empty one, of this format:

This prevents spiders from generating a 404 error when the robots.txt file isn't found. It is basically just good practice to add a blank robots file, at the least, but not essential.

Be Careful

You may be thinking that adding the addresses of folders you do not with robots to index is a good way to prevent spiders from accidentally indexing sensitive areas of your site, like an administration area. While this is true, remember that anybody at all can view your robots.txt file, and therefore find the address(es) you'd rather were not indexed. If that includes your admninistration area, you may have saved them the trouble of searching for it.

There have been websites with unprotected administration areas online, whose admin area was hidden in an unusually named folder for "security" reasons - who added the name of the folder to their robots.txt file, opening up their admin area to anyone who wanted to have a poke around.

You must also be careful when writing your robots.txt file. Robots will usually err on the side of caution. If they do not recognise a command, they may well assume you meant them to stay away. Syntax errors in a robots.txt file can prevent your entire site from being indexed, so check it thoroughly before uploading it!

User-Agent Specific Commands

GoogleBot

Googlebot has no extra commands specific to it, however it is allegedly a little brighter than the average crawler. Googlebot will supposedly understand wild card characters (*) in the "Disallow" field of the robots.txt file. However, Googlebot is the only engine even rumoured to be able to do this, so you would be wise to avoid using wild cards in the disallow field wherever possible.

MSNBot and Slurp

The above code is specific to MSN's spider, "MSNBot", and Inktomi's spider, "Slurp", and instructs the spiders to wait the specified amount of time, in seconds (10 seconds above, default is 1 second if not specified) before requesting another page from your site. MSNBot and Slurp have been known to index some sites very heavily, and this allows webmasters to slow down their indexing speed.

You could technically use this command with a user agent of "*" as well - the robots.txt exclusion standard instructs robots to just ignore commands they do not understand. However, if a robot sees something they do not understand in a robots.txt file, they may just not index your site. If using the "Crawl-Delay" command, you would be wiser to specify the user agents it should apply to.

List of User-Agent Names

• Google: "googlebot"
• Google's Image Search: "Googlebot-Image"
• MSN: "msnbot"
• Inktomi: "Slurp"
• AllTheWeb: "fast"
• AskJeeves: "teomaagent1" or "directhit"
• Lycos: "lycos"

Why are directories useful?

Before we get to the criteria themselves, you may be wondering why it is important to be listed in directories at all, and rightly so. They usually do not provide huge amounts of direct traffic, and you are usually just adding one more site to a list of sometimes hundreds of competing ones. However, they still do provide a useful service.

Despite the lack of massive volumes of traffic, directories do provide a trickle. Most web sites wish for all the traffic they can handle, and while Google is top dog today, directing a vast amount of traffic around the web, who is to say what will happen tomorrow. If directories play a big part in the success of whichever search engine knocks Google off the top spot, you would do well to be listed in advance of that.

Web directories increase the number of in-bound links you have. Each listing will be another inbound link to your site, and many search engines now include link-popularity as a factor in the ordering of their results listings. Some directories are reproduced as well, like DMOZ, so one listing in DMOZ can easily turn into two or three hundreds in-bound links. No bad thing.

What should I do before submitting?

Ensure you have unique content, and a decent amount of it.

The purpose of most directories is not to list the same thing over and over again. For that reason, they insist upon unique content. This is especially relevant for affiliate sites, that often offer the same products as the site they are affiliated to, but with a different look. That would count as a site with the same content, and will not be listed. It is very important that you ensure that your site has a good amount of unique content before submitting to web directories.

The same applies to the quantity of content. A directory will have little reason to list a one-page site with a couple of photos of Dublin in Winter, without content. Before submitting, make sure that your site has enough content to be worth linking to, because if it doesn't, the directory editors just won't add it. A site under construction is as big a turn off, so make sure your site looks complete before submission.

Check your site is stable.

An editor from a directory may visit your site at any point, and may visit several times. A site that is unavailable on a regular basis will not be accepted to a directory, or may be removed if already listed. Make sure your site is available as much of the time as possible, especially when submitting. The same applies to the content of the site - make sure that there are as few broken links as possible, as a site with lots of broken links will almost certainly be rejected.

Check your site is accessible.

Although you may be using Internet Explorer 6 on Windows XP with a screen resolution of 1024 by 768, there is a good chance that the editor who reviews your site is not. The same applies to users as well - if an editor cannot view a site properly because he or she is using Opera, Mozilla or Safari, there is a good chance they will not list your site. Make sure your site works in as many browsers as possible, but at the very least it must work in recent versions of Internet Explorer, Mozilla and Opera, and should degrade well when viewed in older browsers. Ideally, it should also work with screen-readers and PDAs, as well as Mac and Linux based browsers (Safari and IE5 (Mac) and Konqueror and Galeon (Linux) should also be used for testing if possible).

Select your category (or categories) appropriately.

You may have reason to list your site in more than one category in a directory, however editors usually frown on this. Try to submit your site to the most appropriate category for it. If your site would fit in both a normal category and a "regional" category, then two listings may be allowed, but this is the exception to the rule, and does not apply to all directories. It is also tempting to try and submit your site to the category nearest the "top" of the tree, however this may result in your submission being denied if it is not the most appropriate place for it.

Prepare your submission details.

Usually before submission, it is a good idea to prepare a selection of text snippets to cut and paste into the relevant boxes during the submission process. Not only will this save you a little time, but writing your description and selecting your keywords at this stage will allow you some time to think about what you are writing. It is easy enough to open up a copy of notepad (or any text editor) and write down all of the text snippets you'll need while submitting - your name, email address, site name, site URL, description and keywords.

The description of the site is especially important, so do take your time writing it. Try to avoid sales hype in your text, as that may well affect your ability to be listed. Write your description in plain language, describing as accurately and succinctly as possible the contents of the site.

Read the submission guidelines for each engine carefully.

Each of the web directories has its own set of rules and guidelines. While this list covers the common points, there are often extra rules specific to each directory that you should be aware of before submitting. The list of directories at the bottom of this page also includes links to each directory's submission guidelines - you would be wise to check them before submitting your site.

What directories should I add to?

[url=http://www.dmoz.org/]DMOZ[/url]
DMOZ is the Open Directory Project, a massive, human-edited directory, and probably the most reproduced one of them all. It is the basis for the Google directory, among many others. It is free to submit to, and submissions may take anything up to six months (and sometimes even longer) to be processed. Patience is required when submitting to DMOZ. DMOZ is well worth submitting to, because apart from anything else, it is reproduced so much over the web that one listing can turn into 200 within a few months.

• [url=http://dmoz.org/add.html]DMOZ Submission Guidelines[/url]

[url=http://www.yahoo.com/]Yahoo[/url]
Yahoo is probably the most famous web directory you will find. There is an annual fee to pay if you wish to be listed on the Yahoo.com directory, but some of the national ones, eg Yahoo.co.uk, have a free listing available.

• [url=http://docs.yahoo.com/info/suggest/]Yahoo Submission Guidelines[/url]

[url=http://www.looksmart.com/]LookSmart[/url]
Looksmart operates on a pay-per-click basis, meaning that you pay for every single person who clicks on a link to your site from Looksmart. The price does vary depending on your chosen keywords.

• [url=http://listings.looksmart.com/help/guidelines/index.jhtml?loginState=4]LookSmart Submission Guidelines[/url]

[url=http://www.zeal.com/]Zeal[/url]
Zeal is a directory for non-commercial listings. It does feed the LookSmart directory as well as its own, and is used by many people as a way to list articles in LookSmart for free.

• [url=http://www.zeal.com/guidelines/style/]Zeal Submission Guidelines[/url]

[url=http://www.goguides.org]GoGuides[/url], [url=http://www.joeant.com]JoeAnt[/url], [url=http://www.skaffe.com]Skaffe[/url] and [url=http://www.web-beacon.com/]Web-Beacon[/url]
Originally, there was just Go.com. When that closed, two new directories sprang up in its place, created by ex editors of Go - GoGuides and JoeAnt. Division amongst the owners of GoGuides (it appears) has led to two other directories being set up with copies of the GoGuides database - Skaffe and Web-Beacon. All of these usually cost money to submit to, and they can be worth the expense, however that is not true for all sites, and you should consider submissions to these carefully before proceeding. Also, some of these four offer free listings from time to time, so you may wish to simply wait for one of those occasions, though there is no way to tell how long you may be waiting for.

• [url=http://www.goguides.org/DIR/editors_guidelines.php?c=1]GoGuides Submission Guidelines[/url]
• [url=http://www.joeant.com/guidelines.html]JoeAnt Submission Guidelines[/url]
• [url=http://www.skaffe.com/info/help_desk.html]Skaffe Submission Guidelines[/url]
• [url=http://www.web-beacon.com/info/help_desk.html]Web-Beacon Submission Guidelines[/url]

[url=http://www.gimpsy.com/]Gimpsy[/url]
Gimpsy offer both an express paid listing and a free listing option (currently the wait for free listing is at least four months). Gimpsy is unusual in that it lists sites according to "actions" rather than in a traditional category structure.

• [url=http://www.gimpsy.com/gimpsy/doc/faq/faq_suggest_site.php]Gimpsy Submission Guidelines[/url]

[url=http://www.bluefind.com/]Bluefind[/url]
BlueFind is a very new directory, but is growing quickly, and may be worth submitting to. There is a small fee for all sites to be processed, and usually submissions are dealt with quickly and efficiently.

• [url=http://www.bluefind.com/submit.php]Bluefind Submission Guidelines[/url]

Niche Directories
There are literally hundreds of other directories on the web that specialise, unlike the above, in one specific area. These are niche directories, and it is definitely worth the time and effort required to find these and submit your site to them. They will generate more targeted traffic to your site, though there may not be huge amounts of it, and can benefit your link popularity more than the larger directories, due to the consistency of the theming throughout the smaller directories.

And more - DMOZ has a [url=http://dmoz.org/Computers/Internet/Searching/Directories/]category devoted just to web directories[/url], with over 1000 listed. Not all of them are worth your while to submit to, but many will be. This category is especially useful for locating the niche directories specifically relevant to your web site.

Generally, at this stage, search engine results were pretty awful. Aside from the technology itself not being quite up to delivering accurate results, the search engines were also very susceptible to spam (spamming of search engines being a practice whereby people would try and cheat the algorythms and have their sites listed in irrelevant places to improve their traffic) and so it was difficult, at best, to actually find anything.

There were ways around these problems. Some sites used the mythical [url=http://www.addedbytes.com/SEO/META_Tags]META tags[/url] (tags used to describe a page's content) to describe each page of a site, and knew which engines to submit a site to, and how to go about it. They knew what the search engines liked to see on a page and what helped a page to rank well for a certain keyword or keyphrase. These people were, and still are, search engine optimizers, people who can turn a nice looking site into an actual income-generator.

SEO itself is about bringing in a decent quantity of quality traffic to a site. Many SEOs sell their services based upon volumes of visitors, and try to drag as many people as possible to a site, whether or not that person is actually interested. Some, usually the better ones, drag as many people as possible to a site, but qualify them first. There is little point in having (and paying for) large volumes of traffic if that traffic is made up of people who are not interested in your product or services.

SEOs also usually handle PPC (pay per click) campaigns. These are quite intimidating to the average web designer, but not for any good reason. Running a good PPC campaign is often a matter of planning and practice, and trying to ensure that the clicks you pay for a worth your while. Again, many SEOs will run a PPC campaign without any serious thought for the return on an investment made, while the successful ones will ensure that more is made from the visitors arriving than is spent on getting them there in the first place (one of the reasons so many dotcoms collapsed in the late 90s was the lack of thought that went into promotion and marketting - people were sometimes paying £50 per visitor to their site and making an average of £10 for each of them - not the best use of capital).

Optimizing your sites for the search engines is, with all this in mind, basically a five stage process: Preparation, Modification, Submission, Campaigning and Maintenance.

1. Preparation

If you dive into SEO without a thought of planning, you are unlikely to do well. Things to think of though, before you get started, are who your target audience are and how to reach them. If you sell industrial vibrators, for example, you would do very badly if you decided to target the keyword "vibrators" by itself. Try to put yourself in the shoes of your user. Think what age group they are in, and why your product might appeal to them. Planning well can help you to avoid spending your time and money chasing users who are no use to you, even if they do visit your site.

2. Modification

Once you are happy you have your target audience in mind and have planned your SEO work, then it is time to look at your site. It is important to ensure that a search engine spider can index your site, and that you have included appropriate META tags. It is also essential to ensure that your page titles are well formed and descriptive.

If you have frames, I would suggest removing them. Remember that a visitor can land on any page of your website, completely at random, and if they don't see navigation, they may well just leave. In the same vein, try to imagine how a user would react to landing on each page of your site and think about things that might help that user progress from there to the areas of your site where they can bring in some income for you.

3. Submission

Once your site is up to scratch, you can start the submission process. Ignore anyone who tells you they will submit to anything above about 50 search engines, or that they can guarantee a high placement in any engine - SEO just doesn't work like that. Instead, start by building up links to your site from complimentary resources, and getting listed in the major directories. Once that is done, the spiders will find your site. Then, just keep on building in-bound links and improving your content. The rest will just start to fall into place.

4. Campaigning

If you offering a product, it might be worth advertising on a pay-per-click basis. There are tonnes of PPC companies around, and plenty of choice out there, but it is worth remembering that cheap isn't necessarily a good thing. However cheap a PPC campaign is, it is useless if you end up without a profit, so when working on PPC, be sure that, above all else, you are working with a reputable company, that can turn you a profit. It might even be worth hiring someone to start your PPC for you, and you never know - they might even be willing you train you to manage it yourself later.

5. Maintenance

Once you've attained a high ranking, it is very easy to sit back and relax. Much like planning though, maintenance is an important part of SEO. You must keep building up in-bound links, and content on a site. Apart from keeping it fresh for the visitors, this will keep the search engines indexing your site, and every article or page on your site is a potential landing point, and should be able to bring in a little extra traffic, which all mounts over time.

If you have a good business head about you, try and keep an eye on the major search engines and work out what they want to show their visitors. If the most popular engine of the day is showing articles above all else, try and add more articles to your site. If that engine is showing sites without META tags, remove yours. Keeping an eye on the search engines, and thinking of them as businesses, may be one thing that helps you to stay ahead of your competition, which is, after all, the aim of the game.

And finally...

Do remember that there is more to a web site than traffic. You may be thinking that SEO is the only way you can make more from your site - that numbers equal profit. That is sometimes true, but look at other aspects of your site before jumping in the deep end - can you improve your conversion ratio (the number of people who buy products compared to the number of visitors), or can you make your site more usable? Are you targetting the right audience, or will offline advertising help? Is traditional PR important? Are your potential clients or customers online at all?

Last, but definitely not least - have patience. Improving traffic to a site takes months, and a lot of hard work, and does not always have the effect you desire. It is not a quick or easy process, and should not be undertaken lightly. Most people do not have the time or patience to do their own SEO, and of those that do, many fail because of bad planning, or competing for heavily competitive keywords.

That said, try not to be too put off - doing your own promotion can, if it works, be very rewarding, and not just financially.

META tags are a way to describe a page in HTML, invisibly to the user. Many search engines either do not use them at all, or give them little weight. However, they still have their uses and can provide a boost to your search engine placement.

The trick to using them well is to understand what they do, and providing the best possible information within them. It is important to realise as well that changing or adding META tags will not turn your website into a gold mine overnight, but as part of a well formed SEO strategy, they can certainly help.

There are many people who say you should only ever add two or three META tags to your site. There are those who say you should add hundreds. The simple fact is that there are many that could be appropriate to your site, and you should judge each of them on its individual merit.

META tags all go within the HEAD section of your site. That is to say, within the <head> and </head> tags.

Index

• The Title Tag
  Absolutely invaluable.
• The Description Tag
  Useful if used correctly
• The Keywords Tag
  Becoming less useful, but may be worth adding.
• The Robots Tag
  Can be useful in the right circumstances.
• The Content-Type, Content-Style-Type and Content-Language Tags
  Can be useful in the right circumstances.
• The Refresh Tag
  Can be useful in the right circumstances.
• The Pragma Tag
  Can be useful in the right circumstances.
• The Revisit-After and Expires Tags
  Usually useless, unless you actually want to put the spiders off for a certain period of time.
• Useless Tags
  Useless all the time, unsurprisingly.
• ICRA Label
  Useful

The Title Tag

The TITLE tag is NOT a META tag. But it does contain metadata, and it is the most important tag on a page and is closely related to them, so I am including it here.

Title tags are displayed in the top of a browser window, and are often used as a link from search engine results listings, so form them well. They should be descriptive and short (ideally under 70 characters), and they are also often used as bookmark titles, so it is important that you ensure your primary keyword phrase for a page is here, and that the title makes sense all by itself.

The Description Tag

This is one of the few META tags that can be considered important. The text within this is displayed by some search engines as the description to your site. A description tag should usually be kept to under around 150 - 200 characters and it is important to ensure that this tag reads well, and that it describes the page accurately.

There is no point in telling the user that your page contains thousands of pictures of Alicia Silverstone in lacy underwear if when they arrive on the page they see nothing but a sales pitch for tinned goulash. An extreme example, perhaps, but does demonstrate the point that it is better to have visitors who are interested in your product or content than those who aren't. Numbers are unimportant if they don't convert to sales, and this will help to qualify your visitors before they arrive.

The Keywords Tag

Fairly self explanatory, this tag is used to list keywords for your page. These are words you think are relevant to your page - words that if entered into a search engine should return your site. Search engines do not pay much attention to this, if any, as it has been abused for many years, but some do still use it to some small extent, so you may consider it worth adding.

Try to limit yourself to as few keywords as possible (the less keywords you list, the more weight each will likely have), certainly no more than 25, and list them with nothing more than spaces between (some people use commas, however this is no longer necessary). There is also no need to repeat the words listed.

As has been widely reported on the web, this tag is not used by many engines, if at all, and you would be wise to spend your time optimising and improving your site in other ways rather than waste time on this particular tag, in my humble opinion.

The Robots Tag

The ROBOTS META tag is one that is very often used when it should not be. The four variations listed above are four of the more common variations in use, and each accomplishes a different task. Never use this tag unless you wish to prevent a search engine spider from doing something. That's what it's there for.

The first of the examples listed above is completely worthless. If you have it on your site, go and delete it. That tag does nothing more than tell a search engine spider to behave exactly as it normally does. It does not benefit a site, does not get you crawled faster or more often, and will not suddenly make your site more popular than Google.

The second of these can be useful, for example on printer-friendly pages (where the content on the page is a duplicate of the original). This tag tells a search engine spider not to list the page it is viewing, but to follow the links away from the page anyway. The third of these is the reverse of the above, and tells a spider to list a page in it's results but not to follow the links on the page. Both of these have their uses, but these are very rare, so think carefully before adding these before you do.

The last tag tells a spider not to index a page or follow the links on it. It is extremely rare that you would want to use this (why would anyone want a page on the web that people cannot find?) but is included for the sake of completeness (some people use this for login pages or other similar pages they do not want listed).

There are more instructions you can add to this tag, the most notable of which is NOARCHIVE. This simply tells a search engine spider not to serve archived copies of the page to people viewing the search engine results (for example, Google offer a cached copy of sites in search results, and this will prevent Google from doing so). The tag to add to only prevent search engines making archived copies of your site publically available is:

The Content-Type, Content-Style-Type and Content-Language Tags

These are again quite common on some sites, and again have their uses. It is a wise idea if you are using an unusual language or style to mention it here, but by no means essential, as with most META tags. The W3C provide a more comprehensive resource for [url=http://www.w3.org/International/O-charset.html]character set information[/url], so if you do wish to use this, I recommend that as a good place to start reading.

Meta tags that use the "http-equiv" attribute rather than the "name" attribute, like these, allow you define within a document something that would usually be defined in HTTP headers (sent by your server). If you have no control over the headers sent with your web pages, but still need to define a content type or content style type (and so on), these are the tags you are looking for.

The Refresh Tag

Most useful on a chat page, or when a page has moved, this instructs a browser to refresh the page after a certain interval of seconds. If the second half of the content attribute is a URL, the refresh will take the user to the URL specified rather than simply refresh the current page. This can be, and sometimes is, used mischievously to prevent a user from clicking their back button to leave a page, something likely to annoy visitors enough that they may never return.

The Pragma Tag

Not very widely used, this tag asks a browser not to cache a page. Though this can be useful if a page on your site is frequently updated (for example a news site or a forum), it will often just increase your bandwidth bills and slow down your users' browsing experience. There is also no guarantee that a browser will pay attention to it.

Interestingly enough though, Microsoft recommend that if you do want to use this, you add the tag in a second HEAD at the end of the document, like so:

The Revisit-After and Expires Tags

There are a huge number of sites that say you should add the first of these to your site, because it tells search engine spiders how often to index your page. Which is a common misconception. The tag was created by SearchBC, who have said they no longer use it. Originally, it was created as a tool to suggest to the spider how often a page should be indexed. Few have ever been able to agree on the format of the tag. At the end of the day, remember that the search engines do not care how often you want them to index your pages - they will index as and when they feel like it. Some are clever enough to have a rough idea of how often you update your site, and will make use of that. Some are not that bright, and will come around when the mood takes them.

Assuming you are happy for the spiders to index your site as often as possible, as most people are, you would do well to leave this out. The spiders will return to your site as often as they deem fit, and the only way to influence the frequency this occurs at is to just keep adding new content on a regular basis.

The "Expires" tag tells browsers and search engine spiders when the document should be considered expired. This is worth using, of course, if there is a date on which the relevant document will be no longer valid. However, at this time, the search engines will often drop the page from their index - you should use the "Expires" tag only if this is what you want.

Useless Tags

A select few engines sometimes make small use of a select few of these, but most of these (and the others to be found on this [url=http://www.bauser.com/websnob/meta/useless.html]list of useless META tags[/url]) are better placed on a page, or not used at all. Most of these are added automatically by HTML editors, and some are added by over-zealous META tag addicts. In my opinion, these are best avoided, as they do little more than clutter up your code.

ICRA Label

Last but not least, something a little more unusual. The ICRA (Internet Content Rating Association) is an ideal I am happy to support, as they provide a means for helping webmasters to identify their content as suitable (or not) for certain age groups.

Simply put, you can visit their [url=http://www.icra.org/_en/label/extended/]label generator[/url] and tell the generator what your site contains. That data can then be used to help keep any content not appropriate for young eyes away from them. The data is used by some search engines and some browsers can be set to avoid pages without labels.