A robots.txt file (a document detailing the robots.txt exclusion standard is available) is always stored in the root of your site, and is always named in lower case. For example, if a website at http://www.addedbytes.com/ had a robots.txt file it would be found at http://www.addedbytes.com/robots.txt - and only there. Spiders will always search for it in the root of a domain, and will never ever look for it elsewhere. You cannot specify a different name or location for a robots.txt file.

A robots.txt file should be viewed like a list of recommendations. By including one, you are asking the spiders that visit your site to ignore certain things that you would prefer not to be indexed, but they are not obliged to pay attention to that. If you really do not want things indexed, it is far better to disallow access with server-side programming than a robots.txt file.

Writing a robots.txt File

A robots.txt file is a list of instructions. Each instruction is divided into two parts. The first, "User-agent" (case-sensitive), tells robots reading the file which robots should pay attention to the instructions that follow. Usually, this will be a "*", which is a wild card meaning "all robots". The wild card character can only be used in this context, except in the case of Googlebot, which does support it in other places (see User-Agent Specific Commands).

Following this line specifying a user agent are the rules themselves. The rules that apply to a defined user agent must be defined on the lines following the "User-agent" instruction. There can be no blank lines within each set of instructions, and there must be at least one blank line seperating sets of instructions. The instructions are usually of the format: "Disallow: /folder/" or "Disallow: /file.htm". There can only be one instruction per line, and you should really avoid putting spaces before the instructions (though this isn't specifically allowed or disallowed, it is probably best to avoid taking a risk).

Anything following a hash symbol "#" is considered a comment and ignored. At least, according to the standards. Rumours abound, though, that in the past some engines have ignored a line with a hash symbol on it wherever it is placed, so you may want to place each comment on a line by itself.

For example, the following robots.txt file is technically valid:

If you want to prevent robots from indexing anything at all on your site, you could add the following to your robots.txt file:

If you want to prevent all robots, except for a particular one or two, from accessing a folder, you could write a file like this, which will allow GoogleBot to index everything on your site, but prevent all other robots from accessing the folder called, imaginatively, "folder":

Please note: Many people believe that it is necessary to define the robot-specific rules before the general rules. This is not necessary according to the robots.txt exclusion standard, however there is no evidence of it causing problems, so may be worth doing, if there is a small chance it will help things to work as you intend.

Once you have written a robots.txt file, it is often a good idea to run it through a validator to check for errors, as they may do considerable harm if they prevent your site from being indexed. SearchEngineWorld's robots.txt validator is the most proficient of those available, or if you prefer, there is a validator that understands more unusual commands like Crawl-delay available as well.

Example Files

This is the robots.txt file for AddedBytes.com. As you can see, I have disallowed the indexing of a few files, but not many. Specifically, I have asked Google not to index "404.php", which is the page a user is redirected to if a page is not found, and "friend.php", which is linked to from every page, but is there to allow users to refer friends to the site, and so should not really be indexed.

This file, from eBay, is again quite short, and simply specifies a few folders that should not be indexed:

As you can see, Google will still list pages excluded by robots.txt, as Google is still aware they exist. However, Google will not index the content of the page and the page will not show up in searches except where a search includes the address of the excluded page.

Blank robots.txt files

It may be that you do not want to prevent spiders from indexing anything on your site. If that is the case, you should still add a robot.txt file, but an empty one, of this format:

This prevents spiders from generating a 404 error when the robots.txt file isn't found. It is basically just good practice to add a blank robots file, at the least, but not essential.

Be Careful

You may be thinking that adding the addresses of folders you do not with robots to index is a good way to prevent spiders from accidentally indexing sensitive areas of your site, like an administration area. While this is true, remember that anybody at all can view your robots.txt file, and therefore find the address(es) you'd rather were not indexed. If that includes your admninistration area, you may have saved them the trouble of searching for it.

There have been websites with unprotected administration areas online, whose admin area was hidden in an unusually named folder for "security" reasons - who added the name of the folder to their robots.txt file, opening up their admin area to anyone who wanted to have a poke around.

You must also be careful when writing your robots.txt file. Robots will usually err on the side of caution. If they do not recognise a command, they may well assume you meant them to stay away. Syntax errors in a robots.txt file can prevent your entire site from being indexed, so check it thoroughly before uploading it!

User-Agent Specific Commands

GoogleBot

Googlebot has no extra commands specific to it, however it is allegedly a little brighter than the average crawler. Googlebot will supposedly understand wild card characters (*) in the "Disallow" field of the robots.txt file. However, Googlebot is the only engine even rumoured to be able to do this, so you would be wise to avoid using wild cards in the disallow field wherever possible.

MSNBot and Slurp

The above code is specific to MSN's spider, "MSNBot", and Inktomi's spider, "Slurp", and instructs the spiders to wait the specified amount of time, in seconds (10 seconds above, default is 1 second if not specified) before requesting another page from your site. MSNBot and Slurp have been known to index some sites very heavily, and this allows webmasters to slow down their indexing speed.

You could technically use this command with a user agent of "*" as well - the robots.txt exclusion standard instructs robots to just ignore commands they do not understand. However, if a robot sees something they do not understand in a robots.txt file, they may just not index your site. If using the "Crawl-Delay" command, you would be wiser to specify the user agents it should apply to.

List of User-Agent Names

• Google: "googlebot"
• Google's Image Search: "Googlebot-Image"
• MSN: "msnbot"
• Inktomi: "Slurp"
• AllTheWeb: "fast"
• AskJeeves: "teomaagent1" or "directhit"
• Lycos: "lycos"

Any HTML element can be considered a box, and so the box model applies to all HTML (and XHTML) elements.

The box model is the specification that defines how a box and its attributes relate to each other. In its simplest form, the box model tells browsers that a box defined as having width 100 pixels and height 50 pixels should be drawn 100 pixels wide and 50 pixels tall.

There is more you can add to a box, though, like padding, margins, borders, etc. This image should help explain what I'm about to run through:

As you can see, a box is made up of four distinct parts. The outside one, the margin, is completely invisible. It has no background color, and will not obstruct elements behind it. The margin is outside the second part, which is the border. The border outlines the visible portion of the element. Inside the border is the third part of the box, the padding, and then inside that the content area of the box. The padding defines the space between the content area of the box and the border.

(Note that in the image above, the only border of the three drawn that would actually be visible is the solid line - the dashed lines are added to help demonstrate the box model).

When you define a width and a height for your box using CSS, you are defining not the entire area taken up by the content, padding, border and margin. You are actually just defining the content area itself - the bit right in the middle. The padding, border and margin must be added to that in order to calculate the total space occupied by the box. (From this point on, we will use width for demonstrations, but the same principles apply to both width and height).

The above CSS, applied to a box, would mean that that box occupied 300 pixels of space horizontally on the page. The content of the box would occupy 200 pixels of that (dashed line added to demonstrate the edge of the area occupied by the box):

In the above image, you can see that the pale blue area is 240 pixels wide (200 pixels of content plus 20 pixels padding either side). The border is 10 pixels wide either side, making the total width including the border 260 pixels. The margin is 20 pixels either side, making the total width of the box 300 pixels.

In practice, this can cause some confusion. For example, if I have a 100 pixel wide space available, and want to fill it with a pale red box with a dark red border and a small amount of padding, it would be very easy to write the CSS like so:

(A declaration of 0, as used for the margin above, does not require a unit to be added. Any value other than 0 does require a unit, e.g. px for pixels. Also, although "background" is defined as a shorthand property, it is more widely supported than the more correct "background-color".)

However, that will not give us a 100 pixel wide box, as the width declaration defines the content area of the box. The content area of the box will be 100 pixels - the total width of the box as defined above will be 122 pixels:

In order to set the above box to only occupy 100 pixels horizontally, you would need to set the width of the content area to be 100 pixels minus the padding and minus the border, in this case 78 pixels, like so:

To calculate the overall width of a box, including all padding, borders and margins, you would use the following formula:

Compatibility

At this point, you should now have a good understanding of what the box model is, and how boxes should be treated by different browsers. However, as you will soon learn (if you did not know already), not every browser does as it is supposed to. In order to use boxes, and by extension make the most of CSS in your website, you will need to be aware of how the different browsers treat boxes in practice and how to overcome and work around the problems presented by these idiosyncrasies.

Top Notch

Most browsers released in the last few years have no problem with boxes and render boxes correctly. Opera 6 and 7, Mozilla 1 (and by extension other browsers based on the Gecko engine like Netscape 7, Camino and Firefox and other derivatives), Safari, Konquerer (and derivatives) and Internet Explorer 5 for the Mac are all shining examples of how a web browser should behave, all rendering boxes flawlessly. IE 6 for Windows also will render a box correctly, as long as the [url=http://www.addedbytes.com/design/dtds-explained]Document Type Definition[/url] for the page is correct.

Whoops, Mrs Miggins, You're Sitting On My Artichokes

Some browsers don't display a box correctly. Unlike those below here, these browsers are widely enough used on the web that it is usually worth the effort to work through the problem. There are various methods for doing this, some better than others, that follow on. Most notable among the browsers with problems are Internet Explorers 4 and 5 and Internet Explorer 6. IE 6 is easy to work around, by adding a correct DTD (which you should be doing anyway).

Internet Explorer 5 is the main reason there is a box model problem at all. It, unfortunately, does not follow the simple definition for box layout as defined by the W3c. When you define a width for a box and it is rendered in IE5, instead of that width defining the content area of the box, it includes the borders and padding. Margins are added on to the content width correctly, but padding and borders are not. Unfortunately, this leaves us with some unpleasant choices:

1. Use a box model hack
   Hack's like [url=http://www.tantek.com/CSS/Examples/boxmodelhack.html]Tantek's box model hack[/url] are unfortunately something of a necessity. While some might argue that using hacks like this is completely missing the point of using CSS for web design, commercial necessity and the prevalence of IE5 leave us with little in the way of choice. The IE5 box model hack is in use all over the web and has spawned plenty of variants.
2. Add in extra code
   Some might consider this a slightly "better" way of working around this problem. Rather than adding a style sheet hack, you can nest elements within each other. Adding a div within another div means that rather than using padding, you can use just margins, which are handled correctly by IE5. As with the box model hack, it is far from a perfect solution, but there are few other options if you want a site to look the same in IE5 as other more capable browsers.

Hall of Shame

On the one hand, the browsers that I am about to mention are appalling, all failing dismally to render a simple box correctly for one reason or another. On a more positive note, users of these browsers, mostly old versions of current browsers, make up an extremely small, and continually shrinking, portion of web users. While you could probably find a workaround for the bugs in the display of boxes in these browsers, it is almost certainly not worth the effort - you are likely to cause yourself more harm than good with workarounds for these!

Netscape 4's box model is awful, but even worse, the simple box model hacks to fix the problem for IE5 and IE6 will crash Netscape 4. Netscape 4's style sheet support is abysmal overall, and it is being supported less and less. Though it is strictly a personal choice, I don't think it is worth the time and effort to support Netscape 4 any more - it's just not used enough, and the number of users is only ever going to shrink.

Internet Explorer 4 suffers, basically, the same problem as IE5. It treats boxes in a very similar way. However, it falls over in far more ways, and many of the available hacks will crash IE4. As it is also used by few people, and that number is dropping, many designers ignore it.

What does the future hold?

CSS3 promises us the option to determine how we want the user agent to treat boxes, and specify which box model we want to use. Support for CSS3 at a level that will be possible is many many years away yet. Until then, we are stuck with the CSS2 box model, and while IE5 is still used by a significant percentage of the web's population, we are going to have problems with boxes.

A more technical breakdown of HTTP 1.1 status codes and their meanings is available at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. There are several versions of HTTP, but currently HTTP 1.1 is the most widely used.

Informational

• 100 - Continue
  A status code of 100 indicates that (usually the first) part of a request has been received without any problems, and that the rest of the request should now be sent.
• 101 - Switching Protocols
  HTTP 1.1 is just one type of protocol for transferring data on the web, and a status code of 101 indicates that the server is changing to the protocol it defines in the "Upgrade" header it returns to the client. For example, when requesting a page, a browser might receive a statis code of 101, followed by an "Upgrade" header showing that the server is changing to a different version of HTTP.

Successful

• 200 - OK
  The 200 status code is by far the most common returned. It means, simply, that the request was received and understood and is being processed.
• 201 - Created
  A 201 status code indicates that a request was successful and as a result, a resource has been created (for example a new page).
• 202 - Accepted
  The status code 202 indicates that server has received and understood the request, and that it has been accepted for processing, although it may not be processed immediately.
• 203 - Non-Authoritative Information
  A 203 status code means that the request was received and understood, and that information sent back about the response is from a third party, rather than the original server. This is virtually identical in meaning to a 200 status code.
• 204 - No Content
  The 204 status code means that the request was received and understood, but that there is no need to send any data back.
• 205 - Reset Content
  The 205 status code is a request from the server to the client to reset the document from which the original request was sent. For example, if a user fills out a form, and submits it, a status code of 205 means the server is asking the browser to clear the form.
• 206 - Partial Content
  A status code of 206 is a response to a request for part of a document. This is used by advanced caching tools, when a user agent requests only a small part of a page, and just that section is returned.

Redirection

• 300 - Multiple Choices
  The 300 status code indicates that a resource has moved. The response will also include a list of locations from which the user agent can select the most appropriate.
• 301 - Moved Permanently
  A status code of 301 tells a client that the resource they asked for has permanently moved to a new location. The response should also include this location. It tells the client to use the new URL the next time it wants to fetch the same resource.
• 302 - Found
  A status code of 302 tells a client that the resource they asked for has temporarily moved to a new location. The response should also include this location. It tells the client that it should carry on using the same URL to access this resource.
• 303 - See Other
  A 303 status code indicates that the response to the request can be found at the specified URL, and should be retrieved from there. It does not mean that something has moved - it is simply specifying the address at which the response to the request can be found.
• 304 - Not Modified
  The 304 status code is sent in response to a request (for a document) that asked for the document only if it was newer than the one the client already had. Normally, when a document is cached, the date it was cached is stored. The next time the document is viewed, the client asks the server if the document has changed. If not, the client just reloads the document from the cache.
• 305 - Use Proxy
  A 305 status code tells the client that the requested resource has to be reached through a proxy, which will be specified in the response.
• 307 - Temporary Redirect
  307 is the status code that is sent when a document is temporarily available at a different URL, which is also returned. There is very little difference between a 302 status code and a 307 status code. 307 was created as another, less ambiguous, version of the 302 status code.

Client Error

• 400 - Bad Request
  A status code of 400 indicates that the server did not understand the request due to bad syntax.
• 401 - Unauthorized
  A 401 status code indicates that before a resource can be accessed, the client must be authorised by the server.
• 402 - Payment Required
  The 402 status code is not currently in use, being listed as "reserved for future use".
• 403 - Forbidden
  A 403 status code indicates that the client cannot access the requested resource. That might mean that the wrong username and password were sent in the request, or that the permissions on the server do not allow what was being asked.
• 404 - Not Found
  The best known of them all, the 404 status code indicates that the requested resource was not found at the URL given, and the server has no idea how long for.
• 405 - Method Not Allowed
  A 405 status code is returned when the client has tried to use a request method that the server does not allow. Request methods that are allowed should be sent with the response (common request methods are POST and GET).
• 406 - Not Acceptable
  The 406 status code means that, although the server understood and processed the request, the response is of a form the client cannot understand. A client sends, as part of a request, headers indicating what types of data it can use, and a 406 error is returned when the response is of a type not i that list.
• 407 - Proxy Authentication Required
  The 407 status code is very similar to the 401 status code, and means that the client must be authorised by the proxy before the request can proceed.
• 408 - Request Timeout
  A 408 status code means that the client did not produce a request quickly enough. A server is set to only wait a certain amount of time for responses from clients, and a 408 status code indicates that time has passed.
• 409 - Conflict
  A 409 status code indicates that the server was unable to complete the request, often because a file would need to be editted, created or deleted, and that file cannot be editted, created or deleted.
• 410 - Gone
  A 410 status code is the 404's lesser known cousin. It indicates that a resource has permanently gone (a 404 status code gives no indication if a resource has gine permanently or temporarily), and no new address is known for it.
• 411 - Length Required
  The 411 status code occurs when a server refuses to process a request because a content length was not specified.
• 412 - Precondition Failed
  A 412 status code indicates that one of the conditions the request was made under has failed.
• 413 - Request Entity Too Large
  The 413 status code indicates that the request was larger than the server is able to handle, either due to physical constraints or to settings. Usually, this occurs when a file is sent using the POST method from a form, and the file is larger than the maximum size allowed in the server settings.
• 414 - Request-URI Too Long
  The 414 status code indicates the the URL requested by the client was longer than it can process.
• 415 - Unsupported Media Type
  A 415 status code is returned by a server to indicate that part of the request was in an unsupported format.
• 416 - Requested Range Not Satisfiable
  A 416 status code indicates that the server was unable to fulfill the request. This may be, for example, because the client asked for the 800th-900th bytes of a document, but the document was only 200 bytes long.
• 417 - Expectation Failed
  The 417 status code means that the server was unable to properly complete the request. One of the headers sent to the server, the "Expect" header, indicated an expectation the server could not meet.

Server Error

• 500 - Internal Server Error
  A 500 status code (all too often seen by Perl programmers) indicates that the server encountered something it didn't expect and was unable to complete the request.
• 501 - Not Implemented
  The 501 status code indicates that the server does not support all that is needed for the request to be completed.
• 502 - Bad Gateway
  A 502 status code indicates that a server, while acting as a proxy, received a response from a server further upstream that it judged invalid.
• 503 - Service Unavailable
  A 503 status code is most often seen on extremely busy servers, and it indicates that the server was unable to complete the request due to a server overload.
• 504 - Gateway Timeout
  A 504 status code is returned when a server acting as a proxy has waited too long for a response from a server further upstream.
• 505 - HTTP Version Not Supported
  A 505 status code is returned when the HTTP version indicated in the request is no supported. The response should indicate which HTTP versions are supported.

You will doubtless be aware of the basic structure of a web page, if you are a designer. There are two main sections, within a document (you might not know it, but the <html> section can come after the <body> section, if you want):

The above will probably be very familiar to you. Hopefully, you will be aware of the hundreds of other tags available to you as well. But did you know that the following can be both correct and incorrect?

In XHTML, which is case sensitive, the above is gibberish - a tag that has no meaning. In HTML, it's a centered paragraph.

How does a browser know which it should pay attention to? Should it assume that you are writing HTML, and render the above as a centered paragraph? If you were writing XHTML, then no, it shouldn't, because a mistake has been made. It should do nothing. But how does the browser know what language you are writing and how to display it?

This is where the DTD comes in handy. A DTD is a tag that is placed within a page before anything else (including white space). It must be at the very front of the very first line of every page you write. Once there, and correctly formed, your DTD can then tell the browser what language you are writing. If you have written your code correctly, and you have a proper DTD in place, the browser will then render your page according to the standards laid down by the W3C, in most cases.

If you do not include a DTD, browsers will use "Quirks Mode" to render a page. Designed to accomodate poor coding and old hacks, "Quirks Mode" gives you little control and if your page is rendered in "Quirks Mode", it will look different in every single browser. All the hard work you've put in to styling your page and creating a beautiful new site will seem to have been somewhat wasted when you realise that many people cannot see the page properly because it has rendered badly.

So a DTD is essential to any page. It is used by validators to determine what language you are writing, so it can check your code (and if you're a web designer who doesn't validate your code, you should change your ways ... or consider a new career). It is used by browsers to render pages. And if you know what language you are writing, you can use the correct tags and markup for each part of your page, to help it become semantically correct.

DTDs are formed of two parts, and look a bit like this:

The above is a DTD for XHTML 1.1. Some languages have subtle variations (for example, HTML can be Strict (for well-written pages), Transitional (for pages with deprecated tags or not-quite-perfect code), or Frameset (for pages with Frames)), and some, like XHTML 1.1 do not. Each of these variations will have it's very own DTD as well. Some require upper case "HTML" and some lower case. Each of them can be found on the W3C site, and a short list of the common ones is included on the following page.

Adding a DTD

If you are creating a web page, your DTD should reflect the language you intend to use for that page. Add it before anything else - let it be the very first thing you write (before even empty lines), like this:

It is difficult adding a DTD to a page after it has been written, especially if you are using CSS for the look and layout, as when a browser comes out of "Quirks Mode" and renders a page correctly, much of your positioning, as well as padding and margins, may look wrong.

If you are adding a DTD to a page that has already been written, you are going to have some problems. That's a fact. That doesn't make it any less worthwhile adding one. Apart from anything else, you can be very sure you'll remember to add it first on the next site you write. You may want to go for a "Transitional" DTD, which allows you some flexibility (Transitional DTDs allow deprecated tags, like "<b>", and are relatively forgiving, unlike Strict ones).

If you haven't validated a page before (and if you don't have a DTD, chances are you haven't been able to validate your code), you'll probably be looking at something like the HTML 4.0 Transitional DTD, below. Add that to the top of a page, and validate with any one of the many web page validation tools available. Fix the bugs you find, then look at your page in a browser. If things are wrong, it will be down to code - CSS or HTML - and should not take too long to correct.

Once each of your pages has a proper DTD, you can rest easy, knowing that programs fetching and displaying your pages know which language they are written in and which set of rules they conform to. If your page validates, you know the code you have written is pretty close to accurate, although of course a validator cannot check the semantics of a document. Last, you know that, as the browsers now know what language to display, there is a good chance that everyone visiting your site is seeing more or less the same thing.

Common DTDs

The following is a list of those DTDs used often on the web. The chances are that, if you do not currently have a DTD on your pages, it should be one of the below.

• XHTML 1.1
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
• XHTML 1.0 Strict
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
• XHTML 1.0 Transitional
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
• XHTML 1.0 Frameset
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-frameset.dtd">
• HTML 4.01 Strict
  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
• HTML 4.01 Transitional
  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
• HTML 4.01 Frameset
  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN" "http://www.w3.org/TR/html4/frameset.dtd">
• HTML 3.2
  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
• HTML 2.0
  <!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">

Using Accesskeys

Note that in the case of form elements, the accesskeys will always move focus rather than activate the element.

• Internet Explorer
  Press ALT-X, where X is the Accesskey letter. For example, ALT-1 on this site will [i]shift focus[/i] to the link to the homepage.
• Mozilla, Netscape and derivatives
  Press ALT-X, where X is the Accesskey letter. For example, ALT-1 on this site will [i]activate[/i] to the link to the homepage.
• Opera
  Press SHIFT-ESC then the Accesskey, and this will [i]activate[/i] the link.

Why use Accesskeys?

Accesskeys are commonly listed as an accessibility item by many people, but more and more web users are discovering and using accesskeys, as they make navigation around the sites you use the most a little quicker and easier. They should be used, where possible for the simple reason they make a site easier to use for a wider range of people. If you aren't convinced, try not using your mouse for a day and seeing how easy you find using the web. With accesskeys you will find it much easier.

At the very least, they should be used for the major links within your site, such as the search box and home button. Though there is no standard set yet for accesskeys, the following is a list of the common numbers used for a few of the more common links:

• 1 - Home
• 2 - Skip Navigation
• 4 - Search Input
• 9 - Contact / Feedback
• 0 - Accessibility Statement (if there is one)

Adding Accesskeys To Your Site

There is a short list of tags that support the "accesskey" attribute: <a>, <area>, <button>, <input>, <label>, <legend>, and <textarea>. It is added simply as a normal attribute, for example:

It is wise to also a visual indication of which letter is to be used as an accesskey on any one link, for example by underlining that letter within the link. Adding it to the title doesn't hurt either, for example:

It is also wise to avoid picking accesskeys that conflict with special keys already in use in an application. Internet Explorer and Mozilla both use ALT then a letter for accesskeys and this can often create a conflict. It is wise to avoid all of the following letters, as these are all already in use within common browsers: [i]a, b, d, e, f, g, h, t, v, w[/i].

Try and keep your accesskeys consistent, too. If a user spots an accesskey on one page, they may not check on the next page to see if they can still use it on this page. For that reason, it is very important that, which ever accesskeys you do use, you use the same accesskeys on every single page, without fail.

Last, do not be afraid to advertise the fact you are using accesskeys. People will want to know about it, and will use them if they know they are there, so add a list of the ones you are using on your site to your help pages or your accessibility statement!

META tags are a way to describe a page in HTML, invisibly to the user. Many search engines either do not use them at all, or give them little weight. However, they still have their uses and can provide a boost to your search engine placement.

The trick to using them well is to understand what they do, and providing the best possible information within them. It is important to realise as well that changing or adding META tags will not turn your website into a gold mine overnight, but as part of a well formed SEO strategy, they can certainly help.

There are many people who say you should only ever add two or three META tags to your site. There are those who say you should add hundreds. The simple fact is that there are many that could be appropriate to your site, and you should judge each of them on its individual merit.

META tags all go within the HEAD section of your site. That is to say, within the <head> and </head> tags.

Index

• The Title Tag
  Absolutely invaluable.
• The Description Tag
  Useful if used correctly
• The Keywords Tag
  Becoming less useful, but may be worth adding.
• The Robots Tag
  Can be useful in the right circumstances.
• The Content-Type, Content-Style-Type and Content-Language Tags
  Can be useful in the right circumstances.
• The Refresh Tag
  Can be useful in the right circumstances.
• The Pragma Tag
  Can be useful in the right circumstances.
• The Revisit-After and Expires Tags
  Usually useless, unless you actually want to put the spiders off for a certain period of time.
• Useless Tags
  Useless all the time, unsurprisingly.
• ICRA Label
  Useful

The Title Tag

The TITLE tag is NOT a META tag. But it does contain metadata, and it is the most important tag on a page and is closely related to them, so I am including it here.

Title tags are displayed in the top of a browser window, and are often used as a link from search engine results listings, so form them well. They should be descriptive and short (ideally under 70 characters), and they are also often used as bookmark titles, so it is important that you ensure your primary keyword phrase for a page is here, and that the title makes sense all by itself.

The Description Tag

This is one of the few META tags that can be considered important. The text within this is displayed by some search engines as the description to your site. A description tag should usually be kept to under around 150 - 200 characters and it is important to ensure that this tag reads well, and that it describes the page accurately.

There is no point in telling the user that your page contains thousands of pictures of Alicia Silverstone in lacy underwear if when they arrive on the page they see nothing but a sales pitch for tinned goulash. An extreme example, perhaps, but does demonstrate the point that it is better to have visitors who are interested in your product or content than those who aren't. Numbers are unimportant if they don't convert to sales, and this will help to qualify your visitors before they arrive.

The Keywords Tag

Fairly self explanatory, this tag is used to list keywords for your page. These are words you think are relevant to your page - words that if entered into a search engine should return your site. Search engines do not pay much attention to this, if any, as it has been abused for many years, but some do still use it to some small extent, so you may consider it worth adding.

Try to limit yourself to as few keywords as possible (the less keywords you list, the more weight each will likely have), certainly no more than 25, and list them with nothing more than spaces between (some people use commas, however this is no longer necessary). There is also no need to repeat the words listed.

As has been widely reported on the web, this tag is not used by many engines, if at all, and you would be wise to spend your time optimising and improving your site in other ways rather than waste time on this particular tag, in my humble opinion.

The Robots Tag

The ROBOTS META tag is one that is very often used when it should not be. The four variations listed above are four of the more common variations in use, and each accomplishes a different task. Never use this tag unless you wish to prevent a search engine spider from doing something. That's what it's there for.

The first of the examples listed above is completely worthless. If you have it on your site, go and delete it. That tag does nothing more than tell a search engine spider to behave exactly as it normally does. It does not benefit a site, does not get you crawled faster or more often, and will not suddenly make your site more popular than Google.

The second of these can be useful, for example on printer-friendly pages (where the content on the page is a duplicate of the original). This tag tells a search engine spider not to list the page it is viewing, but to follow the links away from the page anyway. The third of these is the reverse of the above, and tells a spider to list a page in it's results but not to follow the links on the page. Both of these have their uses, but these are very rare, so think carefully before adding these before you do.

The last tag tells a spider not to index a page or follow the links on it. It is extremely rare that you would want to use this (why would anyone want a page on the web that people cannot find?) but is included for the sake of completeness (some people use this for login pages or other similar pages they do not want listed).

There are more instructions you can add to this tag, the most notable of which is NOARCHIVE. This simply tells a search engine spider not to serve archived copies of the page to people viewing the search engine results (for example, Google offer a cached copy of sites in search results, and this will prevent Google from doing so). The tag to add to only prevent search engines making archived copies of your site publically available is:

The Content-Type, Content-Style-Type and Content-Language Tags

These are again quite common on some sites, and again have their uses. It is a wise idea if you are using an unusual language or style to mention it here, but by no means essential, as with most META tags. The W3C provide a more comprehensive resource for [url=http://www.w3.org/International/O-charset.html]character set information[/url], so if you do wish to use this, I recommend that as a good place to start reading.

Meta tags that use the "http-equiv" attribute rather than the "name" attribute, like these, allow you define within a document something that would usually be defined in HTTP headers (sent by your server). If you have no control over the headers sent with your web pages, but still need to define a content type or content style type (and so on), these are the tags you are looking for.

The Refresh Tag

Most useful on a chat page, or when a page has moved, this instructs a browser to refresh the page after a certain interval of seconds. If the second half of the content attribute is a URL, the refresh will take the user to the URL specified rather than simply refresh the current page. This can be, and sometimes is, used mischievously to prevent a user from clicking their back button to leave a page, something likely to annoy visitors enough that they may never return.

The Pragma Tag

Not very widely used, this tag asks a browser not to cache a page. Though this can be useful if a page on your site is frequently updated (for example a news site or a forum), it will often just increase your bandwidth bills and slow down your users' browsing experience. There is also no guarantee that a browser will pay attention to it.

Interestingly enough though, Microsoft recommend that if you do want to use this, you add the tag in a second HEAD at the end of the document, like so:

The Revisit-After and Expires Tags

There are a huge number of sites that say you should add the first of these to your site, because it tells search engine spiders how often to index your page. Which is a common misconception. The tag was created by SearchBC, who have said they no longer use it. Originally, it was created as a tool to suggest to the spider how often a page should be indexed. Few have ever been able to agree on the format of the tag. At the end of the day, remember that the search engines do not care how often you want them to index your pages - they will index as and when they feel like it. Some are clever enough to have a rough idea of how often you update your site, and will make use of that. Some are not that bright, and will come around when the mood takes them.

Assuming you are happy for the spiders to index your site as often as possible, as most people are, you would do well to leave this out. The spiders will return to your site as often as they deem fit, and the only way to influence the frequency this occurs at is to just keep adding new content on a regular basis.

The "Expires" tag tells browsers and search engine spiders when the document should be considered expired. This is worth using, of course, if there is a date on which the relevant document will be no longer valid. However, at this time, the search engines will often drop the page from their index - you should use the "Expires" tag only if this is what you want.

Useless Tags

A select few engines sometimes make small use of a select few of these, but most of these (and the others to be found on this [url=http://www.bauser.com/websnob/meta/useless.html]list of useless META tags[/url]) are better placed on a page, or not used at all. Most of these are added automatically by HTML editors, and some are added by over-zealous META tag addicts. In my opinion, these are best avoided, as they do little more than clutter up your code.

ICRA Label

Last but not least, something a little more unusual. The ICRA (Internet Content Rating Association) is an ideal I am happy to support, as they provide a means for helping webmasters to identify their content as suitable (or not) for certain age groups.

Simply put, you can visit their [url=http://www.icra.org/_en/label/extended/]label generator[/url] and tell the generator what your site contains. That data can then be used to help keep any content not appropriate for young eyes away from them. The data is used by some search engines and some browsers can be set to avoid pages without labels.