Apologies to anyone whose comment ended up in the moderation queue.

Today's comment spam review process:

Apologies to anyone whose comment ended up in the moderation queue. I normally keep on top of it, but a couple of weeks of putting it off and what started as a small pile of comments to manage has quickly ballooned into a 10,000 comment monster. Time for a comment spam prevention rethink, I reckon.

del.icio.us describes itself as a social bookmarking site, for it allows all users of the net to share their bookmarks with others. Unlike similar enterprises that went before it, users' shared bookmarks are not listed only under their name. Each bookmark also has a set of "tags" associated with it. These tags are words that identify what that page is about. An article on, well, security in PHP would have the tags "php" and "security". Maybe even "webdev" and "programming" as well.

Each user can pick their own tags for their own bookmarks. You can also browse all available bookmarks by tag - meaning that if you wanted to see all bookmarks about PHP, you would simply browse to the PHP tag, and voila - there you would find all bookmarked pages about PHP.

What's Wrong with Directories?

Tags are an intelligent way of organising data. Regular directories work on a similar basis to a filing cabinet - items are stored within folders within drawers, and often only to be found in one place. Of course, that fails to make use of the power of databases and computers. Tags, on the other hand, allow a single item to be found in all the places it should be, rather than just the one place that is the single best fit.

This site, for example, is listed at DMOZ under "Web Design and Development: FAQs, Help, and Tutorials" - a good fit, but it also contains writing on internet marketing, browsers, usability and accessibility, and there are resources available as well as a blog. DMOZ cannot reflect this with its rigid and antiquated structure. At del.icio.us, however, it is associated with the following tags: css, php, design, web, programming, blog, webdev, blogs, development, webdesign, reference, cheatsheet, resources, code, mysql, tips, apache, web-design, tools, tutorial, tech, tutorials, computer, web-dev, html, database. And that's just the front page - specific articles are all listed with their own tags. This means that sites and pages listed by users of del.icio.us are classified and organised in a much more effective and user-friendly way.

There are more serious flaws in the directory model though. The most significant problem with web directories is the editors themselves. A web directory requires editors in order to function, and these can either be paid employees or volunteers. If your directory has paid editors working for it, you are left with no serious choice but to charge a fee for submission, in order to cover your editors' wages. That system scales pretty well - if the directory succeeds and becomes popular, the fees for the extra submissions should be enough to cover the wages of the extra editors required to process those submissions.

A volunteer system does have advantages over the paid-editor model. Because volunteers are not paid, a listing in a directory with volunteer editors can be free. This means that non-profit information sites and low-traffic sites can be listed in the directory (a fee for submission will usually prevent that), and means that editors can go out and find sites themselves to be listed.

Both of these systems have their problems. Volunteer editors are volunteers - making it much harder to hold them accountable for laziness or incompetence. DMOZ - a directory with around five to ten thousands volunteer editors - is a great example of this: submissions are often not processed for many months, if at all. Also, because it is a volunteer position, uncrupulous folk are far more likely to accept bribes to list or de-list sites - they stand to lose very little if discovered - and there are plenty of people who claim that a great many editors do just that. The system can also scale badly - if tens of thousands of submissions suddenly require processing, it can be very difficult to source the hundreds or thousands of editors needed to manage that influx.

The paid system leads to an exclusive directory, which by definition becomes one that is missing out on a huge amount of quality content. Yahoo's fees of hundreds of dollars for a submission have always seemed to many to be completely disproportionate to the benefit of a listing and for many people are higher than the cost of hosting a site or the income from it. As a result, for a long time (and this is still true to a great extent) Yahoo has been an incomplete directory, lacking the in-depth listings required by today's discerning web surfer.

Why is del.icio.us better?

del.icio.us is different to both of these systems. It is similar to a peer-review system, in fact. One person bookmarking one page can count as a vote for that page. As the user will have added tags as well, their vote tells the system that one person believes that the page in question is related to each of the tags they listed. After a few hundred people have bookmarked the same link, you'll begin to see some tags used more than others, giving you an idea of how closely the target page relates to each of those tags.

What this has created is a kind of directory with a distributed editting system. The editors are volunteers, but because of their sheer numbers it it much harder for any one editor to affect listings. If one editor is lazy, it does not matter - there are thousands more covering the same topic. If an editor makes a mistake, and lists a page or site under the wrong tag, it doesn't matter - the huge numbers of other editors will make up for it.

Spam is likely to become a huge problem for del.icio.us. To a degree, there is already spam within the index, and some work has already been put in to preventing spam. del.icio.us's robots.txt file prevents indexing of the whole site, so no link popularity benefits from spamming the index directly. However, del.icio.us can still generate plenty of traffic and by virtue of the RSS feeds it generates can generate link popularity from other sites.

Luckily, there are plenty of signals they could look for to weed out spam. Their database can already tell them what tags are related, and what sites. If a user starts to list unrelated sites with tags unrelated to those sites, they may well be a spammer. If lots of new users suddenly join and all bookmark the same page instantly, using the same tags, again that may well be spam. IP tracking and the registration system (that requires a valid email and features a turing test) should make automated spam far harder. Ultimately, it may be del.icio.us's own success that makes spamming virtually impossible. With enough users on the site, a spammer may need to create hundreds, even thousands, of fake users to have a site listed in the "popular" section, or listed highly for a specific tag.

What next?

The intention of del.icio.us is not (at the moment) to become or create a directory. It is by pure fluke that they have created a site and a system so able to perform the same function as a directory but without the problems associated with that. It may well be that other similar sites will spring up whose aim is to build a directory, especially those involved in search, with a large user base. I would be greatly susprised if Google, MSN and Yahoo were not already watching del.icio.us very closely and with great interest. I would be equally surprised if none of them bought or created a social bookmarking product in the next few months, as the power of a distributed editting becomes apparent.

Update (14th December 2005)

It appears that I was rather close to the mark with my guess at what would happen next for del.icio.us - they have just been bought by Yahoo. It will be interesting to see how Yahoo integrate del.icio.us with their other services. I only hope they don't mess it up like so many promising sites before!

The decision made by these short-sighted bloggers was to display, on their site, a list of recent referrers to each page. I can't imagine any reason why a visitor might be in the least bit interested in seeing this, but nevertheless many sites now display referrers on every page.

As search engine spiders visit sites, they grab the contents of each page they visit. They use this snapshot in their index - meaning that although a page may change every minute or two, a search engine may be using a single copy of a page for several days, or even weeks.

So a referral URL that is on a page when the spiders come to visit can have quite a bit of value, if the search engine visiting uses link popularity in any way (Google uses link popularity, as do many others).

So marketers have started to use programs to visit pages using a fake referral header, to get their URLs listed on as many sites as possible, in the hopes that this will increase their traffic.

However, this renders log files almost completely useless. These fake visitors usually visit from search engines, having searched for a keyphrase relevant to their own site. They skew statistics relating to number of visitors received, the countries used to visit, the technology used to view the page, how users found the page, how long they spent on the site ... and so on.

A webmaster may find their search engine rankings dropping because of this, and they may find search engines have removed them completely. Many sites that use spam techniques are quickly identified and penalised, and penalties will often be applied to sites that link to them as well.

There are plenty of techniques available for blocking referrer spam, and everyone has their favourite. Personally, I use a combination of two techniques.

The first is fairly simple - my referrer log is not indexable. I don't display referrers on the pages of my site. My referral log is publicly available, but search engines are instructed to ignore it. This removes the main incentive for people to referrer-spam my site (the other reason for this type of spam - the hope that the site owner will themselves visit the spamming URL - is less common, because it has such a low response rate).

Second, I use an .htaccess file to block requests from whatever I've managed to identify as either a crawler designed to find URLs to spam or a spamming URL. This is a relatively simple blacklist, and though it cannot work as a long term solution to this problem, it keeps me happy for now.

To implement this technique on your own site, first make sure you are running Apache with mod_rewrite. If you are, create a file called ".htaccess" (just that, not .htaccess.txt or anything else) and paste the following into it:

Update: 14th September 2005

The list below has been expanded substantially over the last year, and now covers much more spam than before. As stated before, this is not a practical solution to the problem in the long term, as this list can only ever get longer and longer, and may become unmaintainable, or even (eventually) slow a site to a crawl as all the rules are processed. However, as of now, it is still a useful tool.

The above will block just about all of the most common referral spam that I've seen so far. I'm adding to the list constantly (last addition: 14th September 2005) so do check back and see if there are updates if you're using it.

One potential problem with this technique, other than that it will, in time, become useless as too many URLs are added, is that there is always a possibility authentic visitors will be blocked. So, on this site, instead of the last line above, I've actually used something a little more user-friendly:

Instead of a "Forbidden" message, this displays a quick note explaining why there has been an error and that the user can click on a link to proceed. If you want to check this out for yourself, try visiting http://www.addedbytes.com/swingers/block-referrer-spam/ (note the "swingers" portion of the URL). This page will reload with a new URL. Then try visiting http://www.addedbytes.com/spam/block-referrer-spam/. You should find you get a message explaining what has happened, and a URL to click if you want to proceed.

And there we have it. With minimum effort (for now), referral log spamming in my site has been almost entirely removed. Before adding this set of rules and scripts, I was seeing around 200 fake referrals per day in my log files. Now, I see about 3 or 4 a week. Hopefully, this will continue until I can devise a better way of protecting against this kind of problem - before blacklists become an impossibility to manage.