Live test: robots.txt, PR, and forums

[Brian Turner]

An earlier question here made me rethink the way I used robots.txt files on my forums.

Essentially, I was concerned about PR dilution and duplicate content dragging down my forum threads from searches, so I set up the following robots.txt:

 User-agent: *
 Disallow: /graphics/
 Disallow: /images/
 Disallow: /forum/attachment.php
 Disallow: /forum/avatar.php
 Disallow: /forum/editpost.php
 Disallow: /forum/member.php
 Disallow: /forum/member2.php
 Disallow: /forum/misc.php
 Disallow: /forum/moderator.php
 Disallow: /forum/newreply.php
 Disallow: /forum/newthread.php
 Disallow: /forum/online.php
 Disallow: /forum/poll.php
 Disallow: /forum/postings.php
 Disallow: /forum/printthread.php
 Disallow: /forum/private.php
 Disallow: /forum/private2.php
 Disallow: /forum/report.php
 Disallow: /forum/search.php
 Disallow: /forum/sendtofriend.php
 Disallow: /forum/threadrate.php
 Disallow: /forum/usercp.php
 Disallow: /forum/admincp/forum/
 Disallow: /forum/modcp/forum/
 Disallow: /forum/images/forum/
 Disallow: /forum/sendmessage.php
 Disallow: /forum/register.php
 Disallow: /forum/subscription.php 
 Disallow: /forum/profile.php

However, somebody earlier on business-talk asked about crippling links, and I effectively replied that even if you prevent Google from following these links, so long as the Googlebots can read the links as links, then in PR calculations these links are still added into the equation.

So if my own words were true, then there was aboslutely no advantage to using the robots.txt file above - it would prevent duplicate content issues - but nothing more - PR would still effectively be lost.

And then I sat down to thinking how if I simply edited the various different vBulletin templates - ie, the printthread template, the HTML archive template, etc etc etc, then I would have access to tens of thousands more links on every forum I run.

So now I've removed all my robots.txt files and will sit and wait on what happens.

At the moment, Google reports the following:

allinurl:www.chronicles-network.net/forum/
27,400 pages indexed
BUT on "repeat the search with the omitted results included" only lists 9,140 indexed pages

allinurl:www.comparative-religion.com/forum/
31,300 pages indexed
BUT on "repeat the search with the omitted results included" only lists 8,580 indexed pages.

allinurl:www.business-talk.co.uk
7,870 pages indexed
BUT on "" lists 11,900 <- - - odd result

I figure the first set of higher figures in search results for the www.chronicles-network.net and www.comparative-religion.com include orphaned pages - printer versions, etc. The business-talk ersult is just plain odd - but this is Google we're talking about.

Now the big reason for this test is no longer about PR into threads - we know the effect is minimal, and I'm sure Google has it's own duplicate content filters in play and can look after itself.

What really concerns me is advertising - I will be renting out my siets to advertising very very soon - text-link advertising to be precise. And I want to offer such great value for money that people aer strongly enticed to buy.

Look at the above figures - would somebody who bought a text-link ad on my chronicles-network want the link on around 30,000 pages, or nearly 10,000 pages? Why, the latter, of course - think of all that anchor text!

So now I'm going to run all my forums through a test - see what effect the extra content has not on PageRank, but on advertising potential - and see whether this has any adverse effect on traffic.

robots.txt files have now all been removed - I'll be sure to report on the results.