One problem with the blog format is that the same content can show up on several URLs. This content layout is nice for humans. In the case of my blog, the same post content can show up on the main page, a category URL and an archive URL.

Unfortunately, what is convenient for humans is not so good for search engines. There are two aspects of the standard blog format which cause search engine problems. The first is the dynamic nature of some blog URLs. Consider the main page of an active blog. Most only show about ten posts; older posts are removed as newer ones are created. Often this results in a particular URL not containing the content the search engine thinks it does. Personally, I find this incredibly annoying since I often have to search the site using a local search engine after Google has directed me to the main page of a blog. The second problem of the blog format with respect to search engines is that some URLs, like a category URL, contain many posts which are not directly related to a particular search. This results in having to search the page with the browser’s find function after the search engine gets you there.

Both of these problems have been annoying me for some time now. So today I did a little digging. Fortunately there is a solution, the robots meta tag. This tag specifies, on a page by page basis, whether or not the content on the current page should be indexed by the search engine and if links on the current page should be followed.

The solution then, is simple. URLs which contain multiple posts should be marked “noindex,follow” while individual posts should be marked “index,follow”. This should result in the content of each post only being in the search engine database once. I also found a post called A critical SEO Tip for WordPress which describes a way to accomplish this in WordPress. The slightly modified version of this solution which I have added to my WordPress theme’s header.php is below. Unless there are downsides to this approach that I don’t know of, I think every theme author should add something like this to their theme.

<?php
if (is_single() || is_page() || is_author()) {
echo "<meta name=\"robots\" content=\"index,follow\"/>\n";
} else {
echo "<meta name=\"robots\" content=\"noindex,follow\"/>\n";
}
?>