One of the many errors new bloggers makes is not creating a Robots.txt file, so as to prevent penalty duplication by Google

You can also add to the Robots.txt file syntax, of directories and files you don’t wish search engines to index

Well I suggest you do create a Robots.txt file

I did create one but it wasn’t as detailed as I would like it to be. Use this to check the number of supplemental duplicates site:www.YOURDOMAIN.COM -view ***

I checked Google, and saw my Wordpress plugins folder being indexed too. That’s not good as it allows people to exploit it

That can result in a security flaws if you happen to upload sensitive information in there.

This was my previous Robots.txt

User-agent: *
Disallow: /feed/
Disallow: /archives/
Disallow: /category/
Disallow: /trackback/
Disallow: /comments/
Disallow: /wp-admin/
Disallow: /cgi-bin/

The thing is I saw that supplemental pages being indexed by Google and by adding a more detailed syntax of what not allowed or allowed to be indexed for the bots, it will lessen the penalty and thus better rankings in the search results pages. I now exclude my pages, categories, feeds and unknown pages to be indexed with the below new syntax

My Current One Robots.txt

User-agent: *
Disallow: /*?*
Disallow: /*?
Disallow: /?s=
Disallow: /tag/
Disallow: /rss/
Disallow: /feed/
Disallow: /date/
Disallow: /search/
Disallow: /about/
Disallow: /make-money-online/
Disallow: /links-page/
Disallow: /most-
Disallow: /popular-articles/
Disallow: /archive/
Disallow: /archives/
Disallow: /category/
Disallow: /category/*/*
Disallow: /trackback/
Disallow: */trackback
Disallow: /contact-form/
Disallow: /page/
Disallow: /pages/
Disallow: */comments
Disallow: /comments/
Disallow: /comments/feed/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-content/cache/
Allow: /wp-content/uploads/
Disallow: /cgi-bin/
Disallow: /cash/

# Google Image
User-agent: Googlebot-Image
Allow: /*

# Google Googlebot
User-agent: Googlebot
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.wmv$
Disallow: /*.avi$
Disallow: /*.cgi$
Disallow: /*.txt$

# Does anyone care I love Google Apache htaccess
Sitemap: http://www.shamoneymaker.com/sitemap.xml

Observe I also added my sitemap location to the last line in the file. There has also been other bloggers such as Everton, 20 steps and FiLi who have created a Robots.txt and saw a marked increase in their blog traffic. Trust me, Google bots loves Robots.txt

List of online Robots.txt generator if you’re unsure to create one yourself

Any feedbacks? Did you spot any errors I made perhaps

Related Articles