SHARE

Taking about site indexing a proper site index by Search Engine Bots has its base reliance on a robots.txt file which controls the “what content search bots are allowed to index and not”, misuse of this file usually located at the home data directory could lead to total removal of content from the search engine.

Some times people neglect the use of the robots.txt file like its minor and place all point of full concentration on SEO, and sitemap but the truth of the matter is, what’s SEO without content not been indexed .

What is A Robot.txt File
A robot.txt file is the first file checked by search Bots before index your content, it tells search Engine Bots were to crawl and where not to crawl.

But as Mention people misunderstand the concept of this robots.txt file, and thus indeliberately remove there content from the net.

For instance,a single error of direction placement on image on your Blog, could disallow crawl of all images in your Blog, which you might not notice immediately
3 Things Never to Do When Creating A WordPress Robots.txt File

Here is how a Normal Robots.txt file should look like.


User-agent: *

Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /cgi-bin/
Disallow: /wp-includes/
Disallow: /wp-content/themes/
Disallow: /*?*

 

Which you can view at
www.shoutershub.com/robots.txt

1. Don’t Add Unnecessary space
At the beginning of any Line directive

2. Don’t add Comments
Example

#Are you a bot or a human spying on our robots.txt file?

#This is a Robots.txt File for search bots

3 don’t ever Generate a Robots.txt file with a Plugin or tool, create it your self.

Reason be that most WordPress plugin after removal removes all table data created by them, o wen you unknowingly remove them it affects your site all though.

LEAVE A REPLY