Custom Robots Header Tags not only specifies crawling way for search engines but also for open directories, indexing no images, not translating our pages, disabling search engines to save cached copy of our pages and so on.
So lets start the tutorial.
What is Robots Header Tags?
Firstly, we must know about what is robots header tags before adding them to our blog. So we will specify each option briefly here :
1. all : This tag allows crawlers to crawl your blog freely without any boundations.They can easily index, crawl and expose your contents.
2. noindex : This is for secret blogs like if your blog is for private use then you must try to hide the url from exposing it publicly. So this tag lets crawlers to stop crawling your blog.
3. nofollow : All your outbound links are dofollow by default. This let search engines creep upon your pages you page bounded to. If you don't want search engines to look at your outbound links then this tag will helps you.
4. none : It is a combination of noindex and nofollow. You can set it accordingly as per your needs.
5. noarchive : Search engines sometimes saved a copy of your blog pages and show it as cached copy. Enabling this tag do not let crawlers saved cached copy.
6. nosnippet : If you want to keep your post contents secret then you can enable this tag.
7. noodp : This tag disable crawling your blog by Open Directories.
8. notranslate : Enabling this tag do not allow translation of your blog pages.
9. noimageindex : As specified by the name, this tag disable indexing your blog images.
10. unavailable_after : This let webpage indexing after a specified time.
Now, I hope you understood the robots header tags. So now lets start adding these tags to your blog.