If you own a forum/fansite and see that robots are visiting your site, making it look un-proffesinal on the "Whos Online" page seeing "Google Bot" ect. Well this guide will show you how to limit or remove them;
To prevent all robots from indexing a page on your site, you'd place the following meta tag into the <HEAD> section of your page:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
To allow other robots to index the page on your site, preventing only Google's robots from indexing the page, you'd use the following tag:
<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">
To allow robots to index the page on your site but instruct them not to follow outgoing links, you'd use the following tag:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
To allow robots to index the page on your site but instruct them not to index images on that page, you'd use the following tag:
<META NAME="ROBOTS" CONTENT="NOIMAGEINDEX">
Results 1 to 7 of 7
-
22-03-2008, 05:39 PM #1
[TUT] Removing Robots from your site! [/TUT]
Last edited by Lee; 22-03-2008 at 05:42 PM.
-
Good Tut
Also you missed the "> at the end of the 1st meta tagLets set the stage on fire, and hollywood will be jealous.
-
22-03-2008, 05:43 PM #3
Thanks, edited
*REMOVED*
Edited by nvrspk4 (Assistant General Manager): Please do not ask for rep.Last edited by nvrspk4; 24-03-2008 at 04:00 AM.
-
Don't use caps, and this won't stop bad bots.
How could this hapen to meeeeeeeeeeeeeee?
lol.
-
Don't use caps as previously said, and maybe use xhtml as that is more commonly used now me thinks
eg:
HTML Code:<meta name="robots" content="noindex, nofollow" />
-
Not all search engines use meta tags, you are best off using robots.txt if you really want to block crawlers. But why would you want to block them anyway? Without them you site would never get indexed on search engines...
-
Originally Posted by Tomm
Short robots.txt guide (I just wrote up);
(Sulakes below)
User-agent: *
Disallow: /gallery/image_bank/
Disallow: /careers/apply
Disallow: /contact/index.html
e.g:
User-agent: *Disallow: /calon/index.phpPut the file URL e.g: calon/index.php (if in folder named ''calon'')
Disallow: /calon/index.php
And also use 'User-agent: *' at the top of your robots.txt file Enjoy!
Reply With Quote
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts