Googlebot: Google's Web Crawler

[pgolovko]

Some reading material for you guys ....

http://www.google.com/intl/en/webmasters/bot.html

[Simon Gooffin]

thanks for the link :)

I just want to add the following. I'm not sure if anyone noticed we have something like rel="nofollow" for our suggest link and suggest category pages.

4. How do I tell Googlebot not to crawl a single outgoing link on a page?

Meta tags can exclude all outgoing links on a page, but you can also instruct Googlebot not to crawl individual links by adding rel="nofollow" to a hyperlink. When Google sees the attribute rel="nofollow" on hyperlinks, those links won't get any credit when we rank websites in our search results. For example a link,

<a href=http://www.example.com/>This is a great link!</a>

could be replaced with

I can't vouch for this link (http://www.example.com/).

that's what we did in our script. now just imagine you have 1000 categories and everyone has a link to almost similar pages. suggest-link and suggest categories? Do you think google really likes this? hmm, IMHO I will better get rid of 1000 useless similar pages on my site than have them indexed.
These are just my observations based on google and some experience in webmastering :)

[Greg]

Home, Search, New Links, Top Links also have the "nofollow"

[Simon Gooffin]

oops, I screwed up. typo in default template
*** open Layout.php and find the next code ***
if (($key = 'suggest-link') || ($key == 'suggest-category'))

*** change it to the following ***
if (($key == 'suggest-link') || ($key == 'suggest-category'))

*** save changes ***
It will work fine

[subseo]

Speaking of nofollow. I found it useful in PHPLD script, that you coud manually assign nofollow to certain links at the time of approval. I have used it often in my general directory (right dangerous sites I have been deleting right away, but flagging the potentially dangerous sites with nofollow, so that further lowering the risk of my directory being banned in Google at some time in the future)

[Simon Gooffin]

hmm, quite cool idea.. I do like it to be implemented in our script

[dswofford]

oops, I screwed up. typo in default template
*** open Layout.php and find the next code ***
if (($key = 'suggest-link') || ($key == 'suggest-category'))

*** change it to the following ***
if (($key == 'suggest-link') || ($key == 'suggest-category'))

*** save changes ***
It will work fine

What does this change simon ? is everyone suggested to do this?

[Simon Gooffin]

yeah. everyone should do this..
It will fix nofollow problem in menu and this attribute will be used for suggest link and suggest category pages only.