Google indexing problem

[ken heskin]

Hi, I am having problems getting my site indexed within google.
I dont understand Robots but Google tell me that I need to edit/correct the file below....

User-agent: *
Disallow: admin/
Disallow: includes/
Disallow: templates/
Disallow: logout.php
Disallow: suggest-link.php
Disallow: suggest-category.php

I think these files are denying Google Robots access!!!
The files are at http//www.mysite.co.uk/robots.txt

Also my sitemap lists my site as http//mysite.co.uk What happened to the "WWW." ??

[ken heskin]

Where is the support guy's ? :no-no:

[cyanide]

Well, you do have the slash in the wrong place

Should be:
User-agent: *
Disallow: /admin/
Disallow: /includes/
Disallow: /templates/
Disallow: /logout.php
Disallow: /suggest-link.php
Disallow: /suggest-category.php

[ken heskin]

Thanks for the reply. Where can I find the files to edit them ?

Thanks again.

[Dave Baker]

Hello ken,
Please try finding in the Root of your directory. Name of the file robots.txt

@cyanide Welcome to our support forum!

[cyanide]

@cyanide Welcome to our support forum!
Thanks Dave,
Downloaded the free version to give it a whirl... so far so good
Well done

[Simon Gooffin]

Hi,
I'm pretty sure this file can not cause any problem with Google. It's just preventing google bot to index some files from directories. There is no need to index suggest link or suggest category page as in some cases it can be more a problem for you than a positive thing.

[ken heskin]

Many Thanks

[Simon Gooffin]

np :) please ask in case you have any questions.

[OWG]

The restrictions in the htaccess are spot on. The reason thery are there (I assume) is to stop google and other agents from taking and indexing the unimportant pages in your directory. You want the MEAT pages in there, not the =bare bones ones. Simon and the team have prevented google from indexing what will almost certainly be classed as duplicate pages.

[Simon Gooffin]

Simon and the team have prevented google from indexing what will almost certainly be classed as duplicate pages.
Exactly... Just imagine there are 2K categories in your directory. So you might have about 4000 (suggest link and suggest category) pages that are completely similar with one exception - category title where a link or a category is going to be be submitted. I'm pretty sure Google will not like this :no-no:

[gnordvik]

There is another problem with duplicated pages (as far as I can see) not addressed in the support forums.

Each category has a unique title, but if that category is loaded with listings you might have a problem. If you e.g. only show 5 links per page in a specific category and you got 100 listings, then you have generated 20 pages with identical titles and descriptions. This is a problem! Has someone got a fix for this?

[smartpc]

If you login to your google webmaster account you can change the way your site is indexed: ie. yourdomain.com or www.yourdomain.com

when google is first indexing sites they sometimes index without the www , this happened to me before after a little bit it will change to the www. version, but you can tell them now as stated above.

Yahoo is indexing the suggest category etc....

I have loads of indexs with suggest-category.php?id=3/suggest-category.php?id=10/suggest-category.php?id=11 etc

google and most se`s are now indexing 500kb as apposed to the previous 101k of a page file. ;-)

Will check on the multiple pages with same title... but I personally dont think its a problem but thats only my opinion as of now, until find out otherwise.

It would really be impossible to change the title keywords and description for each category page... ie(page 1,2,3,4,5....) you would spend your days doing this... you run a directory I think google knows you run a directory. I cant see it triggering a duplicate content filter as 99% of the content is different on each page.

But will research but I have no intention of doing this. When I truely start with my directory.

If you haven`t touched the robots file there is no problem there, when you say you are having trouble do you mean its indexed but google just won`t index all you pages....?

:crazy-be: nearly worked up a sweat there.......lol

[gnordvik]

I'm not sure to what effect this will affect Google rankings. The reason why i mentioned this is that in Google Webmaster tools under Content analysis they report that I have a lot of Duplicate title tags and a lot of duplicate meta descriptions.

[smartpc]

I will look into it.... and see if I can uncover any definite patterns from google.

[smartpc]

Ok I know what your saying about the theory of duplicate content with relation to title tags and descriptions.

But you have raised a good question in relation to a directory, the only problem I see is that as your directory section grows and you get more and more pages in 1 category/subcategory (in theory) so if you had 2000 listings that were appropriate for that section - if you keep changing the title tags and description for pages you are diluting the initial category meaning, if you gets me.

But I cant honestly give you a definite answer so I will ask somebody much smarter and see the answer they give just might take a bit, if I get an answer at all. It might be a site by site answer. Trusted sites get away with stuff some of us mortals would get a penalty straight away for.

There`s multiple factors that could give positive and negative affects.

[gnordvik]

Thank you for taking the time to examine the problem:good:

I think that a fix where the page number is listed from page 2 and out should be enough.

E.g

Page 1: Title
Met-descr.: Descr.

Page 2: Page 2 - Title
Met-descr.: Page 2. Descr.

Etc.Etc.

If there is an easy solution for doing this then I would be really greatful. I'm not sure if this really has any positive effect at all, but I'm quite sure that this has no negative effects.

Thanks again!