Robots.txt for SMF forum with pretty urls

[Webnauts]

We still need to disallow

/BB?topic       - with no forward slash after BB

I saw some print pages indexed aswell, "Disallow: /BB/*?*" is in the robots.txt which I think should solve it, unless they're old caches.

Have you seen these issue being logged in? If not can you share some example URLs?

I've seen the print pages cached in Google, but about a month ago. It seemed that was all (and profiles) that were wanting to be indexed at the time.

/BB?topic was coming up through stumble buttons. In essence stumble were giving 3 different urls for the same post depending where you were to the stumble button from.

So how did the URL look like?

Like this http:// www. davidcastle.org/ BB?topic or like this http:// www. davidcastle.org/ BB/ BB? topic   

[daniboy]

The first one;   http: // www. davidcastle. org/ BB? topic

[Webnauts]

We still need to disallow

/BB?topic       - with no forward slash after BB

I saw some print pages indexed aswell, "Disallow: /BB/*?*" is in the robots.txt which I think should solve it, unless they're old caches.

Have you seen these issue being logged in? If not can you share some example URLs?

I've seen the print pages cached in Google, but about a month ago. It seemed that was all (and profiles) that were wanting to be indexed at the time.

/BB?topic was coming up through stumble buttons. In essence stumble were giving 3 different urls for the same post depending where you were to the stumble button from.

Thanks buddy. I updated the robots.txt in my above post accordingly.

[Webnauts]

I am sure you have already "killed" the install files?
and reverted any temporary 777 perms back to normal.
Can't remember whether SMF does all that on auto or
nags you.

Just double checking .... not lessons on sucking eggs 

Is this an official invitation for hackers? 

[Webnauts]

A while ago, still when we had the old board I created a robots.txt and I reduced the supplemental pages down to 1%!
I created above a new one http://www.mapelli.info/tools/supplemental-index-ratio-calculator?domain=davidcastle.org%2FBB%2F but I see it is not implemented. And here are the bad news:

Supplemental Ratio for davidcastle.org/BB/: 32.7%

    * Google has a total of 4710 pages indexed from davidcastle.org/BB/
    * 3170 are in the main index
    * 1540 are in the supplemental index

To be honest I am very disappointed...  :

[MuNKy]

Doh! Thats not good, you spent ages sorting all that crap out  :

[Webnauts]

Doh! Thats not good, you spent ages sorting all that crap out  :

That is what really pissed me off Darren. One thing is sure: I will never do that mistake again.

[ash]

Webnauts

I've implemented the robots text now.

By the way i've edited your link above as the site you were linking to was flagged by my firewall / antivirus software as carrying a trojan virus / file and kept crashing my browser.

[MuNKy]

Strange, i use that site all the time and i've not had any problems with it. You sure its not something thats on your PC ash?

[ash]

Strange, i use that site all the time and i've not had any problems with it. You sure its not something thats on your PC ash?

Just tested in on another comp and still getting the same warning. Strange. I'll reinstate the link and see if anyone else is having issues.