Google blocked again

[ixfd64]

A while ago, our robots.txt file blocked everything except for Google, but it is now blocking Google again as well.

Was there a reason for this change?

I feel that this is unfortunate, since we cannot get much publicity this way. Google does cache almost every page it crawls, but the cached pages do not last forever. The cached pages from this forum will probably be gone the next time Google tries to crawl us.

[Xyzzy]

When we allow Google permission to index the forum, a bunch of other search engines come in and put a major strain on the server.

We've checked the syntax multiple times for letting only Google in.

[ewmayer]

Hey, Mike: in the following how-to-set-up-robots.txt page :

http://www.thesitewizard.com/archive/robotstxt.shtml

I note this quote:

"Listing something in your robots.txt is no guarantee that it will be excluded. If you really need to protect something, you should use a .htaccess file (if you are running your site on an Apache server)."

Is that only relevant to directory names, or could it also be for search-engine-specific in|exclusion stuff? [Of course if you don't use Apache it's moot.]

[Xyzzy]

Of course we run Apache.



The .htaccess stuff is, from what we understand from your link, just password protecting directories.

(There are other .htaccess functions, like redirecting stuff, and blocking IP addresses, but we don't think those apply to this discussion.)

For example, we use a .htaccess/,htpasswd deal on our administrative pages, so those never get indexed, but we can't think of a way to allow Google through that kind of system because it requires a password. (Plus, the users would need the password too!)

If there is a way to block search engines this way we are, like the Ferengi, all ears.

[Xyzzy]

For the record, the issue isn't even bandwidth. We can handle a lot more bandwidth than we use currently.

The problem is when a rogue search engine comes in it sometimes tries to index the whole site in one swoop, so the database server gets overloaded.

Google indexes very slowly and uses very little resources. And Google gives us a control panel to control the spider's activity.

We wish there was a way to coordinate with Google to index at certain times, and then we could make a cron job to swap the robots.txt file with a Google-friendly one, at certain times of the day, and then swap it back after a short period of time.

We're sure there is some sort of solution out there.

[xilman]

Originally Posted by Xyzzy

For the record, the issue isn't even bandwidth. We can handle a lot more bandwidth than we use currently.

The problem is when a rogue search engine comes in it sometimes tries to index the whole site in one swoop, so the database server gets overloaded.

Google indexes very slowly and uses very little resources. And Google gives us a control panel to control the spider's activity.

We wish there was a way to coordinate with Google to index at certain times, and then we could make a cron job to swap the robots.txt file with a Google-friendly one, at certain times of the day, and then swap it back after a short period of time.

We're sure there is some sort of solution out there.

Possible idea: do the "rogue" search engines use a consistent set of IP addresses?

If so, block them at the firewall, letting Google and users through unscathed.


Paul

[Xyzzy]

Possible idea: do the "rogue" search engines use a consistent set of IP addresses?

Not that we have been able to determine. However, when we do a lookup of the IP address from within the forum it somehow identifies the "user" as a bot. I think it has something to do with the way the "user" identifies itself. For example, every request for a page is logged with the referring page, the browser, the operating system and a pile of other things.

(It is possible to spoof this data.)

We have all the logs going back a long time but it is a mountain of data.