no google searching of mersenneforum?

[jasonp]

Is google forbidden from indexing mersenneforum.org? There are no results from searches when limited to this URL.

[Xyzzy]

We have the robots.txt file set to block all search engines. We have tried to let just Google in, but the number of spiders they send in can sometimes be a problem. Also, for some reason, a pile of other, unrelated, spiders start hitting us, even though we only specify Google.

Normally, having a few spiders indexing the forum would be okay. We have plenty of bandwidth. But at peak usage times we come close to our database connection limit, and there is no way, that we are aware of, to tell the spiders to only index at certain times.

We are open to suggestions.

[jasonp]

Originally Posted by Xyzzy

We are open to suggestions.

It isn't a big deal for me, most of my google searches only ended up with one or two hits to the forum, even for topics that I know get a lot of discussion here.

It's a little surprising that with search being a multi-zillion dollar industry, that all the players will leave you alone if you say so. You'd think somebody would try to index everything no matter what the content providers say.

[Mini-Geek]

Originally Posted by Xyzzy

We have the robots.txt file set to block all search engines. We have tried to let just Google in, but the number of spiders they send in can sometimes be a problem. Also, for some reason, a pile of other, unrelated, spiders start hitting us, even though we only specify Google.

Normally, having a few spiders indexing the forum would be okay. We have plenty of bandwidth. But at peak usage times we come close to our database connection limit, and there is no way, that we are aware of, to tell the spiders to only index at certain times.

We are open to suggestions.

I don't know of a way to make it wait for a time of day, but if you verify the site in Google Webmaster Tools, you can set its crawl rate to Slower. I think you'll have to re-set the setting every 3 months.

[BlisteringSheep]

Originally Posted by Xyzzy

We have the robots.txt file set to block all search engines. We have tried to let just Google in, but the number of spiders they send in can sometimes be a problem. Also, for some reason, a pile of other, unrelated, spiders start hitting us, even though we only specify Google.

, do you mind sharing what you had put in robots.txt when you tried this?

[Xyzzy]

We've put up a new robots.txt file for Google bots. We used the Google "Webmaster Tool" page to create and test it. We were not aware these tools existed.

Hopefully, the other bots will leave us alone.

Let us know if you start seeing pages show up during Google searches.

[Mini-Geek]

Originally Posted by Xyzzy

Let us know if you start seeing pages show up during Google searches.

http://www.google.com/search?q=site:www.mersenneforum.org

[Xyzzy]

Okay. We will rephrase that statement.

We will forget all about this thread in approximately 46 minutes. In a few days/weeks/months/years, if you should happen to see that some pages are being indexed by Google, and you feel like letting someone know, then maybe post your findings.

We are sure the trolls know all this stuff already and are enjoying watching us stumble around in the dark looking for the light switch. That is okay, though, because we are going to the beach this weekend and we are going to accidentally leave the closet locked.

(We have to lock the closet anyways, because we think the gerbils would get eaten. We have noticed that there are some odd scratch marks on the metal lid of the gerbil cage.)

[Xyzzy]

A small start:

http://www.google.com/search?q=site:mersenneforum.org