mersenneforum.org  

Go Back   mersenneforum.org > Other Stuff > Forum Feedback

Reply
 
Thread Tools
Old 2007-06-21, 13:02   #1
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

DD016 Posts
Default no google searching of mersenneforum?

Is google forbidden from indexing mersenneforum.org? There are no results from searches when limited to this URL.
jasonp is offline   Reply With Quote
Old 2007-06-21, 14:50   #2
Xyzzy
 
Xyzzy's Avatar
 
"Mike"
Aug 2002

22·32·223 Posts
Default

We have the robots.txt file set to block all search engines. We have tried to let just Google in, but the number of spiders they send in can sometimes be a problem. Also, for some reason, a pile of other, unrelated, spiders start hitting us, even though we only specify Google.

Normally, having a few spiders indexing the forum would be okay. We have plenty of bandwidth. But at peak usage times we come close to our database connection limit, and there is no way, that we are aware of, to tell the spiders to only index at certain times.

We are open to suggestions.

Xyzzy is offline   Reply With Quote
Old 2007-06-21, 15:11   #3
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

24·13·17 Posts
Default

Quote:
Originally Posted by Xyzzy View Post
We are open to suggestions.

It isn't a big deal for me, most of my google searches only ended up with one or two hits to the forum, even for topics that I know get a lot of discussion here.

It's a little surprising that with search being a multi-zillion dollar industry, that all the players will leave you alone if you say so. You'd think somebody would try to index everything no matter what the content providers say.
jasonp is offline   Reply With Quote
Old 2007-06-21, 15:31   #4
Mini-Geek
Account Deleted
 
Mini-Geek's Avatar
 
"Tim Sorbera"
Aug 2006
San Antonio, TX USA

10AB16 Posts
Default

Quote:
Originally Posted by Xyzzy View Post
We have the robots.txt file set to block all search engines. We have tried to let just Google in, but the number of spiders they send in can sometimes be a problem. Also, for some reason, a pile of other, unrelated, spiders start hitting us, even though we only specify Google.

Normally, having a few spiders indexing the forum would be okay. We have plenty of bandwidth. But at peak usage times we come close to our database connection limit, and there is no way, that we are aware of, to tell the spiders to only index at certain times.

We are open to suggestions.

I don't know of a way to make it wait for a time of day, but if you verify the site in Google Webmaster Tools, you can set its crawl rate to Slower. I think you'll have to re-set the setting every 3 months.
Mini-Geek is offline   Reply With Quote
Old 2007-06-21, 15:51   #5
BlisteringSheep
 
BlisteringSheep's Avatar
 
Oct 2006
On a Suzuki Boulevard C90

24610 Posts
Default

Quote:
Originally Posted by Xyzzy View Post
We have the robots.txt file set to block all search engines. We have tried to let just Google in, but the number of spiders they send in can sometimes be a problem. Also, for some reason, a pile of other, unrelated, spiders start hitting us, even though we only specify Google.
, do you mind sharing what you had put in robots.txt when you tried this?

Last fiddled with by BlisteringSheep on 2007-06-21 at 15:51
BlisteringSheep is offline   Reply With Quote
Old 2007-06-21, 19:44   #6
Xyzzy
 
Xyzzy's Avatar
 
"Mike"
Aug 2002

11111010111002 Posts
Default

We've put up a new robots.txt file for Google bots. We used the Google "Webmaster Tool" page to create and test it. We were not aware these tools existed.

Hopefully, the other bots will leave us alone.

Let us know if you start seeing pages show up during Google searches.

Xyzzy is offline   Reply With Quote
Old 2007-06-21, 19:54   #7
Mini-Geek
Account Deleted
 
Mini-Geek's Avatar
 
"Tim Sorbera"
Aug 2006
San Antonio, TX USA

17×251 Posts
Default

Quote:
Originally Posted by Xyzzy View Post
Let us know if you start seeing pages show up during Google searches.
http://www.google.com/search?q=site:www.mersenneforum.org
Mini-Geek is offline   Reply With Quote
Old 2007-06-21, 20:04   #8
Xyzzy
 
Xyzzy's Avatar
 
"Mike"
Aug 2002

22×32×223 Posts
Default

Okay. We will rephrase that statement.

We will forget all about this thread in approximately 46 minutes. In a few days/weeks/months/years, if you should happen to see that some pages are being indexed by Google, and you feel like letting someone know, then maybe post your findings.

We are sure the trolls know all this stuff already and are enjoying watching us stumble around in the dark looking for the light switch. That is okay, though, because we are going to the beach this weekend and we are going to accidentally leave the closet locked.

(We have to lock the closet anyways, because we think the gerbils would get eaten. We have noticed that there are some odd scratch marks on the metal lid of the gerbil cage.)
Xyzzy is offline   Reply With Quote
Old 2007-06-24, 23:23   #9
Xyzzy
 
Xyzzy's Avatar
 
"Mike"
Aug 2002

22·32·223 Posts
Default

A small start:

http://www.google.com/search?q=site:mersenneforum.org
Xyzzy is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
ftp.mersenneforum.org Mark Rose Forum Feedback 11 2015-05-14 01:44
Should/can mersenneforum.org get its own emoji? only_human Forum Feedback 14 2013-07-30 20:16
The mersenneforum First 300 ewmayer Lounge 15 2012-08-27 18:35
HELP!! Can't Log In to mersenneforum Unregistered Information & Answers 8 2010-09-06 18:24
archives of mersenneforum.org? masser Forum Feedback 2 2007-12-27 15:34

All times are UTC. The time now is 09:39.

Sun Apr 11 09:39:35 UTC 2021 up 3 days, 4:20, 1 user, load averages: 2.53, 2.38, 2.20

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.