Replacing Entropia's primenet server

[Ian_H]

Originally Posted by Prime95

Prime95 does what maybeso suggests. [...] Prime95 already does what Ian suggests too!

Neat ;)

[ebx]

Originally Posted by Ian_H

Suppose you get a new Pentium V 4.5GHz box next year: should you be expected to run it for 90 days before 'qualifying' for choice assignments? I'd rather trust you, based on a broader view of your reliability.

4.5G will be sweet.

What precentage of work fits this bill? 1%? 5%? If we have 5% of boxes does the job, I dont see it as a problem any more. Excluding one or two fast machines wont hurt. Beside, the different is 2x or 3x at max when comparing P4 to P4, not 60x when we talk about Pentium.

[Ian_H]

Originally Posted by ebx

4.5G will be sweet.

... it will be here sooner than we expect, along with SSE3, Prime95 v25 and complaints that double-checking everything below 33M is taking too long ;)

Originally Posted by ebx

Beside, the different is 2x or 3x at max when comparing P4 to P4, not 60x when we talk about Pentium.

Fair point. And I must apologise for my arithmetic, which was the cause of my underestimating my P4 1.7's performance - in my haste I'd failed to include 6+ P90 years credited for one high number tested.

[trif]

Originally Posted by Prime95

Prime95 does what maybeso suggests. The more your machine is off, the more your RollingAverage in local.ini decreases.

Actually, unless this has been changed in version 22, it doesn't. I normally run factoring under my own account. I installed a second copy of Prime95 in a separate directory in order to participate in the doublecheck gauntlet that TPR had going. My factoring copy didn't run for a month. When I fired it back up after the month was over, my rolling average remained as high as ever. Prime95 compensates for adjustments in how much CPU it is getting when it is running, but if it isn't running, it doesn't adjust.

[Deamiter]

Could it be that your computer was running but not the program? It sounded like George had it set to monitor CPU on time rather than just the amount of time that the program is running but I can't be sure.

[trif]

Originally Posted by Deamiter

Could it be that your computer was running but not the program? It sounded like George had it set to monitor CPU on time rather than just the amount of time that the program is running but I can't be sure.

Yes, but I also had a laptop that wasn't on for a few weeks, and there has been no adjustment of the rolling average. I had also installed Prime95 on my dad's computer, he only uses it sporadically. I didn't bother to try to estimate how many hours per day it would be on (it was going to be factoring anyway), because I figured Prime95 would estimate that better than I could. But it never adjusted, it was compensating for the CPU time that my dad was using for other things, but not for the time it was off.

[lycorn]

Ok, back to the poaching business...

Trif wrote:

There also needs to be in place a mechanism for handling trailing edge assignments that will not complete in a reasonable amount of time, such as contacting the person assigned the exponent. I think that most of these poachers think that if they don't poach then an exponent with a projected completion of 300 days out will just sit there holding up a milestone.

I completely agree with this statement. I am not in favour of poaching, neither have I ever done it, but I understand that it may be irresistible for some people to do so, instead of just sitting there watching a milestone sliding away due to Sxxxxxxx/Cxxxxxxx participants that may even have forgotten they are part of the GIMPS project. So I think those people should be contacted, in a polite manner, stating that they are holding a milestone back, and encouraging them to either release the exponent, or to make their best at speeding up the testing of at least that particular exponent. This strategy is obviously not 100% effective, but I think it would help. It could also be defined a maximum period of time, depending on the exponent range, that any user would be allowed to retain the exponent, regardless of reporting regularly or not. The user should be warned, say 1 month before that date, that the exponent was about to be reassigned to someone else. This should however be stated in the Terms and Conditions for using the program. But I think that definitely something has to be done to help moving up the trailing edge of the project while avoiding unethical poaching, which I agree is harmful to DC initiatives.

[MAD-ness]

I know that people get attached to thier exponents, but wouldn't it be feasible to just make sure that if you are properly assigned an exponent and someone else returns the result (regardless of why they return it or how they got it) and you finish it properly, you get credit for it?

I guess with first time tests, the prize money might be an issue, but again, why not officially credit the first person to be properly assigned and properly complete an exponent?

If someone wants to run an exponent that is holding up a milestone, they are going to do it anyways. Almost always thier goal is to speed up the milestone. If the server gets a good result then the project can move on, but whenever the person who was properly assigned the exponent submits it, they get full credit for it.

In a sense, they might feel like thier work was wasted, that is the only downside I can see.

However, people running these exponents and taking years are most likely not people who are highly active in GIMPS. They are people who fired it up and have left it running but forgotten about it, or people who reserved the exponent then stopped running GIMPS. As it is, if the exponent is poached, they REALLY end up with nothing, so the above mentioned option would actually improve thier situation.

I am not a GIMPS maniac though, so I might be missing something vital.

[aga]

Originally Posted by Prime95

Seriously, this discussion has been useful. I had not considered designing in proxies. I'm not sure how that would work or be implemented. It is true that one server can handle the current load and small outages are a nuisance. But wouldn't proxies serve as insurance against catastrophic failure?

Well, proxies is not the only possible approach. For example, traditional database replication might work well enough - if something drastic happens with master server, slave takes over domain after timeout, and only few last seconds of master activity might be lost. Another example, is mutually replicating servers. This requires much more efforts on database design and coding, but gives even better protection against failures of servers, ensuring that at any moment there is at least one server on internet that can be contacted by clients - and unlike proxies, such servers will be able to provide more efatures to project. More security issues should be kept in mind, tho.

To choose between which db/network architecture is better, one need to ask: how GIMPS should look like in year 2005? 2010? That's a worthy thing to discuss.

Originally Posted by Prime95

I agree with previous posters that there are two major hurdles to solve. One is cost. I think enough interest has been expressed here that the roughly $100 / month cost could be handled. Even an initial server outlay may not be a problem.

Considering that bandwidth requirements of server are very low (what's 30Gb/month? Nothing. But someone told it's 1Gb/month - ridiculously low traffic), I could provide 2 (and maybe 3) accounts on different servers to GIPMS. All servers reside in large data center which ensures excellent network and uptime. Of course, I will require that hosted software should be efficient enough to let other stuff on the servers run smoothly (which should not be much of issue - I currently run mprime on the servers testing exponents - after stopping it it will release a whole bunch of hardware resources).

I saw others were willing to provide servers in their homes - as long as they have fast enough internet connection, that can be used for GIMPS and add good overall value to project. See below.

Originally Posted by Prime95

The second hurdle is design and implementation. Coming up with a grand plan and implementing in stages seems reasonable.

Well, let let me throw in few more thoughts. First, primenet site should be broken into 3 parts:

- module that handles reserving and releasing exponents, as well as storing results.

This is the only module that requires replication. It can not be effectively developed by large community; but in fact, no need to - as someone noted, server-side stuff is not something extraordinary difficult. Working site might be ready for beta testing in less than a month, including planning stage (as alot of planning has already been done).

Also, this module does not need much CPU, RAM or HD resources.

If proxy and/or one of the replication approaches adopted, home computers might be used here even if their internet connection is not too reliable.

- module that maintains accounts. (adding accounts, forming/joinging teams, changing passwords, etc using web forms). No need to have hot replication of this, hot backup is sufficient - if this module is ever down for a couple of days, it will not hurt too much (but with backup, this module can be upped at another server in a matter of few hours).

- module that processes and shows stats.

This is where most hardware resources needed. Fortunatelly, it does not require absolute reliability - if it goes down for a couple of days, that's passable. Note that this module can be duplicated at many independent servers - every stats server just need a stream of updates from master servers. This also greatly reduce security concerns. That's where home computers with good internet connection would be most useful.

Now, about development of all that. I would like to volunteer to development of first 2 modules. I have the neccessary expertise: I'm involved with distributed projects since the old days of RC5-56 and OGR-23; specificially, participate within GIMPS/primenet for over 4 years already; have developed high-traffic server-side applications, specificially webcounter.com. I would prefer to form a small (2-3 persons) team that develop, maintain, and independently provide hosting (to ensure that new primenet never goes down even if some team member dies tomorrow). Development should be cathedral-style.

Regarding stats: providing just the very basic stuff like what currently entropia provides is no problem at all. But if someone wants advanced stats - I suggest that bazaar-style project should be laucnched. It will take alot of work, good stats is not a task one person could easily handle.

Another important module is code within GIMPS client software (prime95/mprime). I believe it's George realm currently. Fortunatelly, new primenet might use currently used HTTP-based protocol - or another HTTP-based protocol; that should not take too much efforts to implement.

If anyone willing to seriously discuss these things (seriously means participating in development and/or hosting hardware worldwide), please drop me a note at: gimps at supplehost dot com. If someone has already formed a team that has started to work, please let me know.

[IronBits]

I haven't been keeping up with this particular project and just now getting started...
I've have a Windows2000 Server, running for several years.
Fantastic up time, only goes down when I reboot it.
I'll offer up the hosting if someelse will offer up the programing et al...
Network is almost always up, went down twice in the past year, for a very short period of time DSLextreme.com...
Xyzzy can call me if you are serious about doing this...

[Prime95]

Originally Posted by aga

traditional database replication might work well enough

Now, about development of all that. I would like to volunteer to development of first 2 modules. I have the neccessary expertise: I'm involved with distributed projects since the old days of RC5-56 and OGR-23;

Another important module is code within GIMPS client software (prime95/mprime). I believe it's George realm currently. Fortunatelly, new primenet might use currently used HTTP-based protocol - or another HTTP-based protocol; that should not take too much efforts to implement.

If anyone willing to seriously discuss these things (seriously means participating in development and/or hosting hardware worldwide), please drop me a note at: gimps at supplehost dot com.

Thanks for offering to help!! As you can see from this thread we are still in the planning stages.

Are you at all familiar with postgresql or mysql? Do either of these perform replication? Does it work? Even if we don't use it now, it would be nice to have the feature available for future plans.

Eight or nine months ago I played with postgresql and it seemed to have all the SQL features needed. Commit, rollback, row level locking using "select for update of". I did not investigate its performance, backup/recovery capabilities, replication, etc.

Your 3 modules approach is a good logical division of work, but I still envision one database. They need to be tied together when we get a result:

Start transaction
Release the exponent from reserved list
Record the result in results database
Update user's and team's statistics
Commit

As you can see all three of your modules were involved within a single transaction.

The current prime95 client uses http and cgi-bin. We have the format of the cgi arguments so we can support old clients. I presume we would upgrade the v23 client to be more functional. Possibilities include:

Giving CPU credit more often (mid-work-unit)
After reporting a result, the server could return your current standing in the stats.
Client could report cpu type, speed, hours-per-day, rolling average to let server pick default work type.
P-1 factoring work units.
Give the user the ability to configure some client options from the server. For example, user could change the memory settings or default work type from a web form. Great for client computers that are not easily accessible.

Feel free to comment. I've been meaning to gather all the suggestions and issues raised in this thread into a MS word document for future planning.