I haven't been keeping up with this particular project and just now getting started...
I've have a Windows2000 Server, running for several years.
Fantastic up time, only goes down when I reboot it.
I'll offer up the hosting if someelse will offer up the programing et al...
Network is almost always up, went down twice in the past year, for a very short period of time DSLextreme.com...
Xyzzy can call me if you are serious about doing this...

[quote="aga"]traditional database replication might work well enough

Now, about development of all that. I would like to volunteer to development of first 2 modules. I have the neccessary expertise: I'm involved with distributed projects since the old days of RC5-56 and OGR-23;

Another important module is code within GIMPS client software (prime95/mprime). I believe it's George realm currently. Fortunatelly, new primenet might use currently used HTTP-based protocol - or another HTTP-based protocol; that should not take too much efforts to implement.

If anyone willing to seriously discuss these things (seriously means participating in development and/or hosting hardware worldwide), please drop me a note at: gimps at supplehost dot com.[/quote]

Thanks for offering to help!! As you can see from this thread we are still in the planning stages.

Are you at all familiar with postgresql or mysql? Do either of these perform replication? Does it work? Even if we don't use it now, it would be nice to have the feature available for future plans.

Eight or nine months ago I played with postgresql and it seemed to have all the SQL features needed. Commit, rollback, row level locking using "select for update of". I did not investigate its performance, backup/recovery capabilities, replication, etc.

Your 3 modules approach is a good logical division of work, but I still envision one database. They need to be tied together when we get a result:

Start transaction
Release the exponent from reserved list
Record the result in results database
Update user's and team's statistics
Commit

As you can see all three of your modules were involved within a single transaction.

The current prime95 client uses http and cgi-bin. We have the format of the cgi arguments so we can support old clients. I presume we would upgrade the v23 client to be more functional. Possibilities include:

Giving CPU credit more often (mid-work-unit)
After reporting a result, the server could return your current standing in the stats.
Client could report cpu type, speed, hours-per-day, rolling average to let server pick default work type.
P-1 factoring work units.
Give the user the ability to configure some client options from the server. For example, user could change the memory settings or default work type from a web form. Great for client computers that are not easily accessible.

Feel free to comment. I've been meaning to gather all the suggestions and issues raised in this thread into a MS word document for future planning.

I have another idea. If our server had the space and bandwidth, we could have the program send checkpoints. We wouldn't do this regularly, just, for example, every quarter of the way through a 33M, or 1/2 way through a first-time test. That way we'd lose less work if someone's hard-drive gets cleared. It would also send the checkpoint file if someone quit gimps or unreserved an exponent with some work on it, so the work wouldn't be lost. Because the checkpoints files are relatively large, we would make checkpoint sending optional (so people could keep the promise of low-bandwidth).

hmmmm.... think 31000 users all sending 2-3 times more data... that sounds like a potential nightmare compared to the few data losses from HD failure or the like. Seems to me that if you're THAT concerned about your data, you're probably a computer fanatic and losing one exponent from the 10+ on your farm REALLY won't kill you.

That said, it really isn't a bad idea, but I vote we stick to the simple method that works rather than trying to impliment a data backup that MIGHT save a few hundred P90 years a month...

[quote="Prime95"]
Are you at all familiar with postgresql or mysql? Do either of these perform replication? Does it work? Even if we don't use it now, it would be nice to have the feature available for future plans.
[/quote]

I can't say much about PostgreSQL; but I use MySQL for years, and extremely satisfied with reliability, performance, compliance to SQL standard, and extra nifty features of MySQL. Should be sufficient to say that yahoo, slashdot, webcounter all use MySQL (see MySQL
documentation for a loooooong list of popular web sites and services that use MySQL). Specificially about webcounter.com (as I trust my personal experience better than long lists provided by involved parties), MySQL is used for all db needs, and there were no reasons to ever regret about choosing MySQL.

In particular, MySQL provides replication for quite a while already, and the feature is very stable. It has some limitations tho (for example: one-way replication can be configured flexibly, but mutual replication works well only with 2 servers; if you add more,
replication performance and relibility seriously suffer). So I have thoughts doing custom replication, it might provide few useful features for multi-server configuration specificially for GIMPS. Not that I like reinventing wheel tho...

Any way, MySQL replication will (must!) be immediatelly used to perform remote realtime backups of the database. As I told, one-way replication is very robust, and can be configured in matter of few minutes.

BTW, MySQL documentation has a chapter titled 'How MySQL Compares to PostgreSQL' - I strongly recommend to read it. Overall, MySQL documentation have always been an excellent well-maintained resource that does not merely cover syntax and utilities in great details, but also discusses alot of other important issues. That's a big plus for long-term projects.

[quote="Prime95"]
Eight or nine months ago I played with postgresql and it seemed to have all the SQL features needed. Commit, rollback, row level locking using "select for update of". I did not investigate its performance, backup/recovery capabilities, replication, etc.
[/quote]

Well, transactions and row level locking are not really neccessary for PrimeNet - even considering that MySQL supports that. When we come down to database design we can discuss this in greater details if neccessary. To prepare for discussion, please read the chapter "MySQL Differences Compared to ANSI SQL92 - Transactions" within MySQL documentation.

[quote="Prime95"]
Your 3 modules approach is a good logical division of work, but I still envision one database. They need to be tied together when we get a result:

Start transaction
Release the exponent from reserved list
Record the result in results database
Update user's and team's statistics
Commit
[/quote]

No-no-no :) When server receives result, it should queue it (this queue might also act as a queue of replication events, if it will be decided to implement custom replication) and immediatelly let client go away. This is the only way to ensure that server can effectively
handle traffic spikes and be unsucceptable to most DoS attacks. As soon as event queued, background thread should be woken up, and it will:

- record result in results database (if it havn't been recorded yet - unique keys will help to make this fast);
- Release exponent (if it havn't been released yet - once again, single query does everything);
- Mark log event as processed or delete it altogether (again single query);
- Process next event, or go sleeping if there are no more unprocessed events.

That's an example how to go without transactions (which means go much much faster). Note that none of the actions require immediate processing - if something gets delayed by few minutes (which is very much for P4 server :) ) nothing wrong happens.

Handing out new exponent can be also done with just one event queue too; tho it will require both preprocessing and postprocessing of event (but programming codes and database structure will still be quite simple and straightforward).

Then, stats: it's excessive to maintain summary tables (absolute majority of stats tables will be just summary tables) within transaction. If something fails, summary tables can be always rebuilt (at least, database must be designed to make it possible). Also, if we speak about
some advanced stats more than just two running totals per account, such processing would take way too long if processed within request. Doing in-transaction stats processing would also make much more difficult to spread stats among few servers, and would notiecably increase traffic between core module and stats module.

Currently PrimeNet shows updated numbers immediatelly; it is not really necessary. If stats gets updated by background thread, delay of a couple of seconds will not hurt. Alternatively, stats could be processed in batches, say once per hour - that's what dnet (once per day) and webcounter (once per hour) do with their traffic. Batch processing usually simplifies stats code.

[quote="Prime95"]
The current prime95 client uses http and cgi-bin. We have the format of the cgi arguments so we can support old clients. I presume we would upgrade the v23 client to be more functional.
[/quote]

By the way, is there document that describes protocol between prime95/mprime and primenet server? I'd appreciate if I could look at it? In fact, we need to decide: do we keep old protocol and just add few features in it keeping protocol backward compatible? Or do we completely
rewrite protocol, making it more flexible extensible (XML based?) etc.

I see this problem with supporting old clients: they make connection to URL within entropia.com domain, yes? I believe entropia.com guys need their domain more than GIMPS. Tho if it were something like gimps.entropia.com, it could be moved around. Problem could potentially be worked around by acting the URL as http proxy to the new servers, or installing primenet2 coserver at entropia - in all cases entropia people would need to make certain efforts.

[quote="Prime95"]
Possibilities include:
[/quote]

I addressed these and few other issues at [url]http://gimps.supplehost.com/primenet2.html[/url] - this message is already long, and if I include the page here it would at least tripple message length. I will also try to provide a copy with this message as attachement but I'm not sure if it will be easily viewable.

(Oh well - the message board software does not allow me to upload html attachement - use the link above)

[quote="Prime95"]
I've been meaning to gather all the suggestions and issues raised in this thread into a MS word document for future planning.[/quote]

Please don't use MS Word file format - I will have problems reading it. At least, please export document to HTML (or plaintext, depending on text formatting you do).

[quote="aga"]
I addressed these and few other issues at [url]http://gimps.supplehost.com/primenet2.html[/url] - this message is already long, and if I include the page here it would at least tripple message length.[/quote]
:shock:

Feel free to clog this thread up... I personally find it easier if everything is in one spot, and the search function really helps me when I need to find something later on... I agree about avoiding MS Word documents... Plaintext is fine for me...

After 100+ posts, clogged is hardly a strong enough adjective!
Anyway, I was just wondering about your difficulties with MS Word since you can read a word file in just about any other WP program. Actually I ask many of my colleagues to send me word docs rather than plaintext simply because I know I will be able to read it, and in many cases people screw up formats they aren't familiar with (not to suggest that George would have trouble with it).

Admitted, a standardized plaintext is more usefull overall, but because of M$'s monopoly for so long, they've effectively standardized the WP market.

Here is plaintext version of what I referred to earlier as html page. This is how I envision things personally (and my opinion is influenced by previous posts in this forum); corrections/comments are solicited.


[b]Feature:[/b] Giving CPU credit more often (mid-work-unit)
[b]prime95/mprime/PrimeNet:[/b] Credit is given only at the end of particular stage.
[b]PrimeNet2:[/b]
Trial factoring might be divided into smaller blocks (with each block having same bits in tested potential factors?). This will break down factoring into small pieces that will be returned faster; it also allows several computers to factor the same exponent simultaneously. It will also allow to assign 'easy' factoring blocks to slow computers and new GIMPS users thus encouraging them, and tasky factiring like '68 to 69 bits' to more powerful computers that have been around for some time already and thus the user has alrady tired to check stats every hour :)

[AG] Not completely in topic, but: few years ago I made java applet that allows to do factoring with java-enabled browser; because of java having specialized API for manipulation with large numbers, the performance was pretty good. That java applet did realtime subdivision and reassembling of factoring work into/from very small pieces. I might revive the idea eventually... It sounds good to create a page with such java applet that states 'you are already helping GIMPS... download native software for your specific operation system/hardware to do the math work faster'.

Can P-1 factoring be broken down into small blocks? Looks like yes (varying B1 and B2), but should it be done (or will it affect overall GIMPS performance)? [George, please advise]

For LL testing only speculative crediting might be done. I strongly believe it should not be handled within core module, but instead stats module should 'average out' values. Doing that within stats module is straightforward and not error prone.

Since LL intermediate files are large, and only increase in size as GIMPS advances to large exponents, breaking LL tests down to blocks is not reasonable. At least not until someone is willing to pay dollars for huge amounts of bandwidth.


[b]Feature:[/b] After reporting a result, the server could return your current standing in the stats.
[b]prime95/mprime/PrimeNet:[/b] Ranks available only at server.
[b]PrimeNet2:[/b]
I strongly advice to avoid bloating client-side software with features like that. Those bits of code are almost impossible to upgrade (how many v16 clients are still running?); it complicates core server module making it more error prone, and more prone for DoS attacks; and so on.

Nevertheless, client-side software can use an additional connection method - not to core module but to stats module - which it can query periodically, and not only show position, but also show a nifty table what every computer in the account performs, and so on. But is that really neccessary? Everyone has web browser - client software might just open browser window for curious user.


[b]Feature:[/b] Client could report cpu type, speed, hours-per-day, rolling average to let server pick default work type.
[b]prime95/mprime/PrimeNet:[/b] At least, PrimeNet shows CPU types and frequencies. Tho there are some bugs (initially my 3 P4 computers were shown as 'Unknown', now two of them spontaneously changed to P-III). Anyway, as far as I know PrineNet does not use the information to make decisions.
[b]PrimeNet2:[/b]
Good estimation of 'speed index' of each computer is definitelly a valuable information. PrimeNet2 should allow users to explicitly select which types of work they prefer or dislike; but default behaviour should be very flexible; it also should be implemented on server, not hardcoded into hard-to-upgrade clients.

BTW, if server knows amount of RAM available, it will be able to avoid assigning too large exponents and provide factoring work for low-RAM computers when all small LL tests have already been assigned/done.


[b]Feature:[/b] P-1 factoring work units.
[b]prime95/mprime/PrimeNet:[/b] Accepts P-1 factoring result messages, and provides client with information if newly reserved exponent have already passed P-1 factoring step or not. P-1 factoring CPU time is not credited (?) and P-1 factoring status is not displayed anywhere.
[b]PrimeNet2:[/b]
Database structure should be flexible enough to allow adding more types of work units easily. For example, why don't to start doing P+1 factoring, the code is already in prime95/mprime? Or someone might discover a new great factoring algorithm tomorrow.

Specificially about P-1 work units, I have this concern: the algorithm requires alot of RAM. Thus low-end computers are probably excluded. Faster computers would likely prefer to do only LL awaiting for someone else to do P-1 - and who will do only P-1 if it requires even more resources than LL, but can not find prime? Other words, with this particular algorithm it might be better to enforce every assignee to do both P-1 and LL if they want to do LL. (This is how it currently goes)

One idea might be to improve client so that if worktodo contains both P-1 and LL work, then P-1 is done during night and LL goes on during days; or something like that. I can assume that there are alot computers out there that have hundreds Mb spare RAM at night but users don't like to give more RAM than sufficient for LL during daytime.

How small P-1 blocks could be done? For example, it is possible to create (ridiciously small) 1-minute P-1 blocks, or there is some long pre-postprocessing even for tiny blocks? Would such tiny blocks be suitable for processing at computers that usually do trial factoring (presumably, low-RAM computers)? Or amount of RAM required for P-1 does not depend on P-1 boundaries?


[b]Feature:[/b] Give the user the ability to configure some client options from the server. For example, user could change the memory settings or default work type from a web form. Great for client computers that are not easily accessible.
[b]prime95/mprime/PrimeNet:[/b] Not supported.
[b]PrimeNet2:[/b]
That's what account module should do, among other things. Since the information should be submitted to clients eventually, the accounts module should be tightly integrated with core module.

But one should be VERY careful with features like that. Choosing preferred type of work on server is probably fine; choosing amount of RAM used by client is NOT. It will cause great number of concerns from GIMPS users, I'm sure - noone wants to ever get prime95/mprime using 1.5Gb of virtual RAM (and thus swapping computer down to death) due to bug or DNS spoofing attack or whatever.

Network-related notes: the settings on the server will not take immediate effect (client should at least check in with the server once again, and/or finish currently running work). Also, this feature precludes project from using proxies (at least, this functionality will make proxy protocol quite more complicated).


[b]Feature:[/b] Exponents pouching
[b]prime95/mprime/PrimeNet:[/b] Readily prefers results from poucher to work being done by 'legal' assignee.
[b]PrimeNet2:[/b]
Pouching should not be credited. It might be considered as tripple-checking, but no credits of any kind are given to such tests.

Why tripple-checking, not double-checking? Currently, poucher may maliciously pouch twice with the same exponent, using 2 different accounts making GIMPS to skip an exponent. Other words, results from pouchers should never be trusted. On the other hand, it's virtually impossible that server assigns the same exponent for checking and doublechecking for 2 -distinct- malicious users that report the same residue - we all hope that absolute majority of GIMPS users don't forge results and don't use faulty hardware.


[b]Feature:[/b] Milestone showstoppers
[b]prime95/mprime/PrimeNet:[/b] Are not handled specificially.
[b]PrimeNet2:[/b]
It is proposed to assign each GIMPS computer a 'reliability level'. New GIMPS computers get zero value; as they process and return exponents the level increases first quickly then slower. Fast computers get additional boost. Computers that ever had timed out exponents get their reliability level noticeably descreased. Computers that ever returned faulty or forged result drop a mile below zero. Reliability levels of other computers in the same account affect reliability level of the new computer. Highest reliability level value can be assigned only manually.

The exponents do not really have any special value - they have the similar chance to find new mersenne prime as other exponents. But humans are just humans... and do like milestones. So smaller exponents are assigned only to computers with high reliability level.

The formula used to compute reliability level should not be too discriminative; probably, computers with very low reliability level (new computers assigned to GIMPS) should be assigned only trial factoring, to ensure that new users get their stats updated often. :) Unless they explicitly set their preferences, of course.


[b]Feature:[/b] Server failsafety
[b]prime95/mprime/PrimeNet:[/b] None.
[b]PrimeNet2:[/b]
Either proxies or replicated servers should be used. If possible, design should allow both: replicated servers serve usual users, and proxies might be greatly appreciated by those who run large computer farms behind firewalls with strict rules.


[b]Feature:[/b] Server-side technologies
[b]prime95/mprime/PrimeNet:[/b] Windows NT server, with [resource-ineffective] cgi scripts and [some] SQL database.
[b]PrimeNet2:[/b]
Proposed is JSP/Servlets server-side technology, with MySQL as backend. POSIX compliant server-side environment is strongly recommended; but choice of Java+MySQL completely avoids portability problems leaving only security and reliability issues to consider. While desktop-oriented operation systems should be avoided, they should still be capable of running PrimeNet2 codes if ever deeemed neccessary (in case of drastic outage, whatever).


[b]Feature:[/b] Database architecture
[b]prime95/mprime/PrimeNet:[/b] Monolitic transaction-oriented (?) SQL database. Database is not authoritative; master database is maintained by George.
[b]PrimeNet2:[/b]
Three modules are proposed:

- core module - serves client-side software enqueries (exponents reservations/relases/status and minimal amount of stuff aside that - to reach best possible performance, reliability and security);
- accounts and teams module - provides web-based interface for the database;
- stats module.

Ideally, to improve flexibility reliability and security all modules should have independent databases. To satisfy practical requirements, first 2 modules need to share a chunk of the database. (Another option to consider is that database of second module is replicated to first module; but this somehow limits possible features).

Stats module needs only read access to part of the database (but might use several temporary and/or summary tables); it sounds reasonable to keep stats database separately (ideally, at different hardware) by replicating relevant data from core module. This makes it impossible for stats module to advertisely affect core database, thus decreasing security concerns.

[AG] Transactions are not needed; moreover, should be avoided.
More discussion on this when we start architecting database structure.

PrimeNet2 should ideally keep master copy of GIMPS database (to provide George with more spare time to speed prime95/mprime code ;) ). (This also means that there should be a feature to release/return large chunks of exponents that are coordinated elsewhere). Whichever server-side configuration is choosen, realtime backups of the database should go to few locations around the world to ensure data safety. At first stage, PrimeNet2 database should be just assigned large (huge) chunk of exponents from the database maintained by George, with existing data imported later without haste.


[b]Feature:[/b] Authentification
[b]prime95/mprime/PrimeNet:[/b] Accounts are protected with passwords.
[b]PrimeNet2:[/b]
Accounts should keep having passwords, of course. Additionally proposed to introduce [randomly generated?] per-computer passwords. This is to avoid keping account passwords on potentially untrusted computers. Account password will be neccessary for tasks like instalation of new computer (and adding it to an existing account), browsing account stats, changing preferenses at web site, etc but a particular computer may easily go on unattended with only per-computer password.

United Devices used a web-based "configuration program" for their clients.
I strongly disliked it, and it seems to cause a LOT of problems with newbies during their too-frequent outages.

I like the "send checkpoints" idea, *if*
1) The bandwidth needed isn't too excessive
2) Poaching of checkpoints is somehow prohibited in the software
3) It's made an optional capability

Tricky part is when a checkpoint is sent, then the machine it's from dies....

Regarding remotely checkpointing LL tests (as well as P-1). That will require RAID with bunch of drives, stream of electricity, U4 case (maybe even U6), and quite a chunk of bandwidth. Rough math shows that resulting expenses will be 3 to 5 cents per checkpoint (plus large setup fees), multiply by number of exponents GIMPS handle... Anyone's willing to pay? :) No? :( Ah, no... :mad: Well, then let's close the topic.

[quote="aga"]When server receives result, it should queue it (this queue might also act as a queue of replication events, if it will be decided to implement custom replication) and immediatelly let client go away. This is the only way to ensure that server can effectively
handle traffic spikes and be unsucceptable to most DoS attacks. As soon as event queued, background thread should be woken up, and it will:

- record result in results database (if it havn't been recorded yet - unique keys will help to make this fast);
- Release exponent (if it havn't been released yet - once again, single query does everything);
- Mark log event as processed or delete it altogether (again single query);
- Process next event, or go sleeping if there are no more unprocessed events.

That's an example how to go without transactions (which means go much much faster). Note that none of the actions require immediate processing[/quote]

Queueing work for better responsiveness is a fine idea, but I still want transactions. Transactions are used to make sure the database never gets into an inconsistent state. In your example, if the server crashes after recording the result but before releasing the exponent, then the database is left in an inconsistent state. Now admittedly, upon rebooting and processing the queue again we can write code to detect that the result has already been reported. However, there are some operations where we cannot detect if they occurred before the system crash.

Transactions have a performance penalty but they do provide a benefit.

I briefly started reading some MySQL docs and see that transactions are supported. I think I saw that Oracle-style consistent read is also supported (a cool feature when generating reports). Subqueries are not (*sigh*), I use them a lot in generating the status.htm page now.

[quote="Prime95"]upon rebooting and processing the queue again we can write code to detect that the result has already been reported.[/quote]

Well, this is the whole idea (plus the fact that we can just let SQL engine to do the work using unique index and avoid cluttering source code with numerous checks)... if we can process events in a way a consistant way (and we can!) than SQL engine provided transactions are excessive, then we should either use transactions and not events, or events and not transactions.

[quote="Prime95"]However, there are some operations where we cannot detect if they occurred before the system crash.[/quote]

Which specificially? :) Well, as you might have noticed, I envision that core module (two first modules) is separated from stats processing. If stats processing performed by the same module, it often requires transactions (other consistent ways of process data make programming code complicated). If we keep core module and stats separated, then core module might go without transactional tables and stats module use transactions, that's reasonable architecture. (But then again: since only replicator writes to source tables in stats module, transactions for read only operations might be excessive).

[quote="Prime95"]Transactions have a performance penalty but they do provide a benefit.[/quote]

They also provide problems. Let's see specificially:

If we adopt/use MySQL replication model for servers that serve requests simultaneously, than we face problem that can be generally described like this: order of transactions at slave server (i.e. at all servers) is not defined. This is going to cause problems in few scenarios, for example:

-- server A accepts result of LL test, and makes neccessary transaction
-- server B decides that the assignment has expired, and makes neccessary transaction
---- even worse if it quickly reassigns the exponent to someone else
-- now servers try to cross replicate transactions and both fail as preconditions at slave differ from master.
---- if something like this happens, MySQL stops replication threads at all servers awaiting for manual intervention :(
-- It can happen that transaction is not getting replicated to another server. Chance is awfully small but non-zero (all scenarious involve server crash)
---- it results from the fact that MySQL uses replication log that's independent of transaction engine and it can not really rollback transaction from replication log
---- MySQL could also have transaction replicated only partially...
------ I believe it's fixed in recent MySQL 3.23.52
------ Don't know status of this issue in MySQL 4.x
------ Tiny chance of skipping transaction at slave still stays

If we use different SQL engine that supports distributed transactions, then what happens if servers temporarily lose route between each other but still serve internet traffic successfully?

-- both servers get hundreds threads stuck awaiting for transactions to start?
-- each server assumes that it's the only server survived, and now we have two competing servers?
---- after servers see each other again, one server dies together with some activity it processed during the partial outage? Otherwise we get transaction ordering problem.

Ok, we could delay timing out by 24 hours to give it neccessary time marging... but similar problem happens if we assign the same exponent simultaneously... if we solely rely on transactions, it is not sufficient; or otherwise we can not really have remote servers working cooperatively.

So, since transaction would work well only with single-server configuration, I (see the beginning of the message) consider events queue as a better approach for this particular task.

Giving the topic more thoughts, looks like GIMPS have 2 options:

-- Use single main server, and few proxies to guard clients against server outages;
---- isn't single server is what we are trying to avoid?
---- proxies raise a whole bunch of authentification issues
---- certain features are hard-to-proxy (if at all possible)
-- Use event-based approach, and several cooperating servers (that presumably can't go down all at once).
---- hard to invent anything beter then this from failsafety point of view

Later today I plan to put here larger text regarding proposed events model.

I don't believe we're trying to avoid having one "master" server - nor do I think we need to. Splitting the web and report stuff out might make sense, but the actual "I deliver the next item to be tested and keep track of what you sent back" server doesn't create that much of a load.

Not sure if we really need any "proxy" servers - GIMP doesn't soak any where near the transaction load of a small-block project like Distributed.net -and they managed with a single "master" server, with fairly few problems - despite a LOT more "blocks" of work per day done per client (ballpark TEN THOUSAND more for a conservative estimate vs. LL work, closer to *a* thousand on factoring) and a LOT more clients active in a given day. I think having a "proxy" server might be nice for when the master needs to be down for maintainance - but even that would be redundant if the client is set up to keep a week or two more "work on hand" than it's check-in interval.

I suspect the data load for each block on transfer was quite a bit larger too for downloading, but somewhat smaller on average for uploading results - but don't know what George sends and recieves out of Prime95, so I am just guessing there.

Has anyone looked at boinc as a solution?

http://boinc.sf.net

Is anyone here?

[quote="barcode"]Is anyone here?[/quote]

At least me. But I see ~zero interest on the topic. Entropia server was down at least two more weekends after the last posts, but looks like noone cares.

So currently I'm in mood to just watch how the situation will evolve, maybe people will recover from holidays and pay more attention to the issue, or maybe replacing the entropia.com will become so important that I will be able to stop awaiting for opinons and ideas and go and implement my own. The offer for help I earlier worded still stays in effect.

After all, [re]implementing straightforward non-scaleable entropia-like server is an easy task and can be done and installed into a hosting account within 48 hours if real need arises. But really interesting, and I strongly believe most useful for GIMPS project in long-term, would be to create a flexible scaleable extendable solution. In fact, I don't care too much if anyone else interested in participating in the work, my experience allows to effectively handle such programming projects alone - but the I would definitelly like to have some assurance that after software matures enough to contain 'this, this and this' features, and passes testing, it will be factualy used. And I would really hate spending a piece of my life doing work that is not going to be appreciated (or used, which is in the end the same thing).

And so far, there is nothing like 'this, this and this' coordinated list exist, as if noone cares... here my message recursively repeats.

I'd like to hear what Scott Kurowski has to say about all of this... It is his baby after all... I've tried to send him email several times but the emails bounce back... I'd also like to have Brad Bernard chime in with his thoughts...

I care but I'm just too busy until Spring. Replacing the server will be my top priority for 2003.

Aga, I appreciate the offer of help and hope we can work together in the Spring. I agree that throwing something together real fast would be a mistake. If we're going to do this we'll need to do it right.

Great! The Spring matches my own plans very well.

And I believe we will start with compiling stuff in this forum into a single document. So everyone's advised posting ideas and suggestions here without waiting for a particular date.

The key idea, as I see it: using multiserver configuration and/or proxies is not for spreading traffic or servers load (but eventually this might become important, too); instead, the primary issue is to ensure that GIMPS project does not wholly depend on a particular server or service, that there is no a single point of failure that can affect entire GIMPS. This is well in sync with the initial George's posting that started this topic.

Well I would like to offer up my time, skills and experience to the process. I have been working as a Architect and Developer for the past 6 years mostly within the Internet and distributed programing arena. My key areas of expertise as effects this project would be, I imagine: PostgreSQL, MySQL, Apache, Linux. My main skill however is Java development, what language are we thinking of writting the server application in?

[quote="aga"]The key idea, as I see it: using multiserver configuration and/or proxies is not for spreading traffic or servers load (but eventually this might become important, too); instead, the primary issue is to ensure that GIMPS project does not wholly depend on a particular server or service, that there is no a single point of failure that can affect entire GIMPS.[/quote]

Using a single server does not imply having a single point of failure which could effect the entire project. If the server is sufficiently generic -- built out of standard hardware, software (Apache etc) and a few easily installable custom scripts -- then it would be possible to replace it quickly if necessary. Given a few days of buffering on the part of client machines, even the complete failure of the server could occur without harm, as a new server could be set up before the client machines ran out of work.

Of course, this requires that regular backups are taken, but that's a much easier task than setting up several machines to serve in parallel.

[quote="barcode"]what language are we thinking of writting the server application in?[/quote]

My personal preference, and area of professional everyday job for the last years, is JSP/Servlets + MySQL + Linux (Slackware). This is a really nice stuff from performance, reliability, features/libraries, security, and portability standpoints.

Thank you for your offer, it sounds like that there is already a development/maintenance team. :) Good.

(We just need to figure out what we are exactly going to do :D )

[quote="cperciva"]Using a single server does not imply having a single point of failure which could effect the entire project.[/quote]
While that's a reasonable note, I disagree. Factually, it all depends on how we define 'failure' (and how we define 'harm' later). If hundreds users/computers are unable to get/return exponents and there are idle computers around, I think that's failure; do you? If such failure happens and GIMPSers are to wait days before the person(s) who can fix stuff will get down to it, that's even worse.

[quote="cperciva"]If the server is sufficiently generic -- built out of standard hardware, software (Apache etc) and a few easily installable custom scripts -- then it would be possible to replace it quickly if necessary.[/quote]
If you are a well-funded business with a non-mission critical resource, yes. For GIMPS, who will pay for the 'standard hardware' when rush replacement becomes neccessary, and who will ensure that provided system software will suit the task best? Who will provide the quick turnaround service, for free and in long term?

Keeping a set of homogeneous servers is anyway a solution that provides better availability, better performance, better resists to DoS attacks, and best use of available bandwidth (if client-side software first tries to access topologically closest server). More tasking to create neccessary software? Yes, but the differene is not too large, and maintenance expenses are greatly reduced, both in terms of human job and hosting hardware.

[quote="cperciva"]Of course, this requires that regular backups are taken, but that's a much easier task than setting up several machines to serve in parallel.[/quote]
As MySQL allows realtime replication, that is the most natural feature to use; it ought to be used notwithstanding if there is a single server or a dozen.

[quote]As MySQL allows realtime replication, that is the most natural feature to use; it ought to be used notwithstanding if there is a single server or a dozen.[/quote]

I think PostgreSQL would be a better choice as it allows cross table transactions and sub-queries which are both very very useful things that MySQL doesn't allow.

I not certain why we need real time replication. Could you elaborate?

[quote="aga"]If hundreds users/computers are unable to get/return exponents and there are idle computers around, I think that's failure; do you?[/quote]

Approximately, yes.

[quote][quote="cperciva"]If the server is sufficiently generic -- built out of standard hardware, software (Apache etc) and a few easily installable custom scripts -- then it would be possible to replace it quickly if necessary.[/quote]
If you are a well-funded business with a non-mission critical resource, yes. [/quote]

I must be missing something here. How would running several servers all the time be cheaper than running one and replacing it if it fails?

[quote]Keeping a set of homogeneous servers is anyway a solution that provides better availability, better performance, better resists to DoS attacks, and best use of available bandwidth (if client-sde siftware first tries to access topologically closest server).[/quote]

The server load should be low anyway -- assigning exponents is not difficult. The bandwidth is insignificant.

[quote]As MySQL allows realtime replication, that is the most natural feature to use; it ought to be used notwithstanding if there is a single server or a dozen.[/quote]

IIRC, George's background is database programming. I think he'd be happier using a database.

[quote]I must be missing something here. How would running several servers all the time be cheaper than running one and replacing it if it fails?[/quote]
GIMPS server is going to be hosted at hosting resources donated by GIMPSers. If we run several servers, then we always know how many active servers we get, and know detailed stats regarding reliability of each one. If one goes down, there is sufficient time (weeks, even months) to find a replacement, install stuff there, test it, and put into production. Note that as there is no need for excellent availability for each partiicular server, GIMPS could effectively use less-then-perfect hosting solutions like old Pentium server behind a home internet connection.

If instead a single server fails, what do we do? Might sound like pre-building a list of those who are willing to donate hosting resources is the solution; in fact, it is not. When server goes down and quick replacement is needed, expect that some people will not answer (at least, will not answer timely), some will tell 'sorry, the offer is no longer available, things changed over here', and even if there are a suitable offer left, it might be that at last moment some hardware or system software problems/incompatibilities will be dicovered, and you GIMPS can not afford purchagsing commercial dedicated server/clolocation, project is stuck. Even if there is a perfect substitution server discovered, it means that someone will need to rush and restore/install everything. We very hope the person will not be on Hawaii. But any way, do you remember that the work is not-paid? There no warranty that a single server will be reinstalled fast!

[quote]The server load should be low anyway -- assigning exponents is not difficult. The bandwidth is insignificant.[/quote]
Looks like bandwidth will be pretty low - usually, and that's good. It's still might be good if European GIMPSers use server in Europe, that gives noticeably better roundtrip and thus less affected by network problems.

But if someone decides to flood GIMPS server, the bandwidth usage can grow very high! If we run several servers, then effectively flooding all servers at once sounds quite unrealistic.

As another idea, I'd like to have stuff designed in a way that more math algorithms can be handled; maybe few that require much higher bandwidth than currently used algorithms. Why don't to start with a state-of-the-art design of the server-side stuff, instead of first cloning entropia and then throw the work away and redoing it from scratch again? After all, entropia server still works, there is no immediate rush.

But that's bandwith. I remember that someone told that the current dual-CPU Entropia server often gets 100% (i.e. 200%) continuous CPU load. While I'm not completely understand why does it happen (maybe poor database design or SQL engine performance, or maybe flooding of stats requests), I would prefer to have configuration where several servers handle the load cooperatively, and if something gets misestimated or GIMPS starts growing fast the problem could be quickly solved by upping an additional server, without need to rush and recode software (or spend thousands bucks for high-end server).

[quote]IIRC, George's background is database programming. I think he'd be happier using a database.[/quote]
Not sure I understand - MySQL is SQL based database engine. Noone is really going to write own database engine, there are better means wasting life.

[quote="aga"]Not sure I understand - MySQL is SQL based database engine.[/quote]

No. MySQL is an SQL based file system. It isn't a database.

(Of course, databases aren't always necessary, and for such tasks, MySQL performs perfectly well. But I don't think this is such a task.)

[quote="cperciva"]No. MySQL is an SQL based file system. It isn't a database.[/quote]

Sorry? Please read MySQL documentation, especially innodb tables handler section. Not that innodb features are needed for the task tho, and it is not innodb that defines software category MySQL falls in. So I understand you even less.

But if you are simply trying to start a next religious war, I will let you to fight down yourself. :devil: As it is not going to have a practical effect, as all the previous religious wars.

MySQL vs PostgreSQL
 

Do we need to start a new thread on this?

I'm not an expert on either of these databases. I did install PostgreSQL here a year ago and satisfied myself that it had all the features I need to implement the reservation system and results logging.

It seems to me that there is a lot more development effort and interest in MySQL than in PostgreSQL. InnoDB has added transactions and row-level locking to MySQL. Subqueries are scheduled for implementation in the near future. If we have outer-joins or views we can probably work around this limitation.

We should develop the list of features we want/require of our database and do some research into which best meets our needs.

[quote="aga"]Keeping a set of homogeneous servers is anyway a solution that provides better availability, better performance, better resists to DoS attacks, and best use of available bandwidth[/quote]

I like the idea of multiple servers if it doesn't put us on the bleeding edge and does not increase costs significantly. My biggest fear with the single server approach is the down time associated with a hardware failure. If the server is remotely located it could take days to get it fixed and the software restored from backup. Obviously, this isn't a fatal problem to GIMPS but it is a major annoyance to users.

Multiple servers has other problems to overcome. More development time, qualifying server operators as interested enough to keep the server upgraded/maintained for years to come, etc.

MySQL is probably the second or third most common SQL database system in use today (behind Oracle and probably DB2, might be 4'th behind Sybase).


I'm looking at a fairly high probability of moving sometime next summer, but I'd be willing to host a server after that if it is decided to go the multi-server route. Remote access won't be a factor unless I'm on vacation - I tend to be around every day.

9-)

Two things. First, I agree with the multiple server part (I think I was talking about it earlier in this thread). I would be willing to host one of the servers on my home connection. Secondly, I believe the reason the server gets maxed out is because it hosts the stats. If we had a multi server design, I think we could get away with one main server doing the stats (because they are less important then saving results and assigning exponents).

Re: MySQL vs PostgreSQL
 

[quote="Prime95"]We should develop the list of features we want/require of our database and do some research into which best meets our needs.[/quote]

To begin with, we should agree upon services GIMPS server/cluster will provide for GIMPSers. That's business layer, and database is just technical, backend layer.

One of the biggest question that will influence many technical details, will GIMPS server keep entire database regarding Mersenne numbers (and thus automatically advance exponents to P-1 queue after trial factoring run ends, to LL tests after P-1 factoring step passes, then to doublecheck queue, if neccessary to tripplecheck, automatically build the http://www.mersenne.org/status.htm table, and so on),

or should it just track subset of assigned exponents, and as soon as exponent gets checked, it gets exported from the onilne database, and then go to the offline database you currently maintain; then you potentially can assign the exponent back to GIMPS server if need arises?

I can think of pros and contras with both approaches; but as you already handle the GIMPS database for years, I'd definitely like to hear your opinion on this.

[quote="Prime95"]I like the idea of multiple servers if it doesn't put us on the bleeding edge and does not increase costs significantly.[/quote]
Well, getting to the bleeding edge of technology can not hurt, will give extra publicity for GIMPS and attract more people to participate.

There is also this issue: if we use... well... Redundant Array of Inexpensive Servers :) then with server-side software there is no need to imlpement any dirty performance optimization tricks, we can keep it academically clean and easy-to-maintain; and performance problems can be solved by simply adding/upgrading servers. That is assuming that interserver comminication will depend linearly or at most nlogn on total GIMPS traffic - according to my preliminary research this should not be a problem at all.

If instead we limit ourselves to a single-server solution (and unlikely the server will be high-end), then it might be essential to do do things like inJVM datacaching and other tricks that make software looking like a can of worms.

I'm surprised regarding the 'increase costs'. I didn't expect that development efforts will receive monetary compensation above US$0.02. :) And as long as we can effectively use low-end servlers, it looks like there are enough people willing to donate some resources to GIMPS.

[quote="Prime95"]Multiple servers has other problems to overcome. More development time, qualifying server operators as interested enough to keep the server upgraded/maintained for years to come, etc.[/quote]
Development time should not be too high.

If it becomes impossible to maintain a particular server, we just exclude it from RAIS and let it die off. Deploying web application onto a new server should be pretty simple, by adding the new neighbour to alive servers, uploading .war file to the new server, specifying server id and db login within configuration files, and prepopulating the database with a snapshot taken from some other alive server - most of that can be automated.

as a side note, I'd be willing to contribute MY US$0.02 to the cause! :(

[quote="aga"]Well, getting to the bleeding edge of technology can not hurt...[/quote]

It can if you're using new database features and all the bugs haven't been found!

While an array of redundant servers is great in theory (and can even reduce costs), I'm not familiar how MySQL implements this. Can you lead us through an example of how reservations are coordinated among several servers? Does each get a block of 1000 to handle independently or do the multiple servers coordinate to always assign the smallest available exponent?

I thought their would be me one main server that keeps the master database while the other servers report back to it. The lesser servers would contact the main server requesting like 1000 exponents. Everyday, the server would contact the main server, report back updated status reports, send returned exponents, and ask for some new exponents to bring the total back up to 1000. That is my opinion for how it should work. My $0.02USD.

I'm interested in how this will work too. There are any number of ways of doing it... It depends really on what our requirements our. How much do we want to micromanage the exponent allocation. Do we want to be able to prove numbers are the n'th Mersenne in the shortest time possible at the expense of say redundancy.

With the method of allocating 1000 exp. to each server and reporting back each day then the stats will always be up to 24 hours out of date. How about each server caching 1000 and each time it assigns one it can pick up another from the master server (and report the completed exp. back if necessary).

Now the question arises on how we assign clients to servers and how we update clients to know about new servers that join the network and old ones that leave. Not difficult but it needs some thought.

Trying out the forum
 

Hi all, Just read most of this thread. I've been talking with Entropia about PrimeNet and changes may be on the way. I'm giving the open source approach a lot of thought, and I think it's possible. More later. -sjk

Re: Trying out the forum
 

[quote="Old man PrimeNet"]Hi all, Just read most of this thread. I've been talking with Entropia about PrimeNet and changes may be on the way. I'm giving the open source approach a lot of thought, and I think it's possible. More later. -sjk[/quote]

Uhh, this may seem impolite, but... who are you? (maybe it's obvious to everyone else, but I'm suffering from both having had wisdom teeth taken out and having had several blood tests taken recently, and my brain isn't functioning perfectly right now)

[quote="adpowers"]I thought their would be me one main server that keeps the master database while the other servers report back to it. The lesser servers would contact the main server requesting like 1000 exponents. Everyday, the server would contact the main server, report back updated status reports, send returned exponents, and ask for some new exponents to bring the total back up to 1000.[/quote]
This sounds like proxies. Some GIMPSers might find them useful (first of all, the top producers who keep the computers within a single location), but proxies look like an optional low-priority addition, not replacement for multi-servers configuration.

Cpervica,
I'd be willing to bet that "Old man PrimeNet " is Scott Kurowski!
Joe O.

[quote="aga"]
This sounds like proxies. Some GIMPSers might find them useful (first of all, the top producers who keep the computers within a single location), but proxies look like an optional low-priority addition, not replacement for multi-servers configuration.[/quote]
Don't knock proxies! Distributed Net used 18 - 21 proxies in addition to its master keyserver very effectively. They did also provided "personal proxies" for the "top producers" and teams. These were just stripped down versions of their proxies so it wasn't that much more work for them. In GIMPS terms, the "personal proxies" could be limited to 100 exponents instead of 1000 for example. Properly done, this could significantly reduce the network traffic, as well as provide a buffer for master server outages.

[quote="Joe O"]Don't knock proxies! Distributed Net used 18 - 21 proxies in addition to its master keyserver very effectively.[/quote] GIMPS isn't distributed.net.[quote]Properly done, this could significantly reduce the network traffic[/quote] ... which is already insignificant ... [quote]as well as provide a buffer for master server outages.[/quote] ... which is already done by clients.

[quote="cperciva"]GIMPS isn't distributed.net.[quote]
That's true, Distributed.Net was/is a significantly larger project in terms of participation! We can learn from them.

[quote="barcode"]I'm interested in how this will work too. There are any number of ways of doing it... It depends really on what our requirements our.[/quote]
Yes, we really don't need diving down to the gore details of a particular implementation immediatelly, but start from upper levels.

How much do we want to micromanage the exponent allocation. Do we want to be able to prove numbers are the n'th Mersenne in the shortest time possible at the expense of say redundancy.

[quote="barcode"]With the method of allocating 1000 exp. to each server and reporting back each day then the stats will always be up to 24 hours out of date. How about each server caching 1000 and each time it assigns one it can pick up another from the master server (and report the completed exp. back if necessary).[/quote]

Yes, this sounds closer to the ideal solution. But I intend to completely avoid 'master server' and make servers truly symmetric. RAIS should satisfy the following requirements:

- it does not matter which server a particular client connects to. Algorithms should be designed in a way to minimize (ideally, avoid) interrequests dependencies.

- nevertheless, under normal conditions after server handles transaction for a particular client, the transaction should become visible on neightbour servers within few seconds. If some server goes down, we don't really want it to carry away 24 hours worth of work.

- any server should be able to continue serving network traffic even if it finds that it's alone survived. But: if a single server falls off from network, it should shut down. Will be shutdown treated as temporary (allowing it to catch up later) or final (will require taking new database snapshot onto the server) will depend on feedback received from other servers when the single server eventually manages to connect somewhere. 'THe only survived' and 'isolated alone' is just a light difference. I think, we could use this metric: 'if server can connect DNS servers (that host a particular domain, or root DNS servers), then it continues working awaiting for other servers to become reachable to synchronize database; if it can not get to DNS servers, it assumes that it got disconnected from internet and should suspend or cease operations'. This is still not final (for example, how to treat situation if only part of DNS servers are accessible?) but generally looks good - DNS is inavoidable entity, and using it as a global coordination point is very natural.

- it is not really needed that RAIS assigns lowest exponent with exponent reservation transaction. It's sufficient that (1) the assigned exponent is not noticeably larger than mimimal unassigned; and (2) minimal exponents does not stay inassigned indefinitelly long. In fact, there should be special rules for least unassigned exponents, to ensure that they are getting assigned only to computers that will return them fast.

Well, I might have missed some details, but you got the idea. Good news is that my preliminary investigation showed that all the basic operations, like exponent assignment, exponent return, exponent update can be prety easily implemented in a way to satisfy all the requirements - in fact, they are not that restrictive as might look at a glance.

[quote="barcode"]Now the question arises on how we assign clients to servers and how we update clients to know about new servers that join the network and old ones that leave. Not difficult but it needs some thought.[/quote]

Assigning clients to servers does not sound good. Every server should be capable of serving any client. There are two methods managing set of servers: first, it just keep alive servers in round-robin DNS; if some server dies off and is not going to become alive again, it's just excluded from round-robin DNS. When new server gets deployed and ready to accept traffic, it's added to DNS zone.

Another approach would be to use dnet-like approach, where client tries contacting geographically closest server (relying on timezone configured at client computer). Just GIMPS client should always reattempt connection if closest known server does not respond (dnetc just gives us). This is not an alternative actualy, but just an improvement option, not of high priority.

proxies vs. distributed redundant servers
 

GIMPS has had proxy servers you can download & install anywhere since 1999. http://mersenne.org/ips/proxy.html

BTW a significant reason for occasional PrimeNet's availability is that all of its traffic routes through one of these PrimeNet proxy agents running on the www.entropia.com web server, forwarding transactions to the real PrimeNet server. Sometimes updating the web site ends up with the proxy offline. It's one of the issues I tabled with Entropia, and hope to address soon. Stay tuned.

That the GIMPS clients point to "entropia.com" is an early design decision I made to ensure IT folks analyzing traffic understood the connections were only going to a single, trusted place - a popular concern of high profile back in 1998... . Managing security & IT impacts is an important requirement for anyone building a grid system.

What are the main utilities for a proxy? Pragmatically, they are:
(a) aggregating network traffic
(b) caching & managing intermediate work state
(c) additional security checkpoint
(d) decentralization of risk/control

My observation about the current entropia web server PrimeNet proxy illustrates the point that a proxy represents an additional system dependency, and therefor is an operational risk. If the proxy goes down its dependent machines are blocked unless they have 'plan B' connectivity pre-configured. Generally, distributed systems are best designed to minimize the number of things that can go wrong or jam up. Add statefulness to proxies and you multiply your headaches exponentially

Consider:
- Prime95 already capably supports (b) independently (& invariantly) of a proxy, and,
- even a stateful proxy needs to synchronize itself to a central state so we still have to deal with (re)connectivity & (re)syncing;
And so my earlier reasoning went... as a result the current version of the free downloadable PrimeNet proxy agent is stateless, providing only (a) network aggregation, a request we received from the IT departments of many GIMPS-participant companies. Was there a compelling argument for anything other than a pass-thru proxy? I might have missed something. Assignment handling for GIMPS is somewhat trickier than the key ranges issued on d.net, I'm uncertain what we'd learn from that.

Regarding (d), complete decentralization is neither possible nor desirable - GIMPS still has to drive the global inputs and outputs with nominal authority and security.

However a few geographically distributed redundant, synchronized open-sourced PrimeNet servers with DNS-directed load-balancing is both interesting and achievable. Most of the synchronization challenges are readily tackeled easiest at the infrastructure level below the application design itself. I believe the redundant distributed open-source server approach holds the most promise as a next step for GIMPS. -sjk

I ran a Distribute.Net personal proxy for a few years.

Mostly what I got out of it was a *single* mass keyrate figure - and a little more buffering for when their master server went down.

It was needed a *lot* more with their projects, though - especially RC5. A semi-fast Athlon box could *easily* go through the contents of a *maxxed out* client buffer in 3 days, and they had the same connectivity problems at times as Entropia has had with the PrimeNet server.

Prime clients can buffer a LOT more work - proxies aren't as needfull, though I suspect they're still very handy for use on a firewall box in some cases.

Not saying that a proxy server group would be no help at all, though - especially if the Prime group grows fast enough....

[quote="Joe O"]Don't knock proxies! Distributed Net used 18 - 21 proxies in addition to its master keyserver very effectively.[/quote]
As have been already told, dnet is not GIMPS.

With dnet, each project consist of finite number of blocks, neither server nor proxies track expiration date or ETA for the particular blocks, and finding *the* block proves that other blocks are 'empty' (in case of cryptography project)., or might be processed quite more effectively (in case of Golomb rulers).

With GIMPS, there is infinite number of exponents but lower exponents are significantly simpler to test; so server tracks expiration dates to be capable of reassigning exponents before GIMPS reaches infinity :) , finding Mersenne prime gives no advantage in regard of testing other exponents, including smaller exponents (at least not until next great math discovery).

So, with dnet proxies do not prevent you from taking the block from one proxy but returning it to another. With GIMPS, all exponents assigned by a particular proxy would become tied to it. It still might benefit few particular computer farms, but generally it does not sound as attractive or GIMPS, so it's not of high priority.

Also, not clear what to do if a particular proxy dies but GIMPS clients under it survive. This might be resovled somehow, but I see no elegant solution (except making backups).

[quote="Joe O"]They did also provided "personal proxies" for the "top producers" and teams. These were just stripped down versions of their proxies[/quote]
During certain period I run dnetpp at a single computer, serving needs of the single computer only. :) And in fact, the 'main' proxies run exactly the same software as personal proxies, all the different is that main proxies owners know ip address of the master keyserver, and dnetc staff track master proxies avaialbility and exclude them from DNS as neccessary.

[quote="aga"]I intend to completely avoid 'master server' and make servers truly symmetric.[/quote]

I read the MySQL chapter on replication at http://www.mysql.com/documentation/mysql/bychapter/manual_MySQL_Database_Administration.html#Replication

It seems to support only a one master / many slaves configuration. All updates must be sent to the Master.

It does briefly mention setting up a circular master-slave relationship along with some warnings about being very careful ordering your updates and inserts.

In the master / slave relationship we still have the single point of failure problem. In the circular relationship it looks like updates won't be propogated uniformly unless every server is on-line!

Am I missing something?

redundant servers
 

Still catching up on this thread.

A spooling message queue might be needed to exert enough explicit control over synchronization. Might provide other opportunities, such as different database platforms at various servers, or visibility into spooled messages for debugging.

While MySQL replication is a great thing for backing database up, and has some other nifty applications (for example, spreading heavy queries along slaves, as mentioned in MySQL documentation), I think GIMPS could use more flexible model.

The ideas is to replicate not SQL update statements, but use higher level operations (messages, events, business transactions), for example:

'computer abcd sent new completion date of 23 Jun 2009 for exponent M12345'
'the exponent M34567 that was earlier in common pool is now assigned to server 5 for allocation'

and even:

'I'm server 4, I'm getting short on safely assignable exponents, please hand me out few if you can'
'I'm server 3, handing out exponent M56789 which was previously allocated to my assignable oool to the server 4'

As can be seen, there will be few messages that will require some 'master' capabilities at one of the server to throw new exponents in. But the 'master' server just need to equally spread them among servers, and then servers will communicate between each other if somehow servers face non-even traffic.

To make this part of the interserver communication algorithm scenario more complete: if a server runs out of safely assignable exponents (got private assignable pool small/empty), and for some reason other servers don't donate exponents fast enough, and a client comes and requests assignment, server just hands off a random unassigned exponent from some other's server assignable pool, and immediatelly generates a message like:

'hey server 3, I just took your exponent M54321, please forgive me. All servers, please reflect the change'

Hopefully the server 3 will not yet assign the exponent to someone else (so searvers should avoid stealing lowest exponents); if it just did it, that's not fatal, that will be go as LL test and double check run in parallel.

And one more note: only exponents still assignable to clients should be assigned to servers. As soon as some server assigns the exponent to client, the exponent in the database loses its server-assignment record. From now on, all servers will be able to handle requests related to the exponents equally (as updating/returning exponent actions do not have potential for race condition).

And one more note: if we assume that alive servers see each other, the messaging layer is pretty straightforward. But that is not requirement: if we ad a bitset 'seenby' field to each message (plus sequential message IDs assigned by originating server), then servers can route messages, and thus RAIS continues functioning even if there are less than n*(n-1)/2 links work. This might happen with remote servers on internet and last for dozens of minutes, I have experienced such behavious quite a few times. Indeed, problems like misconfigured router can last for weeks, that is not sufficient reason to let RAIS to fall aparts.

After wording all that, I naturally expect that someone will tell 'implementing own non-SQL replication does not look good at all, it will involve coding an extra messaging layer'. In fact, using task-specific messages instead of SQL statements is very natural (for example, it allows servers to handle database layer slightly differently, might easily happen if we upgrade software currently - with loosely coordinated servers that might take a week or more), I estimate it will add just 20-50Kb of source code. Considering all the flexibility task-specific messages provide (and also, reduce of interserver network traffic!), 50Kb definitelly worth the typing efforts.

I see Scott today is quite faster than me. :) Also, I'm quite glad to see the very similar approach: as those were developed independently, unlikely there is a drastic double-thinko.

I wonder, what's the current status of the new generation primenet server at Entropia? Is it just in planning 'would-be-nice-to-eventually-have' stage, or maybe there are some specific things ready?

[quote="aga"]While MySQL replication is a great thing for backing database up, I think GIMPS could use more flexible model.

The ideas is to replicate not SQL update statements, but use higher level operations (messages, events, business transactions).[/quote]

Nuts. I had hoped the database would do this work for us:

1) We wouldn't have to write the code, it would come pretested.
2) It makes it incrementally harder for other DC projects to use our open source software (this should definitely be a goal)

One thing is for certain, we'd better debug the heck out of it. If these multiple server databases get out of sync we'd have a pretty big mess on our hands! It wouldn't hurt to have sanity check messages built in:

"I'm server 1 and I think I'm up-to-date as of 12:15 Jan 19 2003. I have 100,000 LL results and 5,200 outstanding reservations, ..."

Other servers could then verify that their databases match. If a discrepancy is found it they could then begin a dialogue to narrow down where the discrepancy is.

Sounds like the perfect application for SOAP-RPC style asyncronous webservice... any thoughts on that? It will allow us to send messages in a strongly typed and fault tolerant way and also enable us to leverage the huge body of code and experience of doing things this way (as compared to a hand rolled solution).

ok now what?
 

I was thinking along soap or XML lines, too.

More about redundant servers:
- How big is N redundant servers? 3, 5, 7?
- Why not evaluate MySQL replication? We will probably learn something useful even if we decide not to use it. What are representative test cases?
- What kind of replicated servers? Simple succession-failover redundancy, or load-balanced w/failover?

Has anyone done the top-down to-do bullet list to figure out all this stuff? I'm happy to kick it off if necessary.

Re: ok now what?
 

[quote="Old man PrimeNet"]- How big is N redundant servers? 3, 5, 7?[/quote]
I was thinking that 2 to 5 should be ok, with 3 optimal. But data structures should allow up to 30 servers - that should cover any imaginable practical need (remember, stats servers might biggyback on core servers without being included into RAIS), but wll not increase database size by more than 4 bytes per record (message).

[quote="Old man PrimeNet"]- Why not evaluate MySQL replication? We will probably learn something useful even if we decide not to use it. What are representative test cases?[/quote]
Well, learning MySQL can not hurt at all. But instead evaluating it I prefer relying on my experience and knowledge.

I see no way how MySQL SQL-level replication could help. First, and most, a MySQL server can not have more than one master. That limits to only 2 servers, or a circle of servers and if the circle breaks, databases will quickly become assynchonized.

Second, installation of MySQL replication requires whole access to MySQL engine. If we do not rely in that, then unprivileged login is sufficient.

Third, MySQL slave requires a login at master with File privilegy. With decentralized servers, that is not secure at all.

Forth, that will not always work. For example, if we keep a table of outstanding assignment, and GIMPS client first updates an exponent and 15 seconds later finishes testing and return result to a different server, then MySQL replication might lead to situation where slave attempts to update record that have been already deleted; that will shutdown slave thread stopping replication until human intervention. With custom messaging layer, that can be gracefully worked around by applying rule 'if the exponent is not assigned, then its update can be safely disregarded' or maybe 'tried later' (and incident logged, so that if situation repeats suspiciously too often, there was some stuff fordebugging).

Fifth, MySQL server can log only one set of replication data. This precludes from using a shared MySQL server.

And so on. Just believe me that for this task raw MySQL replication will cause more pain than use, and only limited solution is possible with raw MySQL replication.

From the other side, MySQL replication is going to be a perfect thing to piggyback stats servers over core server(s); and also do realtime backups.

[quote="Old man PrimeNet"]- What kind of replicated servers? Simple succession-failover redundancy, or load-balanced w/failover?[/quote]
Load-balanced RAIS would be most interesting, and I believe most worthy thing to apply efforts to. As you told that reliability problems with current primenet server will be mostly solved soon, we have few extra months to develop top-notch solution, and not just resort to a minimalistic approach as randomly choosen low-level software permits.

[quote="Old man PrimeNet"]Has anyone done the top-down to-do bullet list to figure out all this stuff? I'm happy to kick it off if necessary.[/quote]
Mostly, in the forum you can find more questions than answers. Majority of people here prefer discussing best color of CPU cooler for GIMPS server rather then share opinion on questions like should database keep all GIMPS related data, or adopt the approach similar to the current primenet implementation when after exponent is tested, it's moved to another database maintained semi-manually offline. Things like that directly influence database structure, notwithstanding how many servers there will be and how they will communicate with each other.

My high-scale vision on the development stages is:

1) Decide which information stored online, and how long does it maintained there. This includes already tested exponents, returns history (to have information useful to estimate how fast and trustful computer is, which influences if small exponents get assigned to the computer)

2) Client-level operations allowed in network. This will/might include: team creation, user account creation (also, creation of anonymous account? Or maybe this should be better moved to client side?), setting/changing team for user, exponent reservation/assignment, exponent return/release, exponent return/result - or maybe 'exponent reservation for trial factoring' etc., reassigning exponent from computer to another computer (within the same user account). Also account alterations (updating email address, maybe joining accounts, etc.) The final list should include all parameter associated with each kind of operation, and their types, and also limitations - for example, I think it's reasonable to add a rule like 'if someone returns an exponent that was not previously assigner to the user, such result is accepted but considered as low-quality, and user who had exponent assigned continues it's testing. Results from computers that earlier returned wrong results are considered as low-quality Exponent is considered as tested only if it have 2 (3?) or more high-quality results' (and probably low-quality results can be throws away most often, as they might be forged or stolen)

3) Now, core part of the database might be written down.

4) Replication messages. This will include all from 2 (but now, the important thing is not lient experience but interserver communication that does not experience dangerous race conditions), plus server-specific messages like 'I'm low on assignable exponents, share some with me'.

5) Database structure from 3 is extended with tables that queue messages, and additonal fields to store things like server id who owns a particular exponent.

6) Write down algorithms used to process each message from 2 and 4. This indeed will be mostly done while stages 2 and 4 take place, but here we formalize everything - if a fault found, we return back to 2 or 4.

7) Decide on auxilary softwares and protocols used (for interserver messaging, etc). Also, agree on protocol modifications or extensions neccessary for GIMPS client-side software.

8) Code everything

9) Test everything

10) Test everything once again :)

11) Switch GIMPS to the new server(s). Current primenet server and new RAIS might coexist for prolonged period of time, there is no need switching all traffic at once.

So if there are no corrections to this list, lets start with 1 (information online database should contain) - that's factually what I'm waiting for. Here yours and George's experience is of highest value. After the stage 1 done, we then can move further at increasing pace.

Re: ok now what?
 

Aga: Thanks for your insights into MySQL replication limitations.

[quote="aga"]I was thinking that 2 to 5 servers should be ok, with 3 optimal. But data structures should allow up to 30 servers - that should cover any imaginable practical need (remember, stats servers might biggyback on core servers without being included into RAIS), but wll not increase database size by more than 4 bytes per record (message).[/quote]

I was thinking 3 or 4 servers. We pretty much agree that one server can handle the load, the added servers are to insure high availability and better geographic distribution. Bandwidth will limit the maximum number of servers.

Re: ok now what?
 

[quote="aga"]1) Decide which information stored online, and how long does it maintained there. This includes already tested exponents, returns history (to have information useful to estimate how fast and trustful computer is, which influences if small exponents get assigned to the computer).[/quote]

All results data will be maintained on the servers. Forever. The only exception might be LL results that are proven incorrect and LL results when a factor is later found.

I can see some stats data being aggregated and deleted. For example, we might keep every CPU credit record for a year so that you can run queries about your CPU contribution over the last week, month, or year.
After a year the individual CPU credit records could be deleted and we just remember your CPU contribution for all of 2002, 2003, 2004, etc.

database & statefulness
 

One big lesson I learned building PrimeNet v1-v4 was where to put statefulness & related client handling decisions. Despite the notion that we 'just hand out exponents', there are about a dozen real-life corner cases to worry about (I'll enumerate these later).

For a long time I wrote system service or stored procedure code to handle questions of exponent fitness for testing by checking certain builds of Prime95 or CPU capability, or when it was ok to 'override' an assignment by an unexpected move to a different machine, when it was poaching, when to reclaim an exponent as 'probably lost' vs. just a slow machine, etc.

What I'd recommend we do this time is to put more of this into the database tables, and keep the logic above it as stateless & decisionless as possible. PrimeNet presently has 3 main SQL tables, exponents, users and machines. Instead of the SQL & C++ spaghetti to decide if a particular exponent can be assigned to a particular machine, I think we just need a table to join against the exponents and the machines tables that is masked for each machine and for each build of Prime95. Maintaining the masking tables is then operations data management.

We'd have a single SQL statement to get assignments (like today) except we wouldn't need to review the fitness for testing of each exponent, and go back to the database for the next one, etc.

What I was thinking about, is to apply some kind of rating for exponents and computers. Rating is represented by an integer number 0..100 (or 0..255). Smaller exponents get higher rating; faster computers get higher rating; longer a particular computer is around, higher rating it gets and so on.

So while assigning an exponent, core of algirithm is 'having computer with rating N requesting exponent, assign it a smallest (or 'almost smallest') exponent that have rating below or equal N'. With suitable db tables structure that can be implemented with a single SQL query (tho 'single SQL query' is not a natural goal, more like heuristic).

All the ratings don't require any particular precision. Each server can run a script once a week or month recalculating ratings of untested exponents and active computers. Note that this update does not require replication over servers, there are no problems if ratings at each server somewhat differ.

So all 'spaghettied' code will be in a non-realtime script, and if someone in mood playing with weights and formulas, the script could be run several times a day. Even if the script will screw something up assigning weird ratings to numbers and computers, it will not cause disaster. All that's needed to ensure overall processing stability, is few powerful computers running GIMPS client-side software, with highest (255) rating assigned. I think I saw George runs 2-way P4 1.6GHz box; that should be sufficient - the box will mostly run small exponents, and thus will be able to process number of exponents per month ensuring smooth overall GIMPS rate. Formulae and weight should be choosen in a way that there were at least 2 dozens of fast computers with weight above 200.

In this thread on poaching, http://www.teamprimerib.com/gimps/viewtopic.php?t=308 , I discuss server upgrades necessary to reduce this blight.

In short, the smallest double-checks and first-time tests are handed out to machines that have previously returned results and can complete the assignment in a reasonable time (say 1 to 3 months). This reduces a poachers temptation to grab an exponent from a slow machine that is holding up a milestone.

We also inform users that the server only guarantees an exponent reservation for one year. The server has smarts that help slow machines get assignments that will complete in a year and are unlikely to hold up a milestone. Paradoxically, this means giving slower machines larger exponents! Giving a part-time Celeron machine an 11M exponent that takes 8 months to complete is more likely to hold up a milestone than giving the same machine a 17M exponent that takes a year to complete.

Scott is right that these time & speed parameters belong in a database so that they can be fine tuned easily.

P.S. If you want to debate this proposal or other aspects of poaching please post in the poaching thread rather than this thread.

view from 50,000 ft
 

I believe we need to proceed in discrete baby steps to avoid a big code-n-go approach that is going to be hard to make work. Taking a more sweeping perspective, let me poke at stick at a few things that have more leverage over design decisions than other things:

1. Migration - First, are we decided that we should migrate clients from the old PrimeNet to the new one? What makes sense?
(a) Migrate the server 'under' the v22 (and earlier) clients into part of a new federation of 3 to 4 servers, and later build a new client and upgrade as many clients as possible?
(b) Or just build a new federation of 3 to 4 servers, and build new client code at the same time, then upgrade as many clients as possible, leaving the old PrimeNet to starve off by slow machine attrition?

Migration path (b) sounds like less work to me if we agree we need a new network protocol (and thus client build) anyway. I believe I will recover ownership of PrimeNet from Entropia, so we have the time for option (b).

2. Federated Servers - What do we need vs. what do we want with redundant servers? Given the fact that currently only one server can handle everything, is load balancing a strong requirement if failover solves the availability issue?
(a) Federated servers with parallel shared database state is cool, but more complicated and adds cross-server update locking latency, which pure mirroring to a standby database does not add. It would make cutting over to a 'plan B' server instantaneous (clients simply redirect themselves to an available alternate server). I confess I like this option but it gives me pause for risk concerns. The only way I personally know how to make this work well w/o pricey OTS productware is via a messaging queue.
(b) A failover database can take advantage of SQL-level replication, but is usually limited to peer database versions; and cutting over from server 'A' to 'B' may require a manual reconfiguration step to activate 'B'. Low risk, but not as cool & flexible moving forward. Only one server is 'online' at any time, but availability is still augmented ok.

3. Client code upgrades or support to run other apps - Are we thinking of addressing this for opt-in users? This can be a big can-o-worms, but I have experience building these a few times already (tho I need to stay clear of my Entropia patents).

I think the 'half-life' of a Prime95 client on a typical GIMPS machine affects this decision. For example, if Prime95 FFTs are good for about 3 years before testing work progresses past its utility, then maybe its better to let it alone and drop from attrition when the machine is upgraded or tossed out. What got me thinking on this was GIMPS has stayed flat with number of parallel CPUs applied but has grown with Moore's Law from 1 teraflop in Y2K to 5+ teraflops today. Yet around 30K machines ran in parallel any given day most of that time, many of those being new installs.

Comments? More on other requirements coming up...

It doesn't look like load balancing is important for the near term - if PrimeNet growth turns exponential, perhaps in a while.

Server availability seems to be the primary concern among those of us that server downtime annoys....

parallel SQL
 

Is there a mySQL or an above-ODBC/JDBC-level freeware equivalent of Microsoft DTC or IBM's MQ Series DB2 add-on to keep databases in sync?

availability requirements
 

[b]Availability (& Redundancy) Requirements[/b]

Looking back, PrimeNet has always had occasional availability outages. Prime95 by design queues up work to do and synchronizes itself to the server when connectivity resumes, meaning unattended Prime95 clients automatically continue uninterrupted productivity in the vast majority of cases... unless the outage is sustained for periods of several days or more.

Let's examine whether the annoyance of an unavailable server is centered upon
(a) those wanting to see immediate or frequent web-site feedback on their account's progress or the overall progress of the search, or
(b) by whomever stages testing work and collects results from the server, or
(c) outages that last longer than the average work-to-do queue.

I'm sure all 3 are true, but what's the true pain point?

(c)? Sustained outages of PrimeNet v4 are perhaps in large part due to the fact that only one Entropia employee supports its operation, and upgrades and bug-fixing have been utterly frozen, with few exceptions, since I started growing Entropia's business in late 1999. This is expected to change soon, so at least one source of annoyance (c) should be mitigated.

(b)? I can understand this one readily. But George has lived with this thorn for years so its hard for me to imagine it being more pressing now, except that when Brad takes off to Europe for a month to install more grid customers (like he just did again Saturday) then PrimeNet's support probably suffers.

(a)? I suspect this is significant. The web site reflects PrimeNet's state in a highly visible way. If the server is down, the best we can do is show partial or stale data on the web site.

We probably agree it would be great if PrimeNet would simply run without any unplanned downtime. Where does it come from? Here's my semi-informed guess/recollection/knowledge:

[b]Sources of PrimeNet v4 downtime (of say, 20 outages):[/b]
1. entropia.com proxy dead or jammed up 55%
2. PrimeNet component dead or jammed up 25%
3. ISP line out / long-term power failure 5%
4. DoS attacks / Internet 'storms' 5%
5. web forms abuse 5%
6. planned maintenance 5%

Note that data corruption / loss is not even listed (though the v1 & v2 servers had this issue). My point here is that perhaps data integrity via replication is not the main concern to focus upon. Important to manage as a risk factor, but not a burning hot spot.

[Sidebar - Note item (5) has been a point of some headaches in the past, too - it seems impossible to provide a web form for manual database use w/o some party eventually writing a script that irresponsibly 'mines' PrimeNet's database; this issue will reappear among the open-source requirements issues to resolve.]

My second observation is that about 4 in 5 outages is due to failure that could be fixed by our efforts here. Note also that except for disaster risk and control being centralized, a fixed-up single server could avoid 85% of the outages (items 1,2,5).

[b]Therefor, a single server with one or two mirrored failover secondaries would suffice[/b] - At this point I want to reiterate support for an open source PrimeNet solution and its shared operating responsibility. After this cursory analysis, however, I believe that a simple fail-over system having mirrored data would be sufficient. Anything beyond that may only be cool or planning ahead, but a want and not a need.

Execute arbitrary app code? - The only thing I can think of that would truely require a parallel shared / federated server would be one requiring high scalability, or low-latency client resyncs (or both), or arbitrary application code deployment & execution - are we going there with this design? Did I miss anything for the analysis?

Re: view from 50,000 ft
 

[quote="Old man PrimeNet"](a) Migrate the server 'under' the v22 (and earlier) clients into part of a new federation of 3 to 4 servers, and later build a new client and upgrade as many clients as possible?[/quote]

I think it should be pretty simple to implement compatible http interface for old clients. It may even be the primary interface.

I'm not sure about old RPC interface. Are there still noticeable amount if RPC-communication clients? I.e. can they be forgotten about (or left at primenet v4 server), or there will be need for RPC->HTTP gaetway? In latter case, how much work we are going to face?

[quote="Old man PrimeNet"](a) Federated servers with parallel shared database state is cool, but more complicated and adds cross-server update locking latency,[/quote]

Locking latency? All interserver communication should be asynchronous, all messages go into queue and then replication thread(s) handle it independently of front end.

And regarding cool... yeah, that would be cool. Isn't GIMPS was started because it's damn cool? :) At least, noone yet explained what the found mersenne primes can be practically used for.

[quote="Old man PrimeNet"](b) A failover database can take advantage of SQL-level replication, but is usually limited to peer database versions; and cutting over from server 'A' to 'B' may require a manual reconfiguration step to activate 'B'. Low risk, but not as cool & flexible moving forward. Only one server is 'online' at any time, but availability is still augmented ok.[/quote]

This is a good note, configuring failover servers is not going to be trivial, there are still a bunch of problems to resolve. As all fruits are high engough, why target at smaller one?

[quote="Old man PrimeNet"]3. Client code upgrades or support to run other apps - Are we thinking of addressing this for opt-in users? This can be a big can-o-worms, but I have experience building these a few times already[/quote]

This might be implemented, but no need to include it into core servers. Doing centralized upgrades need nothing from GIMPS database, not even list of accounts (it would be different if GIMPS client-side software was commercial).

Migration:

I think we should aim for a server that accepts v22 and earlier CGI requests. The protocol is known and shouldn't be hard to support. The new server will require a new client with new or improved messages. One particularly painful migration will be from the current userid scheme to the new not-yet-designed userid/team scheme. Note that v22 contacts mersenne.org directly whereas v21 goes through entropia.com.

Outages:

The biggest annoyance is the length of the outages. With only one Entropia contact it was not uncommon for the server to be down for an entire weekend or worse if Brad was overseas. A single server with the ability for any of a half dozen people given the authority to reboot it remotely would solve our problems.

Multiple servers / Mirrored servers:

We do not have an immediate need for load-balancing or even failover servers. Mirroring through replication is a great way for us to avoid writing code to do oodles of logging and disk backups. It also would let us have an emergency backup machine should the main server have a hardware problem that takes extended time to fix. There is some concern that a single server might not have adequate CPU power once the stats reports are beefed up but that remains to be seen. Even though we only need a single server we must design the server code with multiple servers in mind.

Arbitrary apps:

I would like to add ECM factoring to the server. It would be nice if other small math projects could be added easily. The entire stats engine should work without modification for these other projects.

Re: view from 50,000 ft
 

[quote="aga"]I'm not sure about old RPC interface. Are there still noticeable amount if RPC-communication clients?[/quote]

v22 does not support RPCs. v21 did but had crash bugs.

Re: availability requirements
 

[quote="Old man PrimeNet"](a) those wanting to see immediate or frequent web-site feedback on their account's progress or the overall progress of the search, or[/quote]

After years of GIMPSing I still feel very unhappy if I decide to browse stats and see ISAPI error instead.

( I should admit that after observing the errors multiply times I still survived. :) )

[quote="Old man PrimeNet"](c) outages that last longer than the average work-to-do queue.[/quote]

This does not sound good at all. If it gets down to a point where the 'avarage work-to-do queue' gets exhausted, it already means that half of GIMPS clients stay idle. And I bet, that's mostly fastest computers.

Also, add (d) - if someone just downloaded GIMPS client and can not make it doing some work because PrimeNet is down till Monday, that is effectively lost participant.

[quote="Old man PrimeNet"]3. ISP line out / long-term power failure 5%[/quote]

Hmm, 95% availability? That sounds pretty low. I feel unhappy if my servers experience less than 99.95% availability. This is the level that goes unnoticed (and/or forgiven) almost always.

(I realize that 95% is not pretty legal here as 5% of percentage of downtime not total time; but if that was 99.95% then it didn't worth mentioning but you did. And thinking of even 99% uptime concerns me).

[quote="Old man PrimeNet"]4. DoS attacks / Internet 'storms' 5%[/quote]

Things like PrimeNet are indeed attractive targets for DoS attacks. Here failover servers are pretty useless - one can kick off one server, then another. But if there are several mutually-replicating servers at different geographical locations, each behind a fat link... well, good luck.

Also, replicating servers carry less risk to become target of DoS attack. If it is going to be pretty useless/tasky to establish a -successfull- DoS attack, trying it at all looks much less attractive.

Note that data corruption / loss is not even listed (though the v1 & v2 servers had this issue). My point here is that perhaps data integrity via replication is not the main concern to focus upon. Important to manage as a risk factor, but not a burning hot spot.

[quote="Old man PrimeNet"][Sidebar - Note item (5) has been a point of some headaches in the past, too - it seems impossible to provide a web form for manual database use w/o some party eventually writing a script that irresponsibly 'mines' PrimeNet's database; this issue will reappear among the open-source requirements issues to resolve.][/quote]

I think it's reasonable to allow a certain ip address to contact server only 100 times per hour at most. All further attempts will not be handled with instant error message returned instead. This is going to be pretty easy to implement, and as we are not going restarting servers now and then, tracking ip addresses can be done in RAM thus causing no I/O at all.

Thing that worries me, is a 'hypotetical' computers farm behind gate/firewall, all doing trial factoring. That could generate more than 100 valid requests per hour, especially at the beginning of business day when hardware gets turned on. Ok, let's set limit at 150 er server and use 4 servers. If we resort to failover scheme, the single active sever will not be as protected.

[quote="Old man PrimeNet"]My second observation is that about 4 in 5 outages is due to failure that could be fixed by our efforts here. Note also that except for disaster risk and control being centralized, a fixed-up single server could avoid 85% of the outages (items 1,2,5).[/quote]

But noone can provide alot of time for a project in a prolonged period of time? So half year later your business will again start pooling alot of time, and the 4 of 5 outages will return? I would feel much safer with 2 or 3 operators supervising Prinet. This almost ensures that at any times at least one server have human around, and thus number of alive servers will never reach zero.

[quote="Old man PrimeNet"]Execute arbitrary app code?[/quote]

You mean client-side? Why not. Sounds attractive to include into GIMPS all the non-x86 Unix and Mac powerful boxes. That will require to wide-spread client-side source code, and that in turn will require much higher robustness of server(s).

[quote="Prime95"]I think we should aim for a server that accepts v22 and earlier CGI requests.[/quote]
With JSP/Servlets web application one can emulate CGI interface pretty easily. That's extremely flexible environment (and of course Servlets provide much better performance that CGI).

[quote="Prime95"]The new server will require a new client with new or improved messages. One particularly painful migration will be from the current userid scheme to the new not-yet-designed userid/team scheme.[/quote]

It is not evident why to change protocol? Small improvements should suffice. (And if we nevertheless go changing all at once, I'd like seeing XML based protocol, to ensure that that's a very last large change).

In particular, why change client-server protocol adding teams there? Let's put all new users into 'unassigned' team and give user a web form where he/she can switch team at any moment (client-side software could also connect to the web page submitting new team; but I believe doing too much GUI stuff in client-side software does not worth time, and will burden PrimeNet with supporting the auxilarly pages forever).

[quote="Prime95"]Note that v22 contacts mersenne.org directly whereas v21 goes through entropia.com.[/quote]

I think, for new server(s) there should be subdomain v5.mersenne.org or alike. This removes requirement of hosting all versions at a single server (set of servers), tho it still can be done that way. If for some reason next major prime95/mprime release goes before new server is up, I suggest setting v4.mersenne.org as default server.

[quote="Prime95"]I would like to add ECM factoring to the server. It would be nice if other small math projects could be added easily. The entire stats engine should work without modification for these other projects.[/quote]

Hmm, in fact additional java web applications can be deployed pretty easily. Or the additional code might go as package, then it is possible to write class loader that will provide such packages a jailed, restricted environment. But I would prefer delaying things like that until v6.
I mean, P-1, ECM etc should indeed be in v5 (could you give a full list of currently appealing algorithms?), but as an integral part of web application, not as pluggable modules - latter might turn into a whole ocean of work (which might be interesting, tho, after more important concerns are resolved).

Not sure about other small projects. Database structure is going to be very mersenne-centric (or otherwise it will be fat and slow). But with sufficiently generic messaging layer, cloning the software for other distributed projects should be pretty easy.

Re: availability requirements
 

[quote="aga"][quote="Old man PrimeNet"]Execute arbitrary app code?[/quote]

You mean client-side? Why not. Sounds attractive to include into GIMPS all the non-x86 Unix and Mac powerful boxes. That will require to wide-spread client-side source code, and that in turn will require much higher robustness of server(s).[/quote]

I don't know if that is what he means, but the ability for glucas and mlucas clients to connect directly to the server for work assignment and sending of results would be great.

[quote="aga"]It is not evident why to change protocol? Small improvements should suffice.
In particular, why change client-server protocol adding teams there?[/quote]

V22 Prime95 only has a concept of userid (a team is really just a userid with lots of computers). V23 will have a userid and an optional teamid.

This is one of many v23 changes that will affect the messages passed to the primenet server. Maybe we're just having a terminology problem: I'm not suggesting changing the http/cgi protocol, just changing the messages sent using that protocol.

[quote="aga"]Not sure about other small projects. Database structure is going to be very mersenne-centric (or otherwise it will be fat and slow).[/quote]

The reservations and results will be Mersenne-centric. The teams / users / computers / stats databases should not be.

LL handling
 

I think about 2 things:

First, is v5 should allow running test and doublecheck at the same time, and thus assign exponents for testing and doublechecking at approximately the same time. Of course, client-side software will not show if it's first or second run; also:

- faster computers will factually do first run, and slower ones will doublecheck; instead the timed out exponenets will be assigned to faster computers thus making chances finding a prime are more even.
- milestones will be pretty ordered and predicatable
- servers will need to handle only exponents located in a relatively short range only, which [might] allow some optimizations
- handling all LL assignments similarly will simplify server codes, and will make unnecessary creating special anti-pouching codes.

Disadvantage is that slower computers will always do double-check, that will annoy them. But maybe, that's better for GIMPS as a whole?

Another idea, is running tripple checks too:

- that can be used as final testing of new programming codes. Results of the testing will not be quite useless - it would be interesting to find if there are both test and doublecheck wrong for some old exponent.
- new computers can be assigned tripplecheck at the very first time. First, it will improve experience for new GIMPSers, as the exponent will be tested in a matter of days or hours instead of weeks or months. Second, it will allow server to recognize faulty-hardware computers before they are assigned real work. Third, it will give server better initial estimation on factual computer speed (including hours/day it's on).

Disadvantage is of course the processing power used for tripple checking will not be used for
testing larger exponents. This is pretty serious.

As related idea, it might be reasonable to assign factoring work for first-timers; new computers will have low initial rating anyway, so small LL tests will not be available for them; but if they successfully run few factoring assignment, that should raise their rating quickly and then computer will start receiving 'good' LL assignments.

[quote="Prime95"][quote="aga"]why change client-server protocol adding teams there?[/quote]

V22 Prime95 only has a concept of userid (a team is really just a userid with lots of computers). V23 will have a userid and an optional teamid.

Maybe we're just having a terminology problem: I'm not suggesting changing the http/cgi protocol, just changing the messages sent using that protocol. [/quote]

No, that is not terminology this case. What do I mean, is how dnet currently work: with dnet client, only userid (email address) is configured, dnet client knows nothing about teams; if you want join some team, you can do it at dnetc web site (at project stats pages). That's what I propose for GIMPS to adopt; and overall, don't complicate client and client-server protocols with stuff that can be implemented entirely at server-side.

After all, it's much easier to update server-side software than thousands client-side installations; also, if certain things are implemented at server-side, it makes client-side communication module smaller thus encouraging primenet support with other programs.

Re: LL handling
 

[quote="aga"]First, is v5 should allow running test and doublecheck at the same time[/quote]

This is a non-starter. Our goal is to keep GIMPS fun for the participants, not to make life easier for server development or to produce more orderly milestones. A big joy for GIMPS users is getting an exclusive assignment. You and only you have a chance to make a big find. Even relatively slow computers get this chance.

In general, we let the user decide how he wants to participate (within limits) rather than have the server impose a rigid testing procedure.

[quote="aga"]What do I mean, is how dnet currently work: with dnet client, only userid (email address) is configured, dnet client knows nothing about teams; if you want join some team, you can do it at dnetc web site (at project stats pages).[/quote]

This is a good question for the users - configure everything in one place (the client) vs. two places (the client and the server). Obviously, either method works.

OTOH, we may be delving a little too deep into the details with this issue...

Migration: I saw the v22 source pointing to mersenne.org - I agree that was a necessary change.

> ... a server that accepts v22 and earlier CGI requests.
> The protocol is known and shouldn't be hard to support.

It would be a 'compatibility proxy' server for the 'old guard' GIMPS machines, v16 thru v21 (v14s & v15s ran until late 1998). An evil task, hacking at old code.

> ... will require a new client with new or improved messages.

Undoubtedly.

> One particularly painful migration will be from the current
> userid scheme to the new not-yet-designed userid/team scheme.

The sort order used now is binary byte... does that help or worsen things?


Outages: hopefully soon moot. We should have several admin-like folks.

> Multiple servers / Mirrored servers:

> Even though we only need a single server we must design
> the server code with multiple servers in mind.

ok. if we can mirror databases, shouldn't we be able to build snapshots from replayed transaction logs to run stats for a web server? How often and how large would a log file segment be? Can it be configured for hourly new log file segments that can be quickly post-processed (elsewhere) and then published on a web site (perhaps elsewhere again)?

If we stick with a stateless front end design, we would be able to multiply them as fan-outs from the database server for scalability. Also suggests that a machine-unique ID is passed with every server interaction.

I also recommend we use a different base URL for each version of Prime95 v23+, different in DNS name, path or executable name, to simplify the management of multiple client versions and increase our ability to route specific ones where we want. For example, v23primenet.mersenne.org/v23 or v24primenet.mersenne.org/v24, etc. Any drawback to putting out a new DNS name each build? We would have the flexibility of making the stateless front end routing each version via a different server and/or different URL path. Or the same server. In the initial case, they can share the same server or even the www server, but later we can move these components around as needed.


> Arbitrary apps:

I think you mean, 'add to the server at the code level'. I was thinking of binary modules encapsulating various API-sandboxed apps. I will think more on the source being modular instead...

Separate team ID: thank goodness!

Mostly I'm thinking about protocol, database and boundary changes. What does the database need this time to act upon stateless HTTP parameter-only hits?

Security - How about using HTTPS with HTTP fall-back? Or did the visibility of the HTTP data help GIMPS stay low-profile/harmless? What about trusted source builds vs. public open source builds accessing the network?

team managed at server idea
 

I like the team managed at server/web-site idea. This is what we did at Entropia for two other distributed computing projects. It avoids the whole merge/defect statistics issue by letting the database manage teams as loose federations.

A few more loose ends
 

Comments...?

[u]Web site reporting[/u] - what do we want to do here? - Luke Welsh helped me design the PrimeNet hourly status reports layouts, particularly the ranges & counts columns (we did it over beers at the Faultline microbrewery in Silicon Valley). I think it will be fun to revisit what we want to see there.

[u]Web site manual forms access[/u] - I've seen a few posts here and previously on the Mersenne list suggesting we limit hits by IP-count-per-hour or similar. Is there a quick & easy way to configure this? Otherwise, it's something we could more simply avoid by making a 'sandbox' table exposing a limited subset of exponents.

[u]Web server layer?[/u] - We get free logging, connection handling & web hosting features by staying with a CGI or ISAPI interface, but we can also cut it from the loop this time and take sockets directly. I did CGI for PrimeNet v1-v4 - sometimes if the system service layer stopped on an undecidable rule, we'd have a few hundred pnHttp.exe processes pile up in memory... ISAPI would better mitigate that. I frequently checked the www logs as a debug tool showing URL hits against the other layers. I suppose I'm leaning toward keeping the web server layer status quo for continued flexibility.

[u]Pushing GIMPS upgrades to opt-in machines?[/u] - It's possible to do this safely and reasonably easily, but does it make sense? It would amount to delivering a compressed, encrypted & signed DLL to the machines folks setup as opt-in for upgrades.

[u]Open source client builds, trusted or open?[/u] - Do we propose handling this differently? Today, only folks holding a trusted key can build binaries the server will accept. If we keep it, the key & encoding method should be strengthened. If not, how do we manage providing open source without giving folks a direct door for malicious network access?

> Web site reporting - what do we want to do here?

This is "easy". There are many other DC projects out there to emulate and improve upon. The reason it should be easy is once we create the database layout I think we can get several volunteers to write a plethora of reports: Project-wide tables and graphs, Team & User standings (forever, last year, last month, overtake reports, by project (LL, dblchk, 10M tests, P-1, TF), recent activity), computers, etc.

> Web site manual forms access

A sandbox or per-hour limit or some such. We do need these forms.

We also need a retrieve my password form (emailed to the user).

> Web server layer?

I'm not qualified to comment.

> Pushing GIMPS upgrades to opt-in machines?

Might be nice, might scare some people to death. I vote to ignore for now.

> Open source client builds, trusted or open? - Do we propose handling this differently?

Trusted builds could use a key to gain trusted access.

All open source builds could generate a random key at compile time. We can then limit activity on that key to a few transactions per day. To guard against a really bad guy creating thousands of executables or generating a new key every connect we can also limit the total non-trusted accesses per hour. This is still security through obscurity, but as good as I can come up with in an open source arena.

Another security idea: each reservation should come with a 32-bit key written to worktodo.ini. Without this key you cannot unreserve the exponent (or report a result and obtain cpu credit?). This is to fight malicious unreserving and poaching reserved exponents.

> > Not sure about other small projects. Database structure is going to be very mersenne-centric (or otherwise it will be fat and slow).

> The reservations and results will be Mersenne-centric. The teams / users / computers / stats databases should not

Look at the distributed.net stats code? It's stable, it seems pretty robust, and it DOES handle a lot more load than Mersenne is likely to ever need.

> Pushing GIMPS upgrades to opt-in machines?

As an opt-in, perhaps - but the client will need a LOT of testing before being put into the "push" que if this is done.

> I like the team managed at server/web-site idea

Seems to be the "common" method, and it works.

> This is a good question for the users - configure everything in one place (the client) vs. two places (the client and the server).

Web-based *client* configuration is a PITA in my opinion - it was one of the things I disliked the most about United Devices. Client doesn't need to know about teams, just your user ID and your machine ID and what it's working on, IMO.

I suspect that a lot of new users won't have any interest in a team - those that do probably got recruited by an existing team, which should be easily capable of guiding them through the "join team" process....

9-)

[quote="Prime95"]
Another security idea: each reservation should come with a 32-bit key written to worktodo.ini. Without this key you cannot unreserve the exponent (or report a result and obtain cpu credit?). This is to fight malicious unreserving and poaching reserved exponents.[/quote]

Extremely good idea! :D

[quote="Old man PrimeNet"]ok. if we can mirror databases, shouldn't we be able to build snapshots from replayed transaction logs to run stats for a web server? How often and how large would a log file segment be? Can it be configured for hourly new log file segments that can be quickly post-processed (elsewhere) and then published on a web site (perhaps elsewhere again)?[/quote]

It sounds easiest to deliver data from core server(s) to slave stats server(s) using realtime MySQL replication. Probably no need to go into mess with batches, at least not with v5. MySQL replication allows to filter out data unneeded by stats servers - sounds like it does not cause security problems in this particular case. (Hereby I assume that there will be at least one core server that runs with a private MySQL engine; thus there is GIMPS-only replication log available). If don't filter replication log, then stats servers will also act as backups (even failovers?) for core servers, thus eliminating need of explicit backing up.

This will not work if we want to avoid keeping users passwords at stats servers (i.e. when someone logins at stats server, the stats server checks authorization at one of the core servers). The question is, do we need that? A public stats server could simply log all seen logins anyway. So it sounds better if instead only trustful enough stats servers are used?

But if core servers don't wipe out processed messages quickly, then it might be possible to feed stats servers with customized highly-compressed data batches (factually, only result returns matter). In this particular case, I believe the bandwith saved will not worth the efforts (and extra disk space, and RAM for db caching, used at core servers).

[quote="Old man PrimeNet"]I also recommend we use a different base URL for each version of Prime95 v23+, different in DNS name, path or executable name[/quote]

I suggest using different subdomains, not paths. Subdomains allow most flexible routing of requests using DNS, but paths could be served by different servers only with proxy installed, which is not too robust.

Also, it sounds more natural to use subdomains based on protocol version, not protocol of client. First, if there is a new release of mprime/prime95 that improves LL speed 2fold :) but without any other changes, spawning a new subdomain is strange. Second, if/when there is other primenet-enabled software becomes avaialble, it will be a whole mess to maintain all the versions of indemendent clients. Also, it will be virtually impossible forget changing the domain name with new release - if protocol has changed, it will not work, that's all.

[quote="Old man PrimeNet"]Security - How about using HTTPS with HTTP fall-back? Or did the visibility of the HTTP data help GIMPS stay low-profile/harmless? What about trusted source builds vs. public open source builds accessing the network?[/quote]

HTTPS does not sound good. It gets CPU usage on the server noticeably higher, requires additional roundtrip(s) to establish connection, takes additional RAM to track secure sessions (which is not needed at all), and tripples network traffic. In addition, use of HTTPS precludes use of transparent comperssion protocols ilke MNP5/V42b/ppp-deflate; thus modem users might experience up to 10x worse network performace - with usually overloaded modem links, that might start causing troubles.

And for all that, gives no gain. The usual use of HTTPS solves only 2 tasks - authenticating web site (making sure that there is no DNS spoofing), and prevents third parties from intercepting/altering traffic en-route. Latter is not needed for GIMPS at all, former is a very rare problem, and unlikely will ever affect GIMPS except maybe few internet users who use untrusted DNS server.

Potentially, HTTPS could be used to authentificate users/computers; but for that we need a centralized mean of distributing/signing keys. It's going to cause much more headache than usefulness.

HTTPS could be useful for distributing client-side automatic upgrades. But it sounds better to just cryptographically sign all binary modules/plugins, ship public key with client-side software, and then use unsecure inexpensive means of software distribution.

Re: A few more loose ends
 

[quote="Old man PrimeNet"][u]Web site reporting[/u] - what do we want to do here? - Luke Welsh helped me design the PrimeNet hourly status reports layouts, particularly the ranges & counts columns (we did it over beers at the Faultline microbrewery in Silicon Valley).[/quote]

BTW, this report can be easily made live, assuming that stats servers receive live stream of returned results (unreserved expoenents, too). What it boils down to, is keeping an integer array 15x300 in static object. When stats server receives event, it substracts 1 from one cell and adds it to another; periodicatelly checkpointing array and data stream onto external database. Then, generating HTML page with up-to-a-second correct numbers based on the array will be virtually as fast as serviing out a static page. Not that the realtime numbers are really neccessary, but it might be quite easier implement it that way, instead of tracking the hourly boundaries.

[quote="Old man PrimeNet"][u]Web site manual forms access[/u] - I've seen a few posts here and previously on the Mersenne list suggesting we limit hits by IP-count-per-hour or similar. Is there a quick & easy way to configure this? Otherwise, it's something we could more simply avoid by making a 'sandbox' table exposing a limited subset of exponents.[/quote]

I insist on using JSP/Servlets technology. Aside of the frequently mentioned benefits (excellent portability, great performance resulting from powerful realtime compiler (BTW java runtime includes FFT based math package - it's trivial and fast to check in realtime if the factor submitted is a correct one, or alike)), all the tricks are implemented trivially. Here is basically the idea:

We define an application-wide attribute, container object (hashmap based?). As the very first action of processing web request, is quering the container regarding the amount of accesses seen last hour (or last minute). If it exceeds threshold, the user sent to the forest. Otherwise add a new record into the hashtable and serve the request in a usual manner. Periodically (when container becomes large, or just was not maintained a while), the container is scanned and too old records are wiped out. If that is not sufficient, records regarding ip addresses that made just few accesses (an expected case) are removed too; those ip addresses that submit noticeable traffic might have records aggregated using small intervals.

Things like that will add just few dozens of CPU cycles to processing web request. Which other web technology can provide similar flexibility (except maybe embedding application-level C code into Apache)?

[quote="Old man PrimeNet"][u]Web server layer?[/u] - We get free logging, connection handling & web hosting features by staying with a CGI or ISAPI interface, but we can also cut it from the loop this time and take sockets directly.[/quote]

Let's instead use server-side java. Its performance is so good that things like coding a custom HTTP implementation are not needed at all, even for highest traffic tasks.

Also, JSP/Servlets can be extremely easily mapped (deployed) in a way to emulate CGI, to support the old clients.

[quote="Old man PrimeNet"][u]Open source client builds, trusted or open?[/u] - Do we propose handling this differently? Today, only folks holding a trusted key can build binaries the server will accept. If we keep it, the key & encoding method should be strengthened. If not, how do we manage providing open source without giving folks a direct door for malicious network access?[/quote]

Why don't allow easily registering new client build keys at the server? (I don't mean allow it everyone in the world, this should be coordinated, and there should be stict limit on how many keys an independent developer can have hosted at server). But I'm still not sure if it's really neccessary.

[quote="Prime95"]Another security idea: each reservation should come with a 32-bit key written to worktodo.ini. Without this key you cannot unreserve the exponent (or report a result and obtain cpu credit?). This is to fight malicious unreserving and poaching reserved exponents.[/quote]

Sounds good. But maybe it's sufficient to accept exponent result only if submitted by user (or user+computer?) who was assigned the exponent?

Re: A few more loose ends
 

[quote="Old man PrimeNet"]how do we manage providing open source without giving folks a direct door for malicious network access?[/quote]

Basically, if GIMPS:

- start doing 3, or 4 (or 5? or 6? hmm) tests for each expoenent;
- accepts results only from user (user+computer?) work was assigned to;
- assigns exponents in a way that each exponent is tested at least once by some user who have long-standing participation history and thus can be trusted enough;
- has zero tolerance to the fake results submitted, and if that happens then results, stats, and account itself are promptly wiped off from the database (make discount for hardware failures that came unnoticed tho);
- makes testing results public only after all the 3 (4, 5) tests have been independently made, to ensure that one can not steal a correct result found by someone else

This all combined should be pretty safe (at least, assuming that over 50% participants do not cheat). The question is, will such painful step toward 100% open software increase GIMPS speed notwithstanding the extra tests done? At the moment it does not look so at all; but who knows, it might change eventually - when that happens we might go into details on this.

From the other side, accepting practice that residues become public only after doublecheck completes might be neccessary right now. Currently, PrimeNet server masks the residues, but with every database synchronization they go into George's database, and there residues are not masked. There might be quite a few expoenents there that are considered as double checked but factually LL test have been run only once (or even zero times, with lower probability).

Re: A few more loose ends
 

[quote="aga"]
- accepts results only from user (user+computer?) work was assigned to;
[/quote]

Ouch, that would hurt intrateam trading. If I have a computer die and won't be able to bring it back up, I would wish those exponents to be finished by a team member.

I really like the idea of adding a "security code" to an assignment. Something like an MD5/SHA of a date/time+random seed+M(xxx) should work well.

One minor problem with that would be if I had a machine die, and I don't have the security code, I couldn't give that assignment to someone else. Maybe allow for the "owner(s)" of a Team to allow reassignment to a particular user within the team. Either that, or I see a team keeping a DB of assignments and security codes. :)

That's an idea. If a client is a member of a team, have a couple of options:

1) says get assignment from Server and/or get assignment from Team Pool. If there are exponents that are not available in the team pool (if option checked), then get some from the server and assign it to the team and the user.

2) Allow for "Team Reassignment" checkbox/option. This way the team owners could move work from rogue borgs or dead machines to other people, but only for those who would want to have this feature. Then, if the client comes back and re-checks in, it could see's a status flag when trying to update that says "team reassigned", so it would dump that work assignment. If you don't have the allow team reassignment clicked, then the team owner can't do anything with it, letting an exponent expire normally.

This would help the poaching issue, as well as allow for mass assignments of exponents (say for a Gauntlet or other competition with a block of released exponents given to the team via the server).

Re: A few more loose ends
 

[quote="Paulie"][quote="aga"]- accepts results only from user (user+computer?) work was assigned to;[/quote]
Ouch, that would hurt intrateam trading. If I have a computer die and won't be able to bring it back up, I would wish those exponents to be finished by a team member.[/quote]

If that's the only computer person possesses, he/she might have problems giving checkpoint file to someone else. Is there is no checkpoint, just let exponent to timeout. There is an infinite supply of exponents, so other team members will not starve having few exponents timed out.

If that's not only computer, why don't just move checkpoint to another computer?

If it is really really neccessary move checkpoint file to a different user's computer, I suggest moving it together with ini files and installing a new copy of client-side software in order to finish the run.

Do computers die that often?

After all, does it really worth involving human labor in moving exponents around? (Isn't it better to spend the time earning few extra bucks, and then install an additional computer to run GIMPS software - that will be much better time contribution for GIMPS, even if it will take years to purchase the new computer. This will also provide financial support for CPU vendor(s) to further optimize and speed up CPUs.)

Do I miss something?

Re: A few more loose ends
 

[quote="Old man PrimeNet"][u]Web site manual forms access[/u] - I've seen a few posts here and previously on the Mersenne list suggesting we limit hits by IP-count-per-hour or similar. Is there a quick & easy way to configure this? Otherwise, it's something we could more simply avoid by making a 'sandbox' table exposing a limited subset of exponents.
[/quote]
Many of the suggestions for this involve some decision making and then checking hits against a threshold.

I think we should compare the threshold to a rolling average. This could get messy -- time or memory consuming. But we could fine tune it by increasing or decreasing the rolling window and the update rate. It could even be self adjusting based on resource consumption.

We could also keep an IP list for each user (hidden/encrypted?), with the threshold linked to the user, not the IP. The threshold would start low and go up with the users contributions. That way teams and farms or labs would automatically earn a higher threshold, and attacks with multiple IPs/machines/userIds would get choked off early. The attacker would have to contribute to the cause to increase his threshold!

Malicious use of open source software
 

Aga seems to worry about bogus results being returned from a modified client to pad one's stats.

I and I think Scott are worried about runaway clients eating through all the available exponents in a hurry.

Both concerns are valid, we will have to carefully examine each message that can be sent to the server and look for ways to minimize or eliminate possible damage from modified clients.

We should be able to identify fake LL results without requiring triple-or-more-checking. One new feature I want to add is to make sure the double-check is sent to a different userid and different team. We should also make sure that at least one LL result comes from a trusted prime95 client - this should be easy as 99% of results come from prime95/mprime executables built by me.

I don't know if the current server does this or not - but a DC assignment should *never* be assigned to the same user that did the first time check - and I think it should never be assigned to a teammate, if teams are allowed to swap exponents around.

requirements & more
 

[quote="aga"]It sounds easiest to deliver data from core server(s) to slave stats server(s) using realtime MySQL replication. ... MySQL replication allows to filter out data unneeded by stats servers - sounds like it does not cause security problems in this particular case. (Hereby I assume that there will be at least one core server that runs with a private MySQL engine;[/quote]
I read that mySQL replication can stream transactions to several peer servers concurrently. Can one filter for stats reporting, and the others as failovers not filter?
[quote="aga"]If don't filter replication log, then stats servers will also act as backups (even failovers?) for core servers, thus eliminating need of explicit backing up.;[/quote]
... Unless a corruption propagated and brought all the servers down. Then an occasional checkpoint backup sounds good again. Disk is relatively cheap, and until we are super-confident with the system all backup options are potential safety nets.

[u]Passwords in stats servers:[/u] No big deal? To me, the biggest risk having more servers in the picture is we might get sloppy with our maintenance procedures or server install/change configurations. If done right, however, I don't see much concern about stats servers having login passwords mirrored and used for detailed account viewing. You are right they would need to be protected within a trusted server, though. The v2-v4 systems have been like this for 5+ years, maybe not the best precedent but it has been ok so far.

[u]Security mechanisms cont.[/u] - I like the 32-bit key-per-assigned-exponent idea. Maybe we can do that for the clients, too ... I think someone else said that already - and revoke the access key if we think they are misbehaving?
[quote="Paulie"]I really like the idea of adding a "security code" to an assignment. Something like an MD5/SHA of a date/time+random seed+M(xxx) should work well.[/quote]
A random 32-bit int as a secret key is just as sufficient. Why do more?

[u]Throttling open-source client transaction rates[/u] - George is right that I'm mostly concerned with runaway/maliciously-coded clients. How do we propose handling someone exceeding a throttle rate limit? If they are getting all the way to the main database to decide someone is ok or not, we are taking the transaction hit already.

Here's an idea to whack at - What if throttling was managed by the web layer above the database? (Remember that the web layer can reside on the same server as the database, or on a different one.) The current v4 web layer authenticates the message hash signature of trusted clients. A v5 web layer could maintain a local, self-populated cache/table that tracked hit rates by IP & Unique Machine ID ("MUID"?), and respond by passing the transaction hit normally, ignoring it, revoking its client key (is that evil?) or replying with an error message - all without hitting the main database. The database would be self-maintaining in the sense that it records and gates access, but otherwise does not need loading or purging of data from the main v5 database. Actually I believe throttling can be router or firewall configured, too - any experts?

[u]JSP/Servlets technology[/u] - There are no requirements forcing a choice of Java over any other option. I've led a number of Java projects and believe me it's no cure-all as advertised.
1. First, the right design will make the choice of stateless http-to-database interface, JSP, ASP, CGI, ISAPI, etc., irrelevant anyway.
2. Migrating Pre-v23 Machines: My feeling is we would more likely use the existing PrimeNet code and tweak it to use the new database instead of rewriting that working legacy code in new JSP/servlets - or anything else for that matter. New efforts should focus upon the new protocol & design.
3. Manual Web Forms Pages - Maybe whomever did this sweet forum system should do the forms pages! :) In v4 we have a bunch of CGI Perl scripts invoking a special pnWebForm.exe CGI app to drive this, and from there RPCs into the main system. I'd be happy to see someone run this football downfield in any language... (hey, the Superbowl is in town here this weekend!)

[u]Nominal server load[/u] - Here's a scaling factor for our thinking: the current v4 system using 'slow' CGI sees between 2 to 3 transactions per second spiking to 10 per sec or so; it was stress tested at a rate of nearly 30x that rate continuously for several days.

[quote="aga"]Do computers die that often?[/quote]
Every day - or nearly so. GIMPS has been in a steady state since 1999, losing machines about as fast as new ones join, about 150 per day. It's hard to know when a machine is really 'dead' since they do not send a 'death' message to the server first.

Usually machines just go MIA past a reasonable waiting period, which is presently set in v4 to 90 days of inactivity, where activity is defined as any client transaction with the server. Sometimes we assume they are dead (and expire their exponents), and then they come back online and submit a test result and continue fine, but usually they don't come back.

It would be good to know why machines 'die' and fall offline. Many - perhaps most? - are simply being decommissioned by upgrades to new PCs. Others have a change in network environment that blocks its access to the server. Some folks just quit. Check out this recent chart from the v4 server - the data is not completely correct owing to account transfers, etc., but is reasonably representative:
[img]http://scottkurowski.com/images/pnMachineAges.jpg[/img]
There's a fairly uniform distribution of ages, except the first 4 months being risky or transfers. Not one is over 29 months old, at least as reckoned by machine ID.

Off-Topic query about "29 months"
 

(I would be quite thankful, sir, if you would kindly move that thing over to the right so it does not crimp my genuine Foamation cheesehat.)




(I would be quite thankful, sir, if you would kindly move that thing over to the right so it does not crimp my genuine Foamation cheesehat.)




(I would be quite thankful, sir, if you would kindly move that thing over to the right so it does not crimp my genuine Foamation cheesehat.)




(I would be quite thankful, sir, if you would kindly move that thing over to the right so it does not crimp my genuine Foamation cheesehat.)

[quote="Old man PrimeNet"]Not one is over 29 months old, at least as reckoned by machine ID.[/quote]

Hunh??

My "slow" system has certainly been reporting results to Primenet longer than 29 months. I've never changed my original userid or its original machine ID, and it's been steadily submitting accepted results as recently as last week.

oops errata
 

I updated the image, sorry. The updated chart makes a lot more sense. The oldest still-running node is, not surprisingly, my 'challenge' account's GIMPS_NODE0 at 70 months - the 200Mhz PII that was my main PC when I wrote PrimeNet in 1997.

I wonder what the big jump in machines around the 48 month ago mark was?

Re: requirements & more
 

[quote="Old man PrimeNet"]I read that mySQL replication can stream transactions to several peer servers concurrently. Can one filter for stats reporting, and the others as failovers not filter?[/quote]

Yes, that can be done; all slaves will receive the same replication stream, but each slave can be configured to ignore particular database/tables updates. Bad thing, that you can not configure on the server that 'this table content should not be replicated to the slave 2, but should be to the slaves 3 and 4' - this is a security concern.

[quote="Old man PrimeNet"]... Unless a corruption propagated and brought all the servers down.[/quote]

Yes, this should not be ignored. Good side is that MySQL logs updates within binary log file(s); if you have an old snapshot and all the consequent update log files, you can get database snapshot corresponding to any point in time - by simply starting with the old backup, and applying the updates until desired time point. Thus what should be backed up regularly, is the update logs (they are generally compressed by bzip2 pretty well, thus disk space and even bandwidth to regularly pull the logs to remote storage should not be a problem at all, that will be just a fraction of traffic server sees).

[quote="Old man PrimeNet"][u]Passwords in stats servers:[/u] No big deal? To me, the biggest risk having more servers in the picture is we might get sloppy with our maintenance procedures or server install/change configurations. If done right, however, I don't see much concern about stats servers having login passwords mirrored and used for detailed account viewing. You are right they would need to be protected within a trusted server, though.[/quote]

Well, there is no mean to prevent stats servers from gathering passwords seen; so maybe, it will be sufficient to store not plaintext passwords, but MD5 hash (or at least, send stats servers the hash instead of plain text apssword). This is slightly less secure, but will avoid realtime dependency of stats servers upon core servers.

This basically raises more general question: what the passwords are used for. Currently they are for stats viewing only. With v5, they are going to also authentificate team affiliation changes. I think it's also reasonable to host 'newsletter subscription' stuff at server, too - embedding it with client sounds a bit strange (but I well understand why it did happen that way). And probably few other things?

There are few actions I can think of that might require to keep extra information at stats servers (where we ought keeping only information that's really needed, any way). For example, email addresses. That's is not really needed to show stats. But it will be needed for the usual function of mailing a new random password instead of forgotten one.

Maybe, it will be desirable to use 2 passwords per user, one for browsing stats, and another for tweakiing different things. Let me remind about the idea I expressed while ago: use 3 kind of servers/modules, not 2. The additional server(servers, web application) is for tweaking core database (like password changes), thus stats servers are leave only with stats processing and displaying work and nothing else. I'm still not sure which structure is better for a partiuclar task; we need to describe in details all operations/actions provided for GIMPS clients prior making a decision.

[quote="Old man PrimeNet"][u]Security mechanisms cont.[/u] - I like the 32-bit key-per-assigned-exponent idea. Maybe we can do that for the clients, too ...[/quote]

I still do not understand what's the benefit doing that? I.e. what does key solve, it alows the exponent to move from user to user; do we reallly need that? In my opinion, that just opens few extra possibilities to cheat.

Tho... if the exponent key will allow to avoid an extra password, it might be a worthy thing to use.

[quote="Old man PrimeNet"]A random 32-bit int as a secret key is just as sufficient. Why do more?[/quote]

My quick math shows that to ensure probability of key compromization at level of 0.01% or below against of non-distributed attack, 33 bits of enthropy are quite sufficient. So 32 bits sound ok.

There is another approach possible: each assigned expoenent might be provided with a digital signature (with exponent value, and test number (0 for initial LL, 1 for doublecheck, -56 for fastoring up to 56 bits, etc) signed). But: the private key is used only when exponents are added to the database/marked for additional testing, thus it's possible to avoid keeping the key at all servers. And using a public key, any core server will be and to easily check if the returned exponent AND the work done match the signature (this sounds very useful). This check is just CPU-bound and cause no disk I/O, thus is very robust against DoS attacks.

(I should also mention that this does not involve using cryptographics modules within GIMPS clients - the signature is not more harming than browser cookie; thus no problems with copyrights and laws in certain countries).

[quote="Old man PrimeNet"]1. First, the right design will make the choice of stateless http-to-database interface, JSP, ASP, CGI, ISAPI, etc., irrelevant anyway.[/quote]

You just was mentioning traffic throttling code. With java it goes very naturally - you create a class, with interface consisting of a single method like:

boolean isok (InetAddress ia);

that returns true is the request should be served or false if the requestor should be sent to the forest. The class internally uses static object(s) to store state. Now you call the method at the beginning of each JSP or Servlet:

<%
if (!Throttler.isok(request.getRemoteAddress())) {
out.println("ERROR 12345: you should go to the forest this time");
return;
}
%>

And the rest of the JSP/Servlet code is not affected by the traffic throttling feature at all. What else can be used without sacrificing performance and portability?

[quote="Old man PrimeNet"][u]Nominal server load[/u] - Here's a scaling factor for our thinking: the current v4 system using 'slow' CGI sees between 2 to 3 transactions per second spiking to 10 per sec or so; it was stress tested at a rate of nearly 30x that rate continuously for several days.[/quote]

Well, this does not look much, especially if we remember that it's 2-way SMP box. But for v5 there are already at least 2 CPU-bound additional features considered, and more interesting ideas might pop up. So I'd fel safe knowing that server(s) can handle at least 1000x traffic over anticipated level.

BTW, one more CPU-bound addition: the interserver traffic is going to be highly compressable (as limited set of messages gets repeated again and again with just minor changes); together with the fact that there is no need for instant replication it would be wrong to miss the opportunity compressing the network traffic. This can be done by tunneling traffic over SSH; or piping via gzip; but I personally prefer using java.util.zip.* standard package (which is basically interface to libz) - I bet I need 30 seconds to implement transparent compression of replication traffic that way :) and then the code does not incur overhead like extra CPU context switches.

[quote="Old man PrimeNet"][quote="aga"]Do computers die that often?[/quote]
Every day - or nearly so.[/quote]

And all of them are willing to reassign the exponent (with particular checkpoint file) to someone else? No. Those deads you are talking about didn't even run the 'quit GIMPS' function of mprime/prime95.

But the graph is interesting, thank you.