Replacing Entropia's primenet server

[cperciva]

Originally Posted by aga

If hundreds users/computers are unable to get/return exponents and there are idle computers around, I think that's failure; do you?

Approximately, yes.

If you are a well-funded business with a non-mission critical resource, yes.

I must be missing something here. How would running several servers all the time be cheaper than running one and replacing it if it fails?

Keeping a set of homogeneous servers is anyway a solution that provides better availability, better performance, better resists to DoS attacks, and best use of available bandwidth (if client-sde siftware first tries to access topologically closest server).

The server load should be low anyway -- assigning exponents is not difficult. The bandwidth is insignificant.

As MySQL allows realtime replication, that is the most natural feature to use; it ought to be used notwithstanding if there is a single server or a dozen.

IIRC, George's background is database programming. I think he'd be happier using a database.

[aga]

I must be missing something here. How would running several servers all the time be cheaper than running one and replacing it if it fails?

GIMPS server is going to be hosted at hosting resources donated by GIMPSers. If we run several servers, then we always know how many active servers we get, and know detailed stats regarding reliability of each one. If one goes down, there is sufficient time (weeks, even months) to find a replacement, install stuff there, test it, and put into production. Note that as there is no need for excellent availability for each partiicular server, GIMPS could effectively use less-then-perfect hosting solutions like old Pentium server behind a home internet connection.

If instead a single server fails, what do we do? Might sound like pre-building a list of those who are willing to donate hosting resources is the solution; in fact, it is not. When server goes down and quick replacement is needed, expect that some people will not answer (at least, will not answer timely), some will tell 'sorry, the offer is no longer available, things changed over here', and even if there are a suitable offer left, it might be that at last moment some hardware or system software problems/incompatibilities will be dicovered, and you GIMPS can not afford purchagsing commercial dedicated server/clolocation, project is stuck. Even if there is a perfect substitution server discovered, it means that someone will need to rush and restore/install everything. We very hope the person will not be on Hawaii. But any way, do you remember that the work is not-paid? There no warranty that a single server will be reinstalled fast!

The server load should be low anyway -- assigning exponents is not difficult. The bandwidth is insignificant.

Looks like bandwidth will be pretty low - usually, and that's good. It's still might be good if European GIMPSers use server in Europe, that gives noticeably better roundtrip and thus less affected by network problems.

But if someone decides to flood GIMPS server, the bandwidth usage can grow very high! If we run several servers, then effectively flooding all servers at once sounds quite unrealistic.

As another idea, I'd like to have stuff designed in a way that more math algorithms can be handled; maybe few that require much higher bandwidth than currently used algorithms. Why don't to start with a state-of-the-art design of the server-side stuff, instead of first cloning entropia and then throw the work away and redoing it from scratch again? After all, entropia server still works, there is no immediate rush.

But that's bandwith. I remember that someone told that the current dual-CPU Entropia server often gets 100% (i.e. 200%) continuous CPU load. While I'm not completely understand why does it happen (maybe poor database design or SQL engine performance, or maybe flooding of stats requests), I would prefer to have configuration where several servers handle the load cooperatively, and if something gets misestimated or GIMPS starts growing fast the problem could be quickly solved by upping an additional server, without need to rush and recode software (or spend thousands bucks for high-end server).

IIRC, George's background is database programming. I think he'd be happier using a database.

Not sure I understand - MySQL is SQL based database engine. Noone is really going to write own database engine, there are better means wasting life.

[cperciva]

Originally Posted by aga

Not sure I understand - MySQL is SQL based database engine.

No. MySQL is an SQL based file system. It isn't a database.

(Of course, databases aren't always necessary, and for such tasks, MySQL performs perfectly well. But I don't think this is such a task.)

[aga]

Originally Posted by cperciva

No. MySQL is an SQL based file system. It isn't a database.

Sorry? Please read MySQL documentation, especially innodb tables handler section. Not that innodb features are needed for the task tho, and it is not innodb that defines software category MySQL falls in. So I understand you even less.

But if you are simply trying to start a next religious war, I will let you to fight down yourself. As it is not going to have a practical effect, as all the previous religious wars.

[Prime95]

Do we need to start a new thread on this?

I'm not an expert on either of these databases. I did install PostgreSQL here a year ago and satisfied myself that it had all the features I need to implement the reservation system and results logging.

It seems to me that there is a lot more development effort and interest in MySQL than in PostgreSQL. InnoDB has added transactions and row-level locking to MySQL. Subqueries are scheduled for implementation in the near future. If we have outer-joins or views we can probably work around this limitation.

We should develop the list of features we want/require of our database and do some research into which best meets our needs.

[Prime95]

Originally Posted by aga

Keeping a set of homogeneous servers is anyway a solution that provides better availability, better performance, better resists to DoS attacks, and best use of available bandwidth

I like the idea of multiple servers if it doesn't put us on the bleeding edge and does not increase costs significantly. My biggest fear with the single server approach is the down time associated with a hardware failure. If the server is remotely located it could take days to get it fixed and the software restored from backup. Obviously, this isn't a fatal problem to GIMPS but it is a major annoyance to users.

Multiple servers has other problems to overcome. More development time, qualifying server operators as interested enough to keep the server upgraded/maintained for years to come, etc.

[QuintLeo]

MySQL is probably the second or third most common SQL database system in use today (behind Oracle and probably DB2, might be 4'th behind Sybase).


I'm looking at a fairly high probability of moving sometime next summer, but I'd be willing to host a server after that if it is decided to go the multi-server route. Remote access won't be a factor unless I'm on vacation - I tend to be around every day.

9-)

[adpowers]

Two things. First, I agree with the multiple server part (I think I was talking about it earlier in this thread). I would be willing to host one of the servers on my home connection. Secondly, I believe the reason the server gets maxed out is because it hosts the stats. If we had a multi server design, I think we could get away with one main server doing the stats (because they are less important then saving results and assigning exponents).

[aga]

Originally Posted by Prime95

We should develop the list of features we want/require of our database and do some research into which best meets our needs.

To begin with, we should agree upon services GIMPS server/cluster will provide for GIMPSers. That's business layer, and database is just technical, backend layer.

One of the biggest question that will influence many technical details, will GIMPS server keep entire database regarding Mersenne numbers (and thus automatically advance exponents to P-1 queue after trial factoring run ends, to LL tests after P-1 factoring step passes, then to doublecheck queue, if neccessary to tripplecheck, automatically build the http://www.mersenne.org/status.htm table, and so on),

or should it just track subset of assigned exponents, and as soon as exponent gets checked, it gets exported from the onilne database, and then go to the offline database you currently maintain; then you potentially can assign the exponent back to GIMPS server if need arises?

I can think of pros and contras with both approaches; but as you already handle the GIMPS database for years, I'd definitely like to hear your opinion on this.

[aga]

Originally Posted by Prime95

I like the idea of multiple servers if it doesn't put us on the bleeding edge and does not increase costs significantly.

Well, getting to the bleeding edge of technology can not hurt, will give extra publicity for GIMPS and attract more people to participate.

There is also this issue: if we use... well... Redundant Array of Inexpensive Servers :) then with server-side software there is no need to imlpement any dirty performance optimization tricks, we can keep it academically clean and easy-to-maintain; and performance problems can be solved by simply adding/upgrading servers. That is assuming that interserver comminication will depend linearly or at most nlogn on total GIMPS traffic - according to my preliminary research this should not be a problem at all.

If instead we limit ourselves to a single-server solution (and unlikely the server will be high-end), then it might be essential to do do things like inJVM datacaching and other tricks that make software looking like a can of worms.

I'm surprised regarding the 'increase costs'. I didn't expect that development efforts will receive monetary compensation above US$0.02. :) And as long as we can effectively use low-end servlers, it looks like there are enough people willing to donate some resources to GIMPS.

Originally Posted by Prime95

Multiple servers has other problems to overcome. More development time, qualifying server operators as interested enough to keep the server upgraded/maintained for years to come, etc.

Development time should not be too high.

If it becomes impossible to maintain a particular server, we just exclude it from RAIS and let it die off. Deploying web application onto a new server should be pretty simple, by adding the new neighbour to alive servers, uploading .war file to the new server, specifying server id and db login within configuration files, and prepopulating the database with a snapshot taken from some other alive server - most of that can be automated.

[Deamiter]

as a side note, I'd be willing to contribute MY US$0.02 to the cause! :(