Replacing Entropia's primenet server

[Prime95]

Originally Posted by aga

Well, getting to the bleeding edge of technology can not hurt...

It can if you're using new database features and all the bugs haven't been found!

While an array of redundant servers is great in theory (and can even reduce costs), I'm not familiar how MySQL implements this. Can you lead us through an example of how reservations are coordinated among several servers? Does each get a block of 1000 to handle independently or do the multiple servers coordinate to always assign the smallest available exponent?

[adpowers]

I thought their would be me one main server that keeps the master database while the other servers report back to it. The lesser servers would contact the main server requesting like 1000 exponents. Everyday, the server would contact the main server, report back updated status reports, send returned exponents, and ask for some new exponents to bring the total back up to 1000. That is my opinion for how it should work. My $0.02USD.

[barcode]

I'm interested in how this will work too. There are any number of ways of doing it... It depends really on what our requirements our. How much do we want to micromanage the exponent allocation. Do we want to be able to prove numbers are the n'th Mersenne in the shortest time possible at the expense of say redundancy.

With the method of allocating 1000 exp. to each server and reporting back each day then the stats will always be up to 24 hours out of date. How about each server caching 1000 and each time it assigns one it can pick up another from the master server (and report the completed exp. back if necessary).

Now the question arises on how we assign clients to servers and how we update clients to know about new servers that join the network and old ones that leave. Not difficult but it needs some thought.

[Old man PrimeNet]

Hi all, Just read most of this thread. I've been talking with Entropia about PrimeNet and changes may be on the way. I'm giving the open source approach a lot of thought, and I think it's possible. More later. -sjk

[cperciva]

Originally Posted by Old man PrimeNet

Hi all, Just read most of this thread. I've been talking with Entropia about PrimeNet and changes may be on the way. I'm giving the open source approach a lot of thought, and I think it's possible. More later. -sjk

Uhh, this may seem impolite, but... who are you? (maybe it's obvious to everyone else, but I'm suffering from both having had wisdom teeth taken out and having had several blood tests taken recently, and my brain isn't functioning perfectly right now)

[aga]

Originally Posted by adpowers

I thought their would be me one main server that keeps the master database while the other servers report back to it. The lesser servers would contact the main server requesting like 1000 exponents. Everyday, the server would contact the main server, report back updated status reports, send returned exponents, and ask for some new exponents to bring the total back up to 1000.

This sounds like proxies. Some GIMPSers might find them useful (first of all, the top producers who keep the computers within a single location), but proxies look like an optional low-priority addition, not replacement for multi-servers configuration.

[Joe O]

Cpervica,
I'd be willing to bet that "Old man PrimeNet " is Scott Kurowski!
Joe O.

[Joe O]

Originally Posted by aga

This sounds like proxies. Some GIMPSers might find them useful (first of all, the top producers who keep the computers within a single location), but proxies look like an optional low-priority addition, not replacement for multi-servers configuration.

Don't knock proxies! Distributed Net used 18 - 21 proxies in addition to its master keyserver very effectively. They did also provided "personal proxies" for the "top producers" and teams. These were just stripped down versions of their proxies so it wasn't that much more work for them. In GIMPS terms, the "personal proxies" could be limited to 100 exponents instead of 1000 for example. Properly done, this could significantly reduce the network traffic, as well as provide a buffer for master server outages.

[cperciva]

Originally Posted by Joe O

Don't knock proxies! Distributed Net used 18 - 21 proxies in addition to its master keyserver very effectively.

GIMPS isn't distributed.net.

Properly done, this could significantly reduce the network traffic

... which is already insignificant ...

as well as provide a buffer for master server outages.

... which is already done by clients.

[Joe O]

[quote="cperciva"]GIMPS isn't distributed.net.[quote]
That's true, Distributed.Net was/is a significantly larger project in terms of participation! We can learn from them.

[aga]

Originally Posted by barcode

I'm interested in how this will work too. There are any number of ways of doing it... It depends really on what our requirements our.

Yes, we really don't need diving down to the gore details of a particular implementation immediatelly, but start from upper levels.

How much do we want to micromanage the exponent allocation. Do we want to be able to prove numbers are the n'th Mersenne in the shortest time possible at the expense of say redundancy.

Originally Posted by barcode

With the method of allocating 1000 exp. to each server and reporting back each day then the stats will always be up to 24 hours out of date. How about each server caching 1000 and each time it assigns one it can pick up another from the master server (and report the completed exp. back if necessary).

Yes, this sounds closer to the ideal solution. But I intend to completely avoid 'master server' and make servers truly symmetric. RAIS should satisfy the following requirements:

- it does not matter which server a particular client connects to. Algorithms should be designed in a way to minimize (ideally, avoid) interrequests dependencies.

- nevertheless, under normal conditions after server handles transaction for a particular client, the transaction should become visible on neightbour servers within few seconds. If some server goes down, we don't really want it to carry away 24 hours worth of work.

- any server should be able to continue serving network traffic even if it finds that it's alone survived. But: if a single server falls off from network, it should shut down. Will be shutdown treated as temporary (allowing it to catch up later) or final (will require taking new database snapshot onto the server) will depend on feedback received from other servers when the single server eventually manages to connect somewhere. 'THe only survived' and 'isolated alone' is just a light difference. I think, we could use this metric: 'if server can connect DNS servers (that host a particular domain, or root DNS servers), then it continues working awaiting for other servers to become reachable to synchronize database; if it can not get to DNS servers, it assumes that it got disconnected from internet and should suspend or cease operations'. This is still not final (for example, how to treat situation if only part of DNS servers are accessible?) but generally looks good - DNS is inavoidable entity, and using it as a global coordination point is very natural.

- it is not really needed that RAIS assigns lowest exponent with exponent reservation transaction. It's sufficient that (1) the assigned exponent is not noticeably larger than mimimal unassigned; and (2) minimal exponents does not stay inassigned indefinitelly long. In fact, there should be special rules for least unassigned exponents, to ensure that they are getting assigned only to computers that will return them fast.

Well, I might have missed some details, but you got the idea. Good news is that my preliminary investigation showed that all the basic operations, like exponent assignment, exponent return, exponent update can be prety easily implemented in a way to satisfy all the requirements - in fact, they are not that restrictive as might look at a glance.

Originally Posted by barcode

Now the question arises on how we assign clients to servers and how we update clients to know about new servers that join the network and old ones that leave. Not difficult but it needs some thought.

Assigning clients to servers does not sound good. Every server should be capable of serving any client. There are two methods managing set of servers: first, it just keep alive servers in round-robin DNS; if some server dies off and is not going to become alive again, it's just excluded from round-robin DNS. When new server gets deployed and ready to accept traffic, it's added to DNS zone.

Another approach would be to use dnet-like approach, where client tries contacting geographically closest server (relying on timezone configured at client computer). Just GIMPS client should always reattempt connection if closest known server does not respond (dnetc just gives us). This is not an alternative actualy, but just an improvement option, not of high priority.