mersenneforum.org  

Go Back   mersenneforum.org > Prime Search Projects > No Prime Left Behind

Reply
 
Thread Tools
Old 2009-12-29, 00:02   #12
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

7×11×73 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
I'm confused. Why not just take the CPU time output from LLR or PFGW or whatever on the client side? That's what LLRnet does with LLR. It's quite accurate unless there is an interruption on the server or client side, in which case, the time starts back at 0 for the current test upon restart, which is no big deal because only the time for that one test is too low. Even with that, 99%+ of CPU timings are accurate and you don't have to do any kind of internal calculation.
I could do that, but I am concerned about the marginal case where the client is stopped and restarted. I should be able to detect that condition and send a zero or nothing at all.

Now for an all important update. The coding is done! I let the kids play on their DS most of the day so that I could crank through the code. The changes on the server side are massive, but I'm relatively pleased with how it has turned out. The server code has been almost completely rewritten. The only pieces that were barely touched were the socket and logging code. Note that I'll probably change my opinion of that in the future as I realize that I could have done things much better...

Nevertheless, I hope to do some alpha testing tomorrow. The only standing in my way if my wife. I have a (short) honey-do list. We'll be moving in 2010 and there is a number of things we need to do to get our house ready for sale. I should be able to accomplish most of those tomorrow. I'm hoping (fingers crossed) that I can start up the new server and get a few clients to pound against it during the day while I'm painting.
rogue is offline   Reply With Quote
Old 2009-12-29, 07:05   #13
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

235568 Posts
Default

Quote:
Originally Posted by rogue View Post
I could do that, but I am concerned about the marginal case where the client is stopped and restarted. I should be able to detect that condition and send a zero or nothing at all.

Now for an all important update. The coding is done! I let the kids play on their DS most of the day so that I could crank through the code. The changes on the server side are massive, but I'm relatively pleased with how it has turned out. The server code has been almost completely rewritten. The only pieces that were barely touched were the socket and logging code. Note that I'll probably change my opinion of that in the future as I realize that I could have done things much better...

Nevertheless, I hope to do some alpha testing tomorrow. The only standing in my way if my wife. I have a (short) honey-do list. We'll be moving in 2010 and there is a number of things we need to do to get our house ready for sale. I should be able to accomplish most of those tomorrow. I'm hoping (fingers crossed) that I can start up the new server and get a few clients to pound against it during the day while I'm painting.

That sounds great Mark. Good luck with the alpha testing. Just to confirm: You are testing the hyper-threading. Is that correct? Once you iron out a few wrinkles as needed, I'll assist in testing as necessary.

Quick comment about the CPU timing: I would suggest not worrying about it if the client or server is stopped and restarted. Still just show whatever time is output by LLR or PFGW or whatever program. Sure, the time can be potentially way too short, but it's been that way for years in both LLR and PFGW. If you have a bunch of 900 sec. tests mixed in with a couple of 200 sec. tests, it's clear that that means there was an outage or stoppage in the middle of the latter...not a problem at all. The main need is to be able to estimate future workloads and to verify that our machines are running at full speed. It would be obvious that we'd use the 900 sec. time to do such estimates, even with the few 200 sec. times mixed int here.

Technically it's up to the original programs to correct that problem. I don't think we should be having to code around it on the back end of things.


Gary
gd_barnes is offline   Reply With Quote
Old 2009-12-29, 07:56   #14
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

3×2,083 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
That sounds great Mark. Good luck with the alpha testing. Just to confirm: You are testing the hyper-threading. Is that correct? Once you iron out a few wrinkles as needed, I'll assist in testing as necessary.
Er, multi-threading. And, yes, it sounds like that's what he's testing now.
mdettweiler is offline   Reply With Quote
Old 2009-12-31, 04:14   #15
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

7·11·73 Posts
Default

It's time for another update.

I've been running a number of tests today and so far everything looks good. I have two computers hitting the one server (four clients). Although I made the changes for *nix, I have not tried to compile/run on any *nix system yet. I hope to get to that by the weekend. The worst part I've run into is a bug in the MYSQL ODBC driver on Windows. It is a critical bug that has to be resolved because the server crashes every few hours because of it. I think I have a workaround for it. I'll have a better idea if my workaround works in a couple of days.

On a positive note I discovered a bug with the mail code that goes back a few versions. It will be fixed in 3.0. Anyone who is interested in patching 2.4.7 can PM me. The changes are really simple.
rogue is offline   Reply With Quote
Old 2010-01-22, 19:06   #16
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

562110 Posts
Default

PRPNet 3.1.2 appears to be very stable. I have a patch for an issue in 3.1.2, but the issue it addresses does not affect stability of the software. I am not aware of any other issues. What are the current thoughts about PRPNet 3.1.2 and whether or not it is ready for prime time, specifically if it is ready to replace LLRNet?
rogue is offline   Reply With Quote
Old 2010-01-22, 19:12   #17
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

141518 Posts
Default

Quote:
Originally Posted by rogue View Post
PRPNet 3.1.2 appears to be very stable. I have a patch for an issue in 3.1.2, but the issue it addresses does not affect stability of the software. I am not aware of any other issues. What are the current thoughts about PRPNet 3.1.2 and whether or not it is ready for prime time, specifically if it is ready to replace LLRNet?
The next step on this is for me to set up a test server for this with some small candidates; I've been kind of busy lately and haven't had the time to do this, but will make a concerted effort to do it within the next couple of days. Stay tuned.

Once I've done that, the plan is for Gary and I to dogpile on the server with all of our cores (as well as anyone else who wants to). If the server can handle the load for a few hours with 50+ cores on it, then we'll be satisfied that it's fully ready to roll.
mdettweiler is offline   Reply With Quote
Old 2010-01-26, 02:21   #18
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

7·11·73 Posts
Default

It's time to resurrect this thread. There have been a number of requirements requested in other threads to bring PRPNet closer to LLRNet in functionality. I want to keep all of that in this thread because it is easier for me to keep track of.

Going forward, I am going to ask for more details for each requirement because there have clearly been some misunderstandings because I haven't asked enough questions and made too many assumptions.

When I am given a requirement, I prefer that as much detail can be given as possible. If the requirement means that PRPNet creates a file or webpage, then I need to know the data elements in that file/webpage. If a file is to be created, whether or not it resides on the client or the server. I want well thought out reasons for each requirement. IMO, "because LLRNet does it" is not a very good reason.

Given all of this, I will work with you to flush out all of the details before I begin coding. A misunderstanding can cost me a lot of development/testing time and lead to everyone else's frustration.
rogue is offline   Reply With Quote
Old 2010-01-26, 05:26   #19
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

2×72×103 Posts
Default

Version 3.1.5 presumably splits the results and the prpclient.log file and adds a test time to the results. That's a very good thing and is one of our requirements.

Here are some more:

Just one more on the client side:

1. I'm not sure if this was done with 3.1.5 but it would be helpful if not: Since the results have been separated out of prpclient.log into a separate file and what is left is only server messages, have the option to turn off those server messages. When we run 1000s of teeny tests, those messages likely slow things down a little and the file becomes inordinately huge. I've seen files as big as 50 MB after running small tests for just a few days.


All of these are on the server side:

1. In prpserver.candidates, we need that to contain only k/n pairs and nothing more. Move other info. to a separate file. If you want to leave the little "N 0 active" or "T xxxxx active" in there, that is OK but we need to have exactly one line per k/n pair. This allows us to get an exact count of pairs remaining. We need to know that more exactly when many people have tests reserved. Example:

601*2^745531-1 N 1264463645 active <--- keep this line since it hasn't yet been tested
601*2^745531-1 T 1264463645 kar_bonatxxxxxxxxx.xxx 3 na inprogress na <--- move this line to a separate file
601*2^745625-1 N 0 active
601*2^745711-1 N 0 active
601*2^745751-1 N 0 active
601*2^745789-1 N 0 active

To be more specific: Move the 2nd line above and all lines with the 'T' code in them to a separate file. This will make prpserver.candidates much like LLRnet's knpairs.txt file.

2. As specified in #1, create a new file, perhaps called prpserver.joblist, that has all of the info. shown in the 2nd line above. We need a separate joblist file to see at a glance if there is a problem client. This will make it the equivalent of LLRnet's joblist.txt file.

3. We need the prpserver.candidates to have the option to be sorted in entry sequence order -or- in n-value order. As it is now, it automatically sorts them by k-value. But it's difficult to get the beginning and ending k/n pair for the entire file when it does that. This is needed to determine if we have the entire n-range loaded and exactly where we are at in our testing and easily pull that info. to a web page. (See the web pages that Max has created in the 1st post of the "PRPnet servers for NPLB" thread. Notice that we can't easily pull the correct info. on those pages without a lot of coding due to the issues in #1 and #2.) Also, I'd just like to look in the file and see which pair is next in line to be tested.

Special note: If this "random entry order" option is chosen, then disable the small web page (or whatever it is) that shows the pairs remaining on each k and the min and max n-value of each. Otherwise, it will be a complete mess.

4. Have an option that allows testing the prpserver.candidates file in entry sequence order. Sometimes people want to search backwards. Other times, we might want to search all of one k and then search the rest of the candidates upward by n-value. In other words, just like LLRnet, we want to be able to test in the order in which candidates are entered in that file and have those candidates stay in exactly that order.

5. Rename completed_tests.log, onlytest.removed, and prp.removed to something more meaningful. Completed_tests.log is results for the day so how about daily_results.txt? Onlytest.removed is cumulative results so how about results.txt? Prp.removed is proven primes. Since they are removed, they are proven primes? Is that correct? If so, it could be renamed prime.txt.

One more thing:

What is the difference between prp.removed and PRP.log? To me, they are a duplication and only one is needed so the other could go away. Not a requirement; just an observation.

The main crux of all of the above is that on both the client and server side, we need each file to have it's own single "thing" in it. It is where results are mixed with server messages on the client side and where the equivalent of the joblist is mixed with knpairs on the server side that it causes confusion on pulling the info. that is needed.

Even if half of these things are done, it would help greatly. If the file names as they are currently is what is preferred by most others, then I understand. I'm going by what I've observed here and at a few other projects, although certainly not all of them.


Gary

Last fiddled with by gd_barnes on 2010-01-26 at 05:37
gd_barnes is offline   Reply With Quote
Old 2010-01-26, 11:54   #20
Mini-Geek
Account Deleted
 
Mini-Geek's Avatar
 
"Tim Sorbera"
Aug 2006
San Antonio, TX USA

17×251 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
Just one more on the client side:

1. I'm not sure if this was done with 3.1.5 but it would be helpful if not: Since the results have been separated out of prpclient.log into a separate file and what is left is only server messages, have the option to turn off those server messages. When we run 1000s of teeny tests, those messages likely slow things down a little and the file becomes inordinately huge. I've seen files as big as 50 MB after running small tests for just a few days.
From prpclient.ini:
Code:
// Size limit in bytes for the prpclient.log file.
//    0 - no limit
//   -1 - no log
loglimit=5000000
Isn't prpserver.candidates no longer used now that the MySQL version exists?
Mini-Geek is online now   Reply With Quote
Old 2010-01-26, 12:05   #21
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

2·72·103 Posts
Default

Quote:
Originally Posted by Mini-Geek View Post
From prpclient.ini:
Code:
// Size limit in bytes for the prpclient.log file.
//    0 - no limit
//   -1 - no log
loglimit=5000000
Isn't prpserver.candidates no longer used now that the MySQL version exists?
My ignorance of the MySQL version is showing. Without looking in the DB, where are the remaining k/n pairs shown? I think we want those in a file somewhere. But if the DB is easy enough to access and use and can clearly show what is remaining in the server, then that would work.

We just need to be able to easily access the equivalent of a knpairs.txt and joblist.txt files in LLRnet.

I just now looked in port 7465 that we recently used for version 3.1.4. I don't see the equivalent of those files anywhere. Help! Perhaps Max got rid of them when we started having problems with the server.

Now, I'm wondering if we'll be able to allow for a entry-sequenced testing order. IMHO, that's fairly important, especially for personal efforts.

I'll read some of the readme and documentation about the new server now.


Gary

Last fiddled with by gd_barnes on 2010-01-26 at 12:07
gd_barnes is offline   Reply With Quote
Old 2010-01-26, 12:10   #22
Mini-Geek
Account Deleted
 
Mini-Geek's Avatar
 
"Tim Sorbera"
Aug 2006
San Antonio, TX USA

17·251 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
My ignorance of the MySQL version is showing. Without looking in the DB, where are the remaining k/n pairs shown? I think we want those in a file somewhere. But if the DB is easy enough to access and use and can clearly show what is remaining in the server, then that would work.
Without looking at the DB? Um...nowhere, I suppose.
I'm sure someone experienced in SQL commands could make a command that gets the lists you want.
Quote:
Originally Posted by gd_barnes View Post
We just need to be able to easily access the equivalent of a knpairs.txt and joblist.txt files in LLRnet.

I just now looked in port 7465 that we recently used for version 3.1.4. I don't see the equivalent of those files anywhere.
Are you still trying to look for those in the PRPnet folder? Let me make this clear: With PRPnet 3, all info of uncompleted candidates is only in the DB.

Last fiddled with by Mini-Geek on 2010-01-26 at 12:11
Mini-Geek is online now   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
LLRnet and PRPnet servers for automated LLR mdettweiler Twin Prime Search 230 2020-04-01 03:30
LLRnet/PRPnet rally April 4th-11th mdettweiler No Prime Left Behind 55 2011-04-25 09:35
LLRnet/PRPnet rally January 3rd-10th mdettweiler No Prime Left Behind 48 2011-01-12 10:14
LLRnet/PRPnet rally Oct. 27th-Nov. 3rd mdettweiler No Prime Left Behind 33 2010-12-24 19:16
LLRnet/PRPnet rally June 4th-6th gd_barnes No Prime Left Behind 61 2010-07-30 17:28

All times are UTC. The time now is 22:00.

Sat Apr 4 22:00:53 UTC 2020 up 10 days, 19:33, 0 users, load averages: 1.52, 1.58, 1.58

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.