mersenneforum.org  

Go Back   mersenneforum.org > Prime Search Projects > No Prime Left Behind

Reply
 
Thread Tools
Old 2009-05-03, 01:54   #1
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

7·1,453 Posts
Default Manual stats import discussion

I have some exciting news:

We have now begun the long task of converting and loading manually processed results into our stats database.

See details in this post.


Gary
gd_barnes is offline   Reply With Quote
Old 2009-05-03, 02:04   #2
MyDogBuster
 
MyDogBuster's Avatar
 
May 2008
Wilmington, DE

22×32×79 Posts
Default

Quote:
We have now begun the long task of converting and loading manually processed results into our stats database.
Yeah. This will give Free-DC the CORRECT numbers to shoot for in overtaking ROLP. Now the numbers will mean something besides just the LLRNET totals.
MyDogBuster is offline   Reply With Quote
Old 2009-05-03, 03:26   #3
IronBits
I ♥ BOINC!
 
IronBits's Avatar
 
Oct 2002
Glendale, AZ. (USA)

111310 Posts
Default

Let me get this straight...
Basically, you guys got 1+ years head start before I even heard of NPLB.
I took over the whole project, server wise, when something happened and the project was in jeapordy.
I managed to get Bok and AMDave interested in assisting me with the database, while I cranked out code under Windows to make it all work.
Then AMDave started busting his tail to get everything migrated over to Linux and made all kinds of enhancements, and Bok added NPLB stats to his stats system.
With the new stats system being developed and working rather well...
You started a Team called ROLP, some of you banned together...
Then ROLP got stomped by Free-DC in a short rally, then all of a sudden, the stats system isn't good enough because it was only reflecting how many knpairs were being returned and some credit system had to be developed.
All good ideas and everything is now moving along nicely.
Another rally was called for and Free-DC declared another BBQ run, and whooped ROLP again.

Free-DC is now catching you so quickly in the daily Server stats, even with the new stats system, and I noticed more of the old timers join ranks with ROLP (nothing wrong with that, gives Free-DC more to shoot for)

Now you are adding in all the manual work and information into the database, which is a great idea!
Keeping track of who did what work is very important and a fantastic idea!

Somehow, the mixing of the manual results scores into the active server results scores, especially considering the lead time you guys had, is not such a good idea.
There is no doubt in my mind what-so-ever that you 'old timers' have put a lot of time and effort into the project, even before I heard about the project, and continued to do so even today with manual reservations.

Everyone watches the daily active stats, and when the last few persons joined ROLP, the stats took a big jump for those days, no biggy...

But, to add in all the manual reservations on top of that, and then tell everyone that is the true scores, even though they are very accurate,
is not going to go over very well with the majority of folks that have been watching the stats.
Especially Free-DC.
The target was set on May 1st.
By May 31st, we should catch ROLP, or be very very close, but now the whole Team will be demoralized as you keep adding in more scores for work you have done in the past that they never knew about...
I can't think any more and I'm sure you get my point.
I totally understand what's going on, agree with what you need to do to get it all into the database.
Problem is what to do about offline crunching versus online crunching.
There is an unfair (if you can call it that) advantage to the old timers in that they know how to do go about fuzzing about with manual reservations, getting more done with optimized clients that work faster than the clients used to work online crunching, which is what the majority of new users are going to be...

Simple version.
Started teams and started rallies to get the project moving
Challenges were made by ROLP.
Free-DC easily answers the call, beats ROLP.
Stats scoring system developed, ROLP takes a huge jump in the lead (no problem)
Another Rally is declared
Free-DC answers the call again and beats ROLP.
New challenge issued
Free-DC declares that the Project of the Month is going to be NPLB
Gap starts closing rapidly, old timers on the side lines join ranks with ROLP, one even jumped team affiliation (fine, no problem)
Now offline crunching will be added to the database (understandably so)

I feel like a carrot is being dangled just beyond my reach...

Suggestions?

Last fiddled with by IronBits on 2009-05-03 at 03:52
IronBits is offline   Reply With Quote
Old 2009-05-03, 03:50   #4
MyDogBuster
 
MyDogBuster's Avatar
 
May 2008
Wilmington, DE

54348 Posts
Default

Quote:
Somehow, the mixing of the manual results scores into the active server results scores, especially considering the lead time you guys had, is not such a good idea.
There are a lot of manual files done by Free-DC folks too. Carlos has a ton of them. I just think it's a truer representation of what we have done as a project.

I would have no problem showing the manual results seperately with maybe a summary showing everything combined. My only concern is to get EVERYTHING into the database as a project.

Besides, no matter what we do, it is probably just a matter of time before Free-DC rules the roost. LOL
MyDogBuster is offline   Reply With Quote
Old 2009-05-03, 03:53   #5
IronBits
I ♥ BOINC!
 
IronBits's Avatar
 
Oct 2002
Glendale, AZ. (USA)

3·7·53 Posts
Default

I fully agree the data must go into the database, just bad timing if it's done during the time that Free-DC is closing the gap on the current stats system.

Cock-A-Doodle-Do

Last fiddled with by IronBits on 2009-05-03 at 03:54
IronBits is offline   Reply With Quote
Old 2009-05-03, 04:01   #6
MyDogBuster
 
MyDogBuster's Avatar
 
May 2008
Wilmington, DE

22×32×79 Posts
Default

Maybe they can put off the load till after you kick our butts. I would vote for delaying the load.

Max - Gary. David has a valid point. They based their drive on the current stats setup so changing it wouldn't be kosher.

Last fiddled with by MyDogBuster on 2009-05-03 at 04:04
MyDogBuster is offline   Reply With Quote
Old 2009-05-03, 04:02   #7
IronBits
I ♥ BOINC!
 
IronBits's Avatar
 
Oct 2002
Glendale, AZ. (USA)

100010110012 Posts
Default

I was hoping for a better way to handle it - two stats systems perhaps
One for online (where almost all new users will be for some time)
One for offline (where the gurus know how to work the bothersome manual system, and get a speed advantage)
IronBits is offline   Reply With Quote
Old 2009-05-03, 04:05   #8
Brucifer
 
Brucifer's Avatar
 
Dec 2005

313 Posts
Default

How about just doing away with all team stats.................
Brucifer is offline   Reply With Quote
Old 2009-05-03, 06:15   #9
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

3·2,083 Posts
Default

Hmm...I see what you guys are saying. We had been planning to include all of the manual results in the stats for a long time hence, but you're right, this is probably a bad time to do it with all the Free-DCers counting on zonking ROLP in 31 days.

We'd considered having separate databases for manual and LLRnet, but that has a number of other issues. For instance, I'd always thought it would be really neat to have everything counted together in the "Drive Progress" scatter plot--if we included all results, manual and LLRnet, then it would be a better representation of the drives' progress.

Also, as Ian said, Free-DC has a lot of manual work under its belt, too--Carlos and Beyond both put in a heck of a lot of work manually. That would, most likely, somewhat balance out the increase that ROLP would get from including manual results in the stats.

Though, yes, I do agree that we should hold off for the time being, in light of the Free-DC project-of-the-month thing, so as not to confuse all those users who are watching the stats with an eagle eye yet aren't fully clued in to the existence of manual results. Gary, how would you feel about holding off until a week or so after Free-DC is done with their push? That way, it won't seem as much like the ROLP-affiliated admins are "cheating" at such a crucial time (which we're not, since it was planned for a while anyway, but as David said, it could look a lot like that if it was done right now).

Gary, I'll still download all the manual result files that you put up on crunchford as soon as I can--this will give me a chance to get a lot of the heavy lifting worked out so that as soon as we're ready to do the import, all I have to do is send the files to AMDave and we're all set.

Max

Last fiddled with by mdettweiler on 2009-05-03 at 06:16
mdettweiler is offline   Reply With Quote
Old 2009-05-03, 06:31   #10
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

100111101110112 Posts
Default

Wow, folks. This has been in the works for ages. Max has brought it up numerous times and we finally go to do it and now the objection comes? What's up with that?

Initially I objected because it will be a tremendous amount of work but Max has some automation set up for it and he has voluntered so we're going ahead with it.

Carlos, the big former Free-DC promoter here, was livid that we would NOT do it due to the large amount of manual work that he did. Therefore I'm frankly baffled why the objection now.

To make a project popular, we must have stats. They are the backbone of our project now. Therefore I propose the following:

1. Max, get all of the files properly created in LLRnet format with the correct UserID's, dates, etc. with the results that I have sent you. That alone will take a tremendous amount of time.

2. After you're done with #1, I'll send you more files for the 8th/9th/10th drives (much smaller manual work) as well as the individual-k drive. You can then hack away at those for an extended period.

3. If you are able to finish #'s 1 and 2 plus some additional straggling stuff that I'll send you like the mini-drive in < 3 months, then hold on to them until 3 total months has elapsed.

4. After 3 months; on August 1st, begin loading all of the files into the stats database.

David, by that time it's likely that Free-DC will be so far ahead that the manual loading of files won't matter. Keep in mind that as Max said, Carlos and several Free-DCers will have a large amount of manual files done from our 1st 3 drives.

Does that sound reasonable and politically correct to everyone?


Gary
gd_barnes is offline   Reply With Quote
Old 2009-05-03, 06:39   #11
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

7·1,453 Posts
Default

David,

In response to a couple of points in your post:

I apologize for the timing of this. We in no way intended to keep the carrot just out of reach. It's just the natural progression of the project. I hope that holding off on the manual stats import for 3 months will make it more palatable to people.

I had no idea that Free-DC had made NPLB the project of the month. That is VERY cool! A big thanks to you, AMDave, and Bok for your tireless efforts on the servers, stats, and promotion of the project at Free-DC!

Does my proposed solution seem like a fair one?


Gary
gd_barnes is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Oddity in P-1 data import into mersenne.ca DB Syntony PrimeNet 5 2015-12-10 12:22
CSVs for stats available + New combined stats opyrt Prime Sierpinski Project 3 2010-05-31 08:13
GMP-ECM Manual GrK GMP-ECM 2 2007-12-25 01:55
Manual ecm jasong Factoring 7 2005-08-29 19:17
P4 On Stats HiddenWarrior Hardware 2 2003-08-13 14:39

All times are UTC. The time now is 02:38.

Wed Aug 5 02:38:45 UTC 2020 up 18 days, 22:25, 1 user, load averages: 1.56, 1.85, 1.76

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.