NPLB Free-DC Stats

[IronBits]

Excellent! I got them all, so if you want to move them to someplace else or rename them so they start with * or something, then I'll know those were already processed :)

Looks like I already had results_20081111_1534_GB_nplb_4000.csv
so I'll do some snooping around to avoid duplicates.

[gd_barnes]

A couple of thoughts:

No, manual ranges should not be imported into the database. The stats are intended for LLRnet testing only.

Second, I noticed that I've been stuck on 98 primes found ever since Adam's database went away. Why is that?


Gary

[em99010pepe]

I disagree, manual ranges should also count for the stats, it's unfair. PSP and RS5 count their manual ranges. I am right in this but as usual you guys will give me reason a month or two later, I'll wait. It's a problem of vision, I understand..lol

I remember the discussion we had about the number of llrnet servers for the NPLB project....someone wanted a bunch of them (8)....oh dear....

Carlos

[IronBits]

I don't think Adam's database is getting updated, all attention to stats were shifted to Bok's system.
I've asked a couple of fellas if they would be interested in trying to do something with the old database, but that takes time, lots of it in some cases.
There is nothing I can do personally.

[gd_barnes]

Originally Posted by IronBits

I don't think Adam's database is getting updated, all attention to stats were shifted to Bok's system.
I've asked a couple of fellas if they would be interested in trying to do something with the old database, but that takes time, lots of it in some cases.
There is nothing I can do personally.

I'm very very confused. What is the link that I go to to see how many results and primes that I have processed and found that include all LLRnet servers? Isn't that what you guys have been working so hard towards?

I don't expect Adam's database to be updated if stats are being updated elsewhere but here is what is confusing me: When I go to your main page at nplb.ironbits.net, I see "Welcome to the No Prime Left Behind Project" and then right below that I see "No Prime Left Behind Database" and then I click on that and I see "Primes by User". I then click on that and I see that I've had 98 primes for over 2 months now.

If the above link is the incorrect place to go to get updated stats, do you suppose you could change the link to go to Bok's pages instead?

I need to be able to tell people: Here is the place to go to see your total LLRnet stats on the entire project because we have now imported all of the CSV results.

Sorry...I know there are some links in this thread but I just haven't been able to find the one that is supposed to have up-to-date stats for all imported CSV results from all servers for NPLB.


Thanks,
Gary

[gd_barnes]

Originally Posted by em99010pepe

I disagree, manual ranges should also count for the stats, it's unfair. PSP and RS5 count their manual ranges. I am right in this but as usual you guys will give me reason a month or two later, I'll wait. It's a problem of vision, I understand..lol

I remember the discussion we had about the number of llrnet servers for the NPLB project....someone wanted a bunch of them (8)....oh dear....

Carlos

The two are unrelated matters Carlos. Yes, on the server issue I/we were speaking from ignorance. Now, we're speaking from knowledge of the fact that the effort involved to do that with manual results is extreme.

If I sent you several hundred thousand manual results, do you suppose you can cross reference who did them with our lists, figure out about when they were done, and then write a program to convert them all to the correct CSV format? That's where we're coming from here.

I don't know how to reconcile the fact that some people do more manual work and some more LLRnet work. I suppose from this point forward, we can figure out some way to get future manual results converted to CSV format as we go, but still, some process needs to be set up for it.

If you can demonstrate some automated process for this conversion of manual results to CSV results on other projects, perhaps it can be considered.


Gary

[mdettweiler]

Originally Posted by gd_barnes

I'm very very confused. What is the link that I go to to see how many results and primes that I have processed and found that include all LLRnet servers? Isn't that what you guys have been working so hard towards?

I don't expect Adam's database to be updated if stats are being updated elsewhere but here is what is confusing me: When I go to your main page at nplb.ironbits.net, I see "Welcome to the No Prime Left Behind Project" and then right below that I see "No Prime Left Behind Database" and then I click on that and I see "Primes by User". I then click on that and I see that I've had 98 primes for over 2 months now.

If the above link is the incorrect place to go to get updated stats, do you suppose you could change the link to go to Bok's pages instead?

I need to be able to tell people: Here is the place to go to see your total LLRnet stats on the entire project because we have now imported all of the CSV results.

Sorry...I know there are some links in this thread but I just haven't been able to find the one that is supposed to have up-to-date stats for all imported CSV results from all servers for NPLB.


Thanks,
Gary

Hmm...I think I know why everybody is so confused about this. Adam's database, as far as I know, was used as the foundation for David's new database, and David's stuff was simply imported in on top of Adam's--hence why the "Stats by User" page continues to update whenever David imports new data. However, the reason why the "Primes by User" page doesn't update is probably due to differences between how primes were denoted in Adam's old CSV format, and in David's new CSV format. In Adam's format, the residual was marked as "prime", and there was no special field for denoting whether a number was prime. In David's format, the residual is left blank in case of a prime, and a field called "is_prime" is set to 1 (normally 0 for a non-prime).

David's mentioned a couple of different times to me via email that the stats pages that are being currently displaye are actually just Adam's old pages, and that no modifications have yet been made to how they retrieve and display the stats. Thus, even though new data is being added to the database, Adam's old PHP pages don't quite know how to display all that data. They know how to display the Stats by User page, since that only uses info that was consistent for both Adam's and David's CSV formats, but the Primes by User page looks for info in the format that Adam's old system used--hence why it doesn't "notice" the new primes that are being imported.

David told me in his emails that he's first trying to get a reliable and working solution for getting the data imported, and then after that he'll begin to tackle the front-end stuff. The front-end stuff, at this time, is still essentially unmodified from Adam's setup, hence the confusion.

I hope this explains things a bit--I was confused at first, too, when I read all this, but then I remembered the differences in the CSV formats and it all made sense. I think that's sort of what David meant when he mentioned about getting all of Adam's old database entries converted to a new format consistent with the new entries.

Max

[mdettweiler]

Originally Posted by gd_barnes

The two are unrelated matters Carlos. Yes, on the server issue I/we were speaking from ignorance. Now, we're speaking from knowledge of the fact that the effort involved to do that with manual results is extreme.

If I sent you several hundred thousand manual results, do you suppose you can cross reference who did them with our lists, figure out about when they were done, and then write a program to convert them all to the correct CSV format? That's where we're coming from here.

I don't know how to reconcile the fact that some people do more manual work and some more LLRnet work. I suppose from this point forward, we can figure out some way to get future manual results converted to CSV format as we go, but still, some process needs to be set up for it.

If you can demonstrate some automated process for this conversion of manual results to CSV results on other projects, perhaps it can be considered.


Gary

I believe I mentioned this in my above message, though it may have went somewhat unnoticed due to my having included it in a "P.S." section. So, I'll restate it here. What I was thinking is that a potentially workable way to keep stats for the manual work equally with the LLRnet work, yet while not having to worry about the discrepancies between the data available for each type of work, would be to have two separate databases, one for manual work and one for LLRnet.

We could, then, have two sets of stats pages: one displaying all the various info for LLRnet, and one for manual work. The LLRnet pages would, of course, be somewhat similar to what we've already got in mind. The manual pages, though, would be more tailored to the type of data that we have available for manual ranges. For example, any time/date displays would be clearly shown to be, say, the date when the range was returned--rather than the exact time/date that the individual k/n pairs were returned as with LLRnet, since that information is not specifically available for manual LLR.

Then, for the "common" information, i.e. the data that is available both for manual and LLRnet results, we could have combined stats pages--say, a combined "Stats by User" page, listing the total # of k/n pairs processed by each user, whether manual or LLRnet. This could possibly cull its data from the respective manual and LLRnet "Stats by User" pages, if that would be the easiest way to implement it.

By having separate databases, then we would also avoid another major problem with loading manual results into the LLRnet database: the sheer volume of data. The LLRnet database is already huge, and the pages each take quite a long time to load (at least in their current, highly incomplete incarnations). Adding all the manual data to that would probably double/triple/quadruple the size of the database and thus also the time it takes to generate the various pages. However, putting manual work in a separate database would ensure that neither of the individual databases gets so big that it's unworkable.

Anyway, just my $0.02.

Max

[IronBits]

Well, if I ever get a handle on this database and php stuff, we'll be good to go.
I can help you setup a db and php pages to put on Gary's server when I get it all figured out.
Max is correct, the old .php code that drives the front end has not been touched, and the data in the database fields needs to be modified so it's all the same, then the php files need to be edited to reflect those changes.
Bok is currently super swamped and can't help out for awhile...
Hang in there.

Maybe giving you more stats here: http://nplb.ironbits.net/progress_400.html
will help ease the burden and stats withdrawls.

[gd_barnes]

Originally Posted by IronBits

Well, if I ever get a handle on this database and php stuff, we'll be good to go.
I can help you setup a db and php pages to put on Gary's server when I get it all figured out.
Max is correct, the old .php code that drives the front end has not been touched, and the data in the database fields needs to be modified so it's all the same, then the php files need to be edited to reflect those changes.
Bok is currently super swamped and can't help out for awhile...
Hang in there.

Maybe giving you more stats here: http://nplb.ironbits.net/progress_400.html
will help ease the burden and stats withdrawls.

It's all good. To be honest, I don't care about my own stats at all. But I use them as a barometer to tell whether all of the stats are updating properly.

I know to actract heavy-hitters for extended periods, we'll have to have everything updating accurately so I/we will need to be able to tell them where to go to see their updated and accurate stats.

No problem and we're not in any rush. I just thought all results had been imported and was wondering why there was no increase in primes found by me, which told me that other's primes likely weren't updating either.

Thanks for the explanations David and Max.


Gary

[IronBits]

I'm almost there!
Finally got the bloody thing to go sideways

http://nplb.ironbits.net/progress_400.html