mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   mersenne.ca (https://www.mersenneforum.org/forumdisplay.php?f=173)
-   -   Mersenne.ca Status Report (https://www.mersenneforum.org/showthread.php?t=20678)

Gordon 2015-11-20 09:32

Mersenne.ca Status Report
 
It seems that the [URL="http://www.mersenne.ca/status/tf/0/0/3/0"]status report[/URL] han't been updated since 16th, can someone restart the script process...

kladner 2015-11-20 10:11

[QUOTE=Gordon;416688]It seems that the [URL="http://www.mersenne.ca/status/tf/0/0/3/0"]status report[/URL] han't been updated since 16th, can someone restart the script process...[/QUOTE]

That is James' bailiwick. Perhaps he is busy, traveling. or otherwise out-of-pocket.

I am eternally grateful to all of the volunteer maintainers of all this marvelous machinery. :tu: (Leaving out the bowing smiley in deference to Chris.) :smile:

lycorn 2015-11-23 08:34

:hello:
Any chance of getting it back to life?
Thx.

Syntony 2015-11-23 15:15

[QUOTE=lycorn;416945]:hello:
Any chance of getting it back to life?
Thx.[/QUOTE]

It seems that the daily import process must have stalled, as I haven't found any Primenet results reported since 16th November that have been imported into the mersenne.ca data-base...

Madpoo 2015-11-23 16:45

[QUOTE=Syntony;416974]It seems that the daily import process must have stalled, as I haven't found any Primenet results reported since 16th November that have been imported into the mersenne.ca data-base...[/QUOTE]

Now that I'm back home I can look at the Primenet side of things. I was out of town and had limited access so I couldn't address it sooner.

I'm noticing right away that the daily XML files that James grabs had problems on the 17th and 18th, probably due to the ridiculous TF results that were reported.

Those bogus results were scrubbed, so I just re-created those two XML dumps.

If James is paying attention to this thread, this should be his "all clear" to grab the updated files from 2015-11-17 onward.

For what it's worth, I think we had this problem before when a bogus TF came in... it overflowed something in the code to generate the XML that wasn't expecting such a large Ghz_hours value or something like that. Something in the SQL "CAST" or another.

lycorn 2015-11-23 19:54

Yep, that makes sense. Thanks for your effort.

Syntony 2015-11-24 05:47

[QUOTE=Madpoo;416982]
...
If James is paying attention to this thread, this should be his "all clear" to grab the updated files from 2015-11-17 onward.
...
[/QUOTE]

Seems that it's all rolling again - just checked some results from 17th and they've already been processed into the mersenne.ca data-base...

Good work guys! :bow:

lycorn 2015-11-24 07:49

Hmmm... not yet quite right.
Just check [URL="http://www.mersenne.ca/status/tf/0/0/1/0"]http://www.mersenne.ca/status/tf/0/0/1/0[/URL]
Also, If you try to get data from previous dates, you´ll get the message that no data was found for such dates.
But yes, something has started to move again. Good news...
Thanks a lot to those at the wheel.

Syntony 2015-11-27 05:03

[QUOTE=lycorn;417097]Hmmm... not yet quite right.
...
[/QUOTE]

Indeed, and that's still the case. Today (27th), the latest results that have been imported are for 23rd, so the import process is currently still running three days behind the normal schedule. Looks like some additional 'catch-up' import runs are needed...

blip 2015-11-27 07:27

seems to be working again, even if some days delayed

snme2pm1 2015-11-27 08:21

[QUOTE=blip;417366]seems to be working again, even if some days delayed[/QUOTE]

Till now, I have refrained from comment during this long outage.
It is sad that a rogue one or several account holders have again issued a torpedo so as to seriously disrupt proceedings at PrimeNet and consequentially mersenne.ca.
A greater level of resilience would be desirable.
It seems despite some slight movement, mostly the statistics are quite stale and unhelpful.
Good if James is enjoying a lovely holiday, which seems likely since he hasn't replied here.

blip 2015-11-27 08:43

[QUOTE=snme2pm1;417369]Till now, I have refrained from comment during this long outage.
It is sad that a rogue one or several account holders have again issued a torpedo so as to seriously disrupt proceedings at PrimeNet and consequentially mersenne.ca.
A greater level of resilience would be desirable.
It seems despite some slight movement, mostly the statistics are quite stale and unhelpful.
Good if James is enjoying a lovely holiday, which seems likely since he hasn't replied here.[/QUOTE]
Well, I don't see any serious disruption or anything even remotely close to that, and never saw. I just made the observation, that page again delivers statistics, which is nice and lets me hope, we will have current data again in the near future. As far as I understood, mersenne.ca is an auxiliary server not directly part of primenet, but a close friend of it. So, what risk do you see here for primenet core operations?

snme2pm1 2015-11-27 10:07

[QUOTE=blip;417370]Well, I don't see any serious disruption or anything even remotely close to that, and never saw. I just made the observation, that page again delivers statistics, which is nice and lets me hope, we will have current data again in the near future. As far as I understood, mersenne.ca is an auxiliary server not directly part of primenet, but a close friend of it. So, what risk do you see here for primenet core operations?[/QUOTE]

So far as I am aware, the mersenne.ca statistics have been largely dysfunction since 2015-11-16 when bogus assignment results were submitted to PrimeNet.
I am aware that mostly, the presentation of statistics faltered with an error message in recent days.
I am also aware that results presented by mersenne.ca. for example, earlier today claiming to be most recent, were instead several days old.

ramgeis 2015-11-27 12:26

Actually it says that the most recent data is from 24 Nov 2015 but does not claim the data to be most recent.
Unfortunately the data for that date is not in sync either. I assume it is still missing the updates from the days with the bogus results.
Nevertheless it is already helpful that the status page is up again as a rough overview is better than no overview at all.

lycorn 2015-11-27 12:57

Absolutely. That means things are "moving again".
But in fact there are still malfunctions to correct: e.g. try to get comparison of dates. You´ll find funny things like the Unfactored exponents largely outnumbering those Factored.

Madpoo 2015-11-27 17:30

[QUOTE=lycorn;417391]Absolutely. That means things are "moving again".
But in fact there are still malfunctions to correct: e.g. try to get comparison of dates. You´ll find funny things like the Unfactored exponents largely outnumbering those Factored.[/QUOTE]

I'm sure once James is back in the saddle (I presume he's vacationing, but that's a guess), he'll get it straightened out on his side.

The bogus results that mucked it up: since that was the second time it messed up the daily XML dump of logs that he uses, I should probably do something about that. At the very least I could put something in there so it just ignores any result from that day with a GHz_Days value that is obviously wrong (and overflows the XML generation script... can't remember why but I think I'm casting the "float" value to something else, or rounding it in some way maybe, I'm not sure why it choked on those crazy values).

The one thing I won't bother fixing is displaying "factored to bits" higher than 99 on the web "exponent report" page. It assumes it'll be 2 digits so 2^777 showed up as 2^77. It's a display only issue and it's because I used some SQL shortcuts to parse that part out of the result text, and fixing it seemed like work. :smile: I can't imagine we'll *actually* trial factor anything beyond 99 bits anytime soon. If we get to that point, I'd be happy to correct it then. LOL

EDIT: Looking at my daily XML generation script... I'm casting the float value for GHz_Days to NUMERIC(10,4) to round it to 4 sig digits. Which of course barfs when the FLOAT value itself is super crazy big. I'll just exclude any results with impossibly large numbers which will "solve" this and also prevent feeding funky data to James' system in the first place.

If anyone is curious, the record for the most GHz_Days given for a single result is:
M1217 completed 250000 ECM curves, B1=10000000, B2=1000000000000

That earned 277238.39321597 GHz Days.

In fact, the user that submitted that (Never Odd Or Even) has the top 19 largest GHz-Days results. Also, of the top 20, 19 are ECM efforts and #20 is a TF to 82 bits for M9007753 (by dbaugh, who has declared that exponent his nemesis for defying his early LL test effort... topic was covered in another thread recently). :smile:

It may be odd that he's spent so much time and resources to find *additional* factors for M1217 considering 3 factors are known, but hey, do what you enjoy.

Anyway, given these stats, I think it would be safe to say we wouldn't expect to see a single result come in that exceeded 300,000 GHz Days? Someone obviously could decide to run hundreds of thousands of ECM curves on something in one big chunk, but I think it's more typical to do smaller # of curves in an assignment, not 250,000 at once like M1217 got.

I'll pick a "safe" # to exclude results higher than. And this would only be for the daily XML report generation... the server itself would still potentially accept a funky result which is fine since we can weed out the goofballs after the fact.

chalsall 2015-11-27 18:47

[QUOTE=Madpoo;417417]If anyone is curious, the record for the most GHz_Days given for a single result is:
M1217 completed 250000 ECM curves, B1=10000000, B2=1000000000000[/QUOTE]

Just to put on the table as a question, is ECM normalized to TF, P-1 or LL (or DC)?

I personally don't care, except when it breaks the statistics.

LaurV 2015-11-28 02:30

[QUOTE=Madpoo;417417]
If anyone is curious, the record for the most GHz_Days given for a single result is:
M1217 completed 250000 ECM curves, B1=10000000, B2=1000000000000
That earned 277238.39321597 GHz Days.

In fact, the user that submitted that (Never Odd Or Even) has the top 19 largest GHz-Days results. Also, of the top 20, 19 are ECM efforts and #20 is a TF to 82 bits for M9007753 (by dbaugh, who has declared that exponent his nemesis for defying his early LL test effort... topic was covered in another thread recently). :smile:

It may be odd that he's spent so much time and resources to find *additional* factors for M1217 considering 3 factors are known, but hey, do what you enjoy.
[/QUOTE]
I repeatedly "complained" about NOOE, but George said in the past that he is trusting that user. He might have a reason, like knowing him personally or so (I assume). He used to communicate directly with George, who backed him. I "was convinced" that his "highest ever" LL was bogus (due to form of exponent and reporting times, with 38 minutes and 83 seconds or so), but your own TC later proved the result legit :blush:.

I still believe all (or part of) his "big results" for ECM are bogus, he jumped in a very short time to the top of the top in ECM and stayed there, over people who "dedicated their life" to ECM factoring. OTOH, these exponents can be done on the GPU, so it may be that he really invested big computing resources and some time resources, into it. You may check with George if these big results are not manually added by George himself.

Till proved wrong, I still believe he didn't do any ECM for this exponent, possible that he only "took the cream" and reported the [B][U]huge[/U][/B] qty of ECM done by the Cunningham project. Or he just considered that information, and decided to get some GHzDays of credit, based on the fact that 'the task was done, anyhow', so it is not a "false information" added to the data base. I mean, I don't contest the number of curves, this exponent had even more ECM done, inside of the Cunningham project. I only contest the fact that NOOE did it by himself.

snme2pm1 2015-11-28 02:58

[QUOTE=Madpoo;417417]The bogus results that mucked it up: since that was the second time it messed up the daily XML dump of logs that he uses, I should probably do something about that. At the very least I could put something in there so it just ignores any result from that day with a GHz_Days value that is obviously wrong ...
I'll just exclude any results with impossibly large numbers which will "solve" this and also prevent feeding funky data to James' system in the first place ....
I'll pick a "safe" # to exclude results higher than. And this would only be for the daily XML report generation... the server itself would still potentially accept a funky result which is fine since we can weed out the goofballs after the fact.[/QUOTE]

Would it be feasible to introduce (or hopefully strengthen) result income side vetting so as to raise an exception alert when various criteria are met.
Various parameters might be in play apart from merely the credit.
That alert could trigger a message to interested parties indicating that a submitted result deserves scrutiny.
I have an awkward feeling about trying to pick an arbitrary number for credit alone so as to cause deletion of data at a later migration stage.

Prime95 2015-11-28 03:12

[QUOTE=LaurV;417439]I still believe all (or part of) his "big results" for ECM are bogus, he jumped in a very short time to the top of the top in ECM and stayed there, over people who "dedicated their life" to ECM factoring. OTOH, these exponents can be done on the GPU, so it may be that he really invested big computing resources and some time resources, into it. You may check with George if these big results are not manually added by George himself.[/QUOTE]

It is easy to get "inflated" ECM credit by using GMP-ECM -- especially for small exponents. If you look at the example madpoo quoted, B2 = 10000 * B1 instead of the more normal 100 * B1. The server calculates CPU credit assuming prime95's poor stage 2 algorithm instead of GMP-ECM's superior stage 2 algorithm.

In a way this is similar to people using mfaktc climb the charts with tons of TF credit. The overall top producers page is inaccurately comparing apples (prime95 users) to oranges (GPU TFers) to lemons (GMP-ECM users).

kladner 2015-11-28 04:23

[QUOTE=Prime95;417446]It is easy to get "inflated" ECM credit by using GMP-ECM -- especially for small exponents. If you look at the example madpoo quoted, B2 = 10000 * B1 instead of the more normal 100 * B1. The server calculates CPU credit assuming prime95's poor stage 2 algorithm instead of GMP-ECM's superior stage 2 algorithm.

In a way this is similar to people using mfaktc climb the charts with tons of TF credit. The overall top producers page is inaccurately comparing apples (prime95 users) to oranges (GPU TFers) to lemons (GMP-ECM users).[/QUOTE]

I can't address the comparisons between types of work directly, but I would offer that almost any procedure can be misused to game the system. I don't know if I 'deserve' the credits I get for LLTF, but I am helping the effort to keep TF out in front of LL needs, and those GPUs do drink the 'electric fluid' freely, [I]and[/I] turn out lots of cycles. I guess that could be taken as an argument that credit is an incentive to invest the hardware and the power to turn out those cycles.

I only took up factoring when I got a GTX GPU, so I can't relate to doing it on a CPU. I wonder how the cycles-per-watt would work out doing a similar amount of TF on a recent Xeon system, as well as the time required. Still, I pay my CPU dues with 75% DC, 25% LL, on a not-very-efficient-or-terribly-productive CPU.

I don't care that much if someone runs up large, possibly bogus numbers. I run a single, stock (AMD FX) system, with two (overclocked) 580 GPUs. I mostly look at GPU72 graphs for standing, but I am not disappointed when I look at PrimeNet statistics, either. Like LaurV, I do have to admit to being a credit whore, but I try to come by credit while contributing to the purpose of the project.

Sorry for what might seem like a rant. I'm probably just expelling guilt over my power usage. :smile:

VictordeHolland 2015-11-28 11:30

[QUOTE=Prime95;417446]The server calculates CPU credit assuming prime95's poor stage 2 algorithm instead of GMP-ECM's superior stage 2 algorithm.[/quote]
Maybe GMP-ECM credit could be given for CPUtime instead of the inflated results. But then all the previous results would have to be changed to be fair. On the other hand, a bit extra credit for the extra trouble might persuade people to use the more efficient GMP-ECM stage 2 on small exponents.
[quote]
In a way this is similar to people using mfaktc climb the charts with tons of TF credit. The overall top producers page is inaccurately comparing apples (prime95 users) to oranges (GPU TFers) to lemons (GMP-ECM users).[/QUOTE]
Yep, and that is inevitable with developments in software and hardware. Changing it now will only have an effect for a few years until the next super LL-card or AVX5.0 comes along.

Madpoo 2015-11-28 17:38

[QUOTE=Prime95;417446]It is easy to get "inflated" ECM credit by using GMP-ECM -- especially for small exponents.[/QUOTE]

In the case of M1217 and the ECM results from NOoE, the messages have what I think are the checksums (no assigmnent ID though, but that's not unusual).

The weird thing is, a lot of his results have the *same* 64-bit checksum value even though the message itself is different (diff # of curves or bounds). Some are the same message (same curves/bounds/checksum) which, again, isn't unusual for ECM since people can and do run the same # of curves and bounds on parallel threads.

But then, how do we know he really ran 17 x 91 curves, or just did that one time and submitted 17 times?

Also, the app ID indicated in the result says it came from: "Windows64 Prime95 v26"

250,000 curves with those bounds would have taken quite a bit of time, and for all I know he really did it: I have no opinion. :smile:

bloodIce 2015-11-28 20:28

[QUOTE]But then, how do we know he really ran 17 x 91 curves, or just did that one time and submitted 17 times?[/QUOTE]There are several ways of submitting reports, which will look legit, but are not (and you and George know them all + some extra). This particular user is also on my watch for several years and I will not be amazed if something is fishy. However, we cannot blame someone without evidence of wrongdoing, so by definition s(he) is innocent. There are two ways to contra-act the problem. Firstly factor completely 1217 (:grin:), but more seriously why not implement a ECM/Pm1/TF checksums dependent on the data/time of the [STRIKE]completion[/STRIKE] start of a task in any further versions of prime95.

Gordon 2015-11-28 22:33

[QUOTE=bloodIce;417544]There are several ways of submitting reports, which will look legit, but are not (and you and George know them all + some extra). This particular user is also on my watch for several years and I will not be amazed if something is fishy. However, we cannot blame someone without evidence of wrongdoing, so by definition s(he) is innocent. There are two ways to contra-act the problem. Firstly factor completely 1217 (:grin:), but more seriously why not implement a ECM/Pm1/TF checksums dependent on the data/time of the [STRIKE]completion[/STRIKE] start of a task in any further versions of prime95.[/QUOTE]

Because there are quite a few of us who do Stage 1 in P95 and stage 2 in gmp-ecm, no check-sums to check-in!

Gordon 2015-12-02 23:54

[QUOTE=Madpoo;417522]In the case of M1217 and the ECM results from NOoE, the messages have what I think are the checksums (no assigmnent ID though, but that's not unusual).

The weird thing is, a lot of his results have the *same* 64-bit checksum value even though the message itself is different (diff # of curves or bounds). Some are the same message (same curves/bounds/checksum) which, again, isn't unusual for ECM since people can and do run the same # of curves and bounds on parallel threads.

But then, how do we know he really ran 17 x 91 curves, or just did that one time and submitted 17 times?

Also, the app ID indicated in the result says it came from: "Windows64 Prime95 v26"

250,000 curves with those bounds would have taken quite a bit of time, and for all I know he really did it: I have no opinion. :smile:[/QUOTE]

Is perchance the checksum - Wd9: 00000000

Madpoo 2015-12-03 03:26

[QUOTE=Gordon;418073]Is perchance the checksum - Wd9: 00000000[/QUOTE]

LOL... nope. I would have noticed that. :smile:

Dubslow 2015-12-24 07:32

Does anyone know if and where I might find the mersenne.ca equivalent of [url]http://www.mersenne.info/exponent_status_line_graph/1/0/[/url] ?

lycorn 2016-01-26 08:11

Prime count not yet updated in the reports

[URL="http://www.mersenne.ca/status/tf/0/0/1/0"]http://www.mersenne.ca/status/tf/0/0/1/0[/URL]

snme2pm1 2016-02-10 05:41

[QUOTE=Gordon;416688]It seems that the [URL="http://www.mersenne.ca/status/tf/0/0/3/0"]status report[/URL] han't been updated since 16th, can someone restart the script process...[/QUOTE]

Seems to be another pause in progress, now we're at UTC 2016-02-10 yet response from such as above reflect 2016-02-07.
It seems odd that mersenne.ca contents list doesn't include such (not quite so) new facilities, such that a person can encounter them without learning a link from mersenneforum.org.

James Heinrich 2016-02-11 13:14

Hmm, seems this thread is about mersenne.ca but I wasn't subscribed to it so I didn't know there was stuff going on. Thanks [i]snme2pm1[/i] for bringing my attention to it.

[QUOTE=Dubslow;420056]Does anyone know if and where I might find the mersenne.ca equivalent of [url]http://www.mersenne.info/exponent_status_line_graph/1/0/[/url] ?[/QUOTE]That would be here: [url]http://www.mersenne.ca/status/ll/0/0/2/0[/url]

[QUOTE=lycorn;424097]Prime count not yet updated in the reports[/QUOTE]Thanks, fixed.

[QUOTE=snme2pm1;425807]Seems to be another pause in progress, now we're at UTC 2016-02-10 yet response from such as above reflect 2016-02-07.
It seems odd that mersenne.ca contents list doesn't include such (not quite so) new facilities, such that a person can encounter them without learning a link from mersenneforum.org.[/QUOTE]I changed the database structure on 2016-02-08 and that actually went fine, but one small unrelated typo at the same time broke the daily jobs, so they didn't run for the last 3 days. I have spent the last hour or so rebuilding that data so it should be working as expected now. I also added a link to visualization from the menu (and changed the big graph link on the home page to point to visualization rather than graphs).

LaurV 2016-02-12 02:26

Nice! Thanks!
BTW I think you should start to collect clLucas results in that table of cudaLucas[URL="http://www.mersenne.ca/cudalucas.php?model=490"] fire power[/URL]. By the same logic as the [URL="http://www.mersenne.ca/mfaktc.php?sort=ghdpd"]sister-table[/URL] of mfaktX results contains both AMD and Nvidia boulders, now after msft's last additions, clLucas became quite competitive, at least (or "not only"?) for some ranges where power of 2 fft's are adequate (I would argue about "not only", thinking about the prices for new amd cards, but that is a different discussion). The idea is that the table of xxLucas could contain AMD results too. What do you think?

LaurV 2016-02-12 05:39

[time limit]
It occurred to me that the prices in the tables have to be updated too. Especially for a K80, you could buy it now for $3800 (one used at $3900 and 13 new at $4100 on Amazon, just for example, the first popup when search by google). Therefore lowering the price from $5k on the table will [B][U]really[/U][/B] improve their "james' value ratio" (don't laugh, that is a very good indicator! :razz:), making it highly comparable with the Titans, especially as we switch to higher and higher exponents, more memory (and especially ECC memory!) for larger FFTs becomes [U]a must[/U] (and it should also count toward the score, but that is a different story). One can not anymore use the cheaper version of the gtx580 (the one with 1.5GB memory only) to run the higher FFTs required for a LMH exponent, at least 2-3GB is a must, therefore that will soon be only the niche of Titans and Teslas (or the elite versions of gtx 5/6/780 or 90 with 3GB of RAM) [edit, or other newer thingies yet to come, not forgetting AMD cards as clLucas got faster now]

airsquirrels 2016-02-12 05:42

[QUOTE=LaurV;426040][time limit]
It occurred to me that the prices in the tables have to be updated too. Especially for a K80, you could buy it now for $3800 (one used at $3900 and 13 new at $4100 on Amazon, just for example, the first popup when search by google). Therefore lowering the price from $5k on the table will [B][U]really[/U][/B] improve their "james' value ratio" [don't laugh, that is a very good indicator!], making it highly comparable with the Titans, especially as we switch to higher and higher exponents, more memory (and especially ECC memory!) for larger FFTs becomes [U]a must[/U] (and it should also count toward the score, but that is a different story).[/QUOTE]

I was just speaking with chasall about this. I have a decent selection of AMD cards I can provide at least an initial seed of benchmarks to make this happen if someone tells me what we need.

LaurV 2016-02-12 05:49

James is the guy you have to talk to.
With chalsall you talk about Barbadian Rum and swimming pools :razz:

James Heinrich 2016-02-12 09:33

[QUOTE=LaurV;426020]BTW I think you should start to collect clLucas results in that table[/QUOTE]The NVIDIA side is already messy enough, but I have no objections to including AMD data as well. I didn't even know clLucas existed. If people want to send me benchmark data I can start including it in the chart, but nobody has ever done so therefore I have no data to present.

[QUOTE=LaurV;426040]It occurred to me that the prices in the tables have to be updated too. Especially for a K80, you could buy it now for $3800[/QUOTE]The pricing data is pulled from a local computer retail store so only is updated regularly for items currently being sold. Also please note that all prices are in Canadian dollars which can vary from +0% to +50% on the US dollar. My price source doesn't happen to carry any K80 right now, but a quick search for other Canadian prices for K80 gave me $5999, $5999, $6295, $6400 as the best 4 prices I could find, so listing $5000 on the table is actually a bit lower than it should be.

James Heinrich 2016-02-12 09:39

[QUOTE=airsquirrels;426042]I have a decent selection of AMD cards I can provide at least an initial seed of benchmarks to make this happen if someone tells me what we need.[/QUOTE]For now the same benchmark used for CudaLucas would be great:[quote=http://www.mersenne.ca/cllucas.php]Please send me CUDALucas benchmarks to improve the accuracy of this chart:
Please send me the first 30000 iterations of [code]CUDALucas 57885161[/code] in your results. Please email results to [email]james@mersenne.ca[/email][/quote]

ramgeis 2016-02-26 15:30

The database seems to not be in sync anymore as the date for the most recent data is "22 Feb 2016"

James Heinrich 2016-02-26 16:49

Thanks. The data feed from PrimeNet had an unexpectedly-large dump of data on 2016-Feb-23, more than triple the typical daily amount of data. The XML file itself was 254MB, so PHP ran out of its allocated 256MB trying to process it.

I'm working to manually import the data from the 2016-Feb-23, and once that's done the subsequent days shouldn't be a problem. I'm currently running into some timeout issues where it's taking longer than PHP will let a single script run for, but I'll get through it eventually. I'll update when it's fixed.

Mark Rose 2016-02-26 17:12

That would be the skipped-TF results of mine that Madpoo imported :)

chalsall 2016-02-26 17:31

[QUOTE=Mark Rose;427491]That would be the skipped-TF results of mine that Madpoo imported :)[/QUOTE]

Yaw see! This is why we can't have nice things.... :wink:

Mark Rose 2016-02-26 17:56

[QUOTE=chalsall;427493]Yaw see! This is why we can't have nice things.... :wink:[/QUOTE]

How was I to know 394,767 result lines would cause such trouble! Doesn't AirSquirrels do that like every day? ;)

James Heinrich 2016-02-26 18:09

23-Feb finished, finally[code]Inserted (511,782 results after 1,953 seconds)
Done in 2,206 seconds[/code]

James Heinrich 2016-02-26 18:41

Data sync is now up-to-date.

For comparison:[code]
23-Feb = Inserted (511,782 results after 1,953 seconds); Done in 2,206 seconds
24-Feb = Inserted ( 83,524 results after 644 seconds); Done in 885 seconds
25-Feb = Inserted ( 90,642 results after 510 seconds); Done in 751 seconds
[/code]

Madpoo 2016-02-26 21:01

[QUOTE=James Heinrich;427486]Thanks. The data feed from PrimeNet had an unexpectedly-large dump of data on 2016-Feb-23, more than triple the typical daily amount of data. The XML file itself was 254MB, so PHP ran out of its allocated 256MB trying to process it.[/QUOTE]

Umm... yeah... about that... :smile:

When I saw the post about mersenne.ca not updating since the 22nd, my first thought was "uh oh, I just dumped nearly 400K new results into the database on the 23rd... coincidence?"

Sorry about that. If I had it to do over I may have chosen to spread the data dump over several days to avoid any ripple effects.

James Heinrich 2016-02-26 21:16

[QUOTE=Madpoo;427516]Umm... yeah... about that...[/QUOTE]:max: :tantrum: :edit: :rant:

No worries, it's all good now :cool:

Dubslow 2016-02-26 23:29

[QUOTE=Mark Rose;427491]That would be the skipped-TF results of mine that Madpoo imported :)[/QUOTE]

[QUOTE=chalsall;427493]Yaw see! This is why we can't have nice things.... :wink:[/QUOTE]

I would rather think that this...

[QUOTE=James Heinrich;427486]so PHP ran out of its allocated 256MB trying to process it.[/QUOTE]

...is why we can't have nice things. :razz:

snme2pm1 2016-03-11 07:13

ISO 8601
 
James, I appreciated that you rapidly responded to a suggestion to make dates more universal some months ago, by using more ISO 8601 rather than Canadian format for the currency of TF statistics.
It was of interest to further discover that Canada has adopted ISO 8601 as a non-mandatory standard for many purposes.
Further it would seem that Canada is one of the worst places in the world for ambiguity of multiple date formats.
I observe an anomaly at mersenne.ca, whereby for TF statistics we see radio button "the most recent data (dd mmm yyyy)" then in the next column another radio button "a specific date:" yyyy-mm-dd.
I didn't venture to look for further non ISO dates at your site.
There seems to be compelling reasons to adopt ISO dates, everywhere.

James Heinrich 2016-03-11 13:34

[QUOTE=snme2pm1;428731]It was of interest to further discover that Canada has adopted ISO 8601 as a non-mandatory standard for many purposes.
Further it would seem that Canada is one of the worst places in the world for ambiguity of multiple date formats.[/QUOTE]Canada is one of the worst places for mixed measurement system standards. Temperature is Celsius, except for cooking. Produce is priced per pound and sold per kg. Size and weight are metric, except for things people don't know in metric like their own height and weight, the size of their TV, or the size of a 2x4 lumber or 4x8 plywood at the hardware store. Everything has a normal unit of measure, you just have to know what it is. If you talk weather in Fahrenheit, or speed in mph, or ask someone how many kg they weigh and you'll get confused looks.

:ick:

I didn't know Canada had any standards, formal or otherwise, for dates. Anything and everything goes, so you really have no idea what 06/09/12 means. It could be the same as 12/09/06, but who knows.

I found a few outlier date formats on mersenne.ca that I have now brought more inline with ISO 8601 format.

science_man_88 2016-03-11 13:41

[QUOTE=James Heinrich;428758]Canada is one of the worst places for mixed measurement system standards. Temperature is Celsius, except for cooking. Produce is priced per pound and sold per kg. Size and weight are metric, except for things people don't know in metric like their own height and weight, the size of their TV, or the size of a 2x4 lumber or 4x8 plywood at the hardware store. Everything has a normal unit of measure, you just have to know what it is. If you talk weather in Fahrenheit, or speed in mph, or ask someone how many kg they weigh and you'll get confused looks.

:ick:

I didn't know Canada had any standards, formal or otherwise, for dates. Anything and everything goes, so you really have no idea what 06/09/12 means. It could be the same as 12/09/06, but who knows.

I found a few outlier date formats on mersenne.ca that I have now brought more inline with ISO 8601 format.[/QUOTE]

I like conversions so I'm the outlier. I hated conversions in pharmacy though because the US ones ( some still used) are inconsistent with each other and it's done mostly for speed I think. I have a body fat scale that can weigh in kg so I know roughly ( plus I use the wii fit and it's set to kg, I'm still not happy about my weight at times)

petrw1 2016-03-11 15:44

[QUOTE=James Heinrich;428758]Canada is one of the worst places for mixed measurement system standards. [/QUOTE]

Pretty bad considering we converted to metric 49 year ago.

James Heinrich 2016-03-11 16:03

[QUOTE=petrw1;428765]Pretty bad considering we converted to metric 49 year ago.[/QUOTE]Is that a Canadian Year, British Year, or US Year measurement? :unsure:

By my calculation it's only 45 (perhaps 46) years since [url=https://en.wikipedia.org/wiki/Metrication_in_Canada]conversion[/url] started (and 31 years since the following goverment messed it up :furious:).

Mark Rose 2016-03-11 16:49

[QUOTE=James Heinrich;428766]Is that a Canadian Year, British Year, or US Year measurement? :unsure:

By my calculation it's only 45 (perhaps 46) years since [url=https://en.wikipedia.org/wiki/Metrication_in_Canada]conversion[/url] started (and 31 years since the following goverment messed it up :furious:).[/QUOTE]

[YOUTUBE]ghFntxvlqvM[/YOUTUBE]

Skip to 10:00. Explains everything.

petrw1 2016-03-11 17:12

[QUOTE=James Heinrich;428766]Is that a Canadian Year, British Year, or US Year measurement? :unsure:

By my calculation it's only 45 (perhaps 46) years since [url=https://en.wikipedia.org/wiki/Metrication_in_Canada]conversion[/url] started (and 31 years since the following goverment messed it up :furious:).[/QUOTE]

My failing memory told me it was 1967....I was close-ish.

Recall however that the U-S-of-A attempted to go metric a few years later (1975) and .... well they still do Miles and Fahrenheit.....

I think (cue whining) Canada would have been more successful if our neighbor to the South played along.

James Heinrich 2016-03-11 17:51

[QUOTE=petrw1;428775]Canada would have been more successful if our neighbor to the South played along.[/QUOTE]Oh, most certainly. Things like railroads continuing to operate in miles is directly attributable to the USA not converting. We'd buy metric lumber too if not for the neighborly influence.

Madpoo 2016-03-11 17:52

[QUOTE=James Heinrich;428758]I didn't know Canada had any standards, formal or otherwise, for dates. Anything and everything goes, so you really have no idea what 06/09/12 means. It could be the same as 12/09/06, but who knows.

I found a few outlier date formats on mersenne.ca that I have now brought more inline with ISO 8601 format.[/QUOTE]

FYI, I've tried to update the mersenne.org website (and any new code I've added) to use yyyy-mm-dd wherever possible. It's an international audience and the ambiguity of other regional date formats would be too horrible to work out.

My day job involves working with a variety of int'l websites, each with their own metric or standard oddities... kind of like what you said, where it's all metric *except* XYZ which is standard for some reason.

At least we're not England where they weigh people in "stones" :smile:

Canada can use the excuse of their curious neighbor to the south and our linked economies for why certain things there are still standard measure. Personally I still get confused whenever I drive up to Vancouver and have to look at the "other dial" on my speedometer... I won't deny that I've sometimes gone 60 MPH when the speed limit was 60 KPH. Whoops.

I even once convinced a friend of mine who had never been to Canada before that once we went through the border, we would switch sides of the road and also have to do conversions to metric time. She bought it, until we actually got there...

science_man_88 2016-03-11 18:24

[QUOTE=James Heinrich;428778]Oh, most certainly. Things like railroads continuing to operate in miles is directly attributable to the USA not converting. We'd buy metric lumber too if not for the neighborly influence.[/QUOTE]

rough cut 2 by 4 is 5.08 by 10.16 cm I think but after smoothing it's closer to 3.81 by 8.89 cm

Mark Rose 2016-03-11 18:35

[QUOTE=Madpoo;428779]all metric *except* XYZ which is standard for some reason.[/quote]

Metric IS standard. The system used in the US can basically be called American at this point. ;)

snme2pm1 2016-03-12 01:42

[QUOTE=Mark Rose;428783]Metric IS standard.[/QUOTE]

The wonderful thing about standards is that you can ascribe to so many.
Even "Metric" has evolved to be in many places replaced by SI, (International System of Units) or Système international d'unités as it commenced.
For example, I abhor seeing any measurement quoted as cm, which are not a preferred unit of measurement by SI, though certain places in the world still cling to it.
SI tends to favour 10^3 changes in the unit multiplier symbols.
My caliper (prior to a decimal point) has units of mm, not cm.

Xyzzy 2016-03-12 01:52

[url]https://xkcd.com/927/[/url]

Madpoo 2016-03-13 05:21

[QUOTE=Xyzzy;428815][url]https://xkcd.com/927/[/url][/QUOTE]

That's pretty much the way it works. I'm reminded of "Esperanto", because, you know, we needed another language to fix everything that was wrong with every other language.

Syntony 2016-04-12 12:49

Another results import stall?
 
Seems that the mersenne.ca data import process might have stalled again - results from yesterday (11th April) have yet to appear...

James Heinrich 2016-04-12 14:38

I saw that last night but wanted to give it an extra little time to get working again by itself, but it didn't.
The problem is actually upstream on mersenne.org, yesterday's data file is mysteriously 404. I'll fire off an email to Aaron to see what's up.

Syntony 2016-04-12 16:37

OK, looks like it's up and running again... :bow::bow::bow:

James Heinrich 2016-04-12 16:41

Yeah it's running again, and it wasn't Aaron's fault. :redface:

It was basically an ugly interaction of timezones that caused some confusion.

Mark Rose 2016-04-12 17:17

I'm almost to the point of setting all my clocks at home to UTC.

lycorn 2016-04-12 21:40

[QUOTE=James Heinrich;431401]Yeah it's running again.[/QUOTE]

Not really:

[B]Data as of 2016-04-11T23:59:00+00:00
Error: no data (unexpected?)[/B]

James Heinrich 2016-04-13 04:41

[QUOTE=lycorn;431422][B]Data as of 2016-04-11T23:59:00+00:00
Error: no data (unexpected?)[/B][/QUOTE]Is it still doing that? If so, where, please?

lycorn 2016-04-13 12:44

Last night that message was displayed at [URL="http://www.mersenne.ca/status/tf/0/0/1/0"]http://www.mersenne.ca/status/tf/0/0/1/0[/URL] instead of the usual "Data as of 2016-xx-xxT23:59:00+00:00".
This morning at 08:00 local time (UTC + 1) it was OK.

petrw1 2016-04-13 16:21

No data found for 2016-04-11

James Heinrich 2016-04-13 18:04

[QUOTE=petrw1;431467]No data found for 2016-04-11[/QUOTE]Yeah, I couldn't find it either.
Failing any better data I have replicated 2016-04-10 into 2016-04-11.

snme2pm1 2016-04-17 02:10

[QUOTE=James Heinrich;428758]I found a few outlier date formats on mersenne.ca that I have now brought more inline with ISO 8601 format.[/QUOTE]

Spotted one that was apparently overlooked.
[url]http://www.mersenne.ca/index.php?submitresults=1[/url]
[CODE]Note: effective 14-Aug-2015 results in PrimeNet range (below 1000M)
are no longer accepted here, all relevant data is now pulled from
mersenne.org and all results should be submitted only there[/CODE]
Evidently some of your paragraphs are terminated by a period or other appropriate punctuation symbol, but not all.

James Heinrich 2016-04-17 02:27

Well that's nitpicky, but ok. Fixed.

snme2pm1 2016-04-17 03:32

[QUOTE=James Heinrich;431474]Yeah, I couldn't find it either.
Failing any better data I have replicated 2016-04-10 into 2016-04-11.[/QUOTE]

That sounded cavalier at the time, but I assumed you knew best.
But could that relate to apparent strange arithmetic exhibitions in tables lately, or is something else in play?
[CODE]Data as of 2016-04-15T23:59:00+00:00 (compared with 20 days earlier: 2016-03-26T23:59:00+00:00)
Range Exponents Prime Factored Unfactored <62 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 >85
9.90M 610
9.91M 614
9.92M 616 -1 +1
9.93M 631
9.94M 604
9.95M 618 -1 +1
9.96M 622
9.97M 608
9.98M 597 +1 -1 -1
9.99M 614
Total 0 +1 -1 -3
[/CODE]
I don't see a +2 total for column 70.
[CODE]Data as of 2016-04-15T23:59:00+00:00 (compared with 20 days earlier: 2016-03-26T23:59:00+00:00)
Range Exponents Prime Factored Unfactored <66 66 67 >67
9.0M 6,183 +3 -3 -227 +224
9.1M 6,245 +5 -5 -1,026 +1,021
9.2M 6,260
9.3M 6,223
9.4M 6,178 +1 -1 -2
9.5M 6,273 +1 -1 -3
9.6M 6,203
9.7M 6,211 -1
9.8M 6,180 -1
9.9M 6,134 +1 -1 -3
Total 0 +11 -11 -1,260 +221
[/CODE]
That looks odd, in many ways.
For example what was the disposition of the -3 from row 9.9M column 67. It says 1 was factored, but where did two others end up? Similarly other rows are funny.
How does the total +221 for column 67 arise?

James Heinrich 2016-04-17 15:28

[QUOTE=snme2pm1;431774]But could that relate to apparent strange arithmetic exhibitions in tables lately, or is something else in play?
It says 1 was factored, but where did two others end up?[/QUOTE]It was just a display issue, nothing wrong with the underlying data. In this case mostly related to the dynamic range of the table (only shows relevant bit levels depending on the range you're looking at). The table at the time you saw it went up to 2[sup]67[/sup] but the "missing" exponents had been factored to 2[sup]70[/sup] and didn't show on the table.

It should be fixed now:
[url]http://www.mersenne.ca/status/tf/20160415/20/4/900[/url]

James Heinrich 2016-04-21 06:36

mersenne.ca will be going offline for a few hours for some database fiddling.

James Heinrich 2016-04-21 16:04

8 hours later it's still chugging along. Not surprising since it involves importing a 10GB [SIZE="1"](7.6GB data, 13.2GB index, packaged in a 10.5GB SQL file)[/SIZE], 116-million-record SQL file. Unfortunately there's no easy way (I know of ?) to check on its progress.

A little background: I'm replacing the main database table that holds the archive of all results submitted to PrimeNet. I discovered about a month ago that not all the data was getting in, in specific when similar results exist from the same user at the same timestamp then it would reject the "duplicate". In a few rare cases there are true duplicates, however most of these records that were being skipped were things like sequential TF, e.g. from 64-65,65-66 but submitted as two results that ended up in the database as the same timestamp. So I changed the parsing code to allow those kind of not-quite-duplicates by including the bounds in the unique index, which of course increased the database size very notably. The record count went up by about 10% (106M to 116M), data storage went up 40% from 5.5GB to 7.6GB, but the index jumped 100% from 6.6GB to 13.2GB. :cry:

I'll post back when the site's working again.

James Heinrich 2016-04-21 17:32

From what I can see the database is now 6.2 of an expected 7.6GB after about 9 hours.

Mark Rose 2016-04-21 18:36

[QUOTE=James Heinrich;432124]8 hours later it's still chugging along. Not surprising since it involves importing a 10GB [SIZE="1"](7.6GB data, 13.2GB index, packaged in a 10.5GB SQL file)[/SIZE], 116-million-record SQL file. Unfortunately there's no easy way (I know of ?) to check on its progress.
[/quote]

Use pt-online-schema-change (and run it in screen or tmux) for table alters. It's part of the Percona toolkit. You get the added bonus of no downtime. In fact, I'm using it to alter some indexes as I type this on a table that's updated several times a second. The table is about 90 GB with about 323M rows. It reports regular process and says I have about 3 hours to go. It works with all variants of MySQL, not just the Percona flavour.

For imports, you could always pipe it through pv:

pv import.sql | mysql

Not entirely accurate, but easy. If you have file-per-table turned on, you can also watch the size of the table grow on disk. Lastly, you can always watch `show table status like 'table_name';` and compare the row count to `wc -l import.sql`.

chalsall 2016-04-21 19:00

[QUOTE=Mark Rose;432146]Not entirely accurate, but easy. If you have file-per-table turned on, you can also watch the size of the table grow on disk. Lastly, you can always watch `show table status like 'table_name';` and compare the row count to `wc -l import.sql`.[/QUOTE]

I recently encountered a similar, but entirely different, issue.

I wanted to have 8.5" by 14" (AKA Legal paper) perforated in a particular way, and then offset printed with a particular process ink. Pantone 804C, to be specific.

Kind of strange how difficult it is to have atoms processed.

I did finally find a shop who could do the job, but it was going to cost me USD $1,000 just to produce the stamp. And they couldn't print in the ink I needed.

James Heinrich 2016-04-21 19:13

I'm just peeking at the filesize of the .MYD file in /var/lib/mysql
Currently 6,803,300,352 (of 8,163,918,264).

Unfortunately the process is slow because (as I understand it) DISABLE KEYS only works for non-unique indexes. And there's a big unique index getting thrashed around in there.

Perhaps pv would be useful next time, but I'm not likely to restart it now just for that :smile:

Mark Rose 2016-04-21 19:25

[QUOTE=James Heinrich;432151]I'm just peeking at the filesize of the .MYD file in /var/lib/mysql
Currently 6,803,300,352 (of 8,163,918,264).[/quote]

MyISAM? It's been almost a decade since I used that :)

InnoDB is better in nearly every way since MySQL 5.5, and is the default table engine. It does require a minute or two of server configuration before blindly using it though.

[quote]
Unfortunately the process is slow because (as I understand it) DISABLE KEYS only works for non-unique indexes. And there's a big unique index getting thrashed around in there.[/quote]

Yes. It has to throw an error before a duplicate row is added, and the fastest way to check that is having the unique index.

[quote]Perhaps pv would be useful next time, but I'm not likely to restart it now just for that :smile:[/QUOTE]

:)

James Heinrich 2016-04-21 19:33

I suppose next time I do something foolish like this I should import the datadump without indices, and add them once the data is imported, since I know there are no unique-index conflicts. Lessons learned :)

Mark Rose 2016-04-21 20:25

[QUOTE=James Heinrich;432155]I suppose next time I do something foolish like this I should import the datadump without indices, and add them once the data is imported, since I know there are no unique-index conflicts. Lessons learned :)[/QUOTE]

With MyISAM you shouldn't even need a primary key for the import, so include that in the unique-index list.

(InnoDB will create an internal one if you don't provide one.)

James Heinrich 2016-04-22 02:57

7,931,805,696 of 8,163,918,264 after ~19 hours

I'm almost tempted to re-try the import once this is done as just a data dump with no index at all, see how fast that goes by comparison. And then just add a single primary key that covers my unique criteria. I don't need a separate autoincrement primary key field, and I found I had a couple indexes on there that don't really need to be there, so if that works it would be considerably smaller: data about 5% smaller dropping the autoincrement field, but index only about a third the size. Testing locally it only took 53 minutes to add the primary key on my fields.

James Heinrich 2016-04-22 06:41

Well, lesson learned there. It took ~20 hours to import the database with indexes updating during import. To import the data (minus the autoincrement column) with [i]no[/i] indexes it took... 35 minutes. I'm adding the primary key now, based on local testing that should take about an hour. So about 10x faster.

James Heinrich 2016-04-22 16:27

So, finally, up and running with the second imported version of the database.
Indexing took a bunch longer on my server (~5 hours) than it did locally (50 mins) -- CPU difference (i7-3930K vs Atom C2750), I'm sure -- but still far far far quicker than importing while updating 2 unique indexes (~22 hours).

However, looking at how some of my other queries reference this table I'm also going to need to add a non-unique index on date_received. That'll take a couple hours I'm sure, but the site should be usable in the meantime.

Madpoo 2016-04-22 17:04

[QUOTE=James Heinrich;432206]Well, lesson learned there. It took ~20 hours to import the database with indexes updating during import. To import the data (minus the autoincrement column) with [i]no[/i] indexes it took... 35 minutes. I'm adding the primary key now, based on local testing that should take about an hour. So about 10x faster.[/QUOTE]

I feel bad now for not finding time to re-do the data dumps with (made up, if necessary) random timestamps or random milliseconds to make the datetime itself unique, always.

Gordon 2016-04-23 15:45

[QUOTE=James Heinrich;432244]So, finally, up and running with the second imported version of the database.
Indexing took a bunch longer on my server (~5 hours) than it did locally (50 mins) -- CPU difference (i7-3930K vs Atom C2750), I'm sure -- but still far far far quicker than importing while updating 2 unique indexes (~22 hours).

However, looking at how some of my other queries reference this table I'm also going to need to add a non-unique index on date_received. That'll take a couple hours I'm sure, but the site should be usable in the meantime.[/QUOTE]

Unfortunately the report is wrong. If you look at the 6.6m range it tells you there are 601 candidates still at 66 bits, that is wrong. There were 119 left and I just checked those in.

If you do the [URL="http://www.mersenne.org/report_factoring_effort/?exp_lo=6600000&exp_hi=6699999&bits_lo=1&bits_hi=66&tftobits=72"]drill down[/URL] you will see there are none left at 66.

What happened to the missing 482?

More importantly, what else is missing?

James Heinrich 2016-04-23 16:26

[QUOTE=Gordon;432328]Unfortunately the report is wrong. More importantly, what else is missing?[/QUOTE]Hmm, that's actually a valid point. :redface:

I had processed all the primenet logs offline and imported the final processed database table. However, what I failed to consider is that other tables are also modified during the primenet data parsing. I might need to roll back the data by a couple weeks and reparse it on the server to make sure everything is lined up.

James Heinrich 2016-04-23 17:15

Actually I just need to run through the primenet data table I already have and make sure the TF limit tables are updated appropriately. I think it should only be the last 3-4 days or so that might be affected, but to be on the safe side I'm running through everything from 2016-04-01 onwards.

James Heinrich 2016-04-24 14:59

[QUOTE=Gordon;432328]Unfortunately the report is wrong. If you look at the 6.6m range it tells you there are 601 candidates still at 66 bits, that is wrong. There were 119 left and I just checked those in.[/QUOTE]Should be all straightened out now.

Gordon 2016-04-24 15:34

[QUOTE=James Heinrich;432420]Should be all straightened out now.[/QUOTE]

Looks good to me :smile:

Gordon 2016-04-27 21:13

[url]www.mersenne.ca[/url] just returns the Apache test page....

James Heinrich 2016-04-27 21:22

[QUOTE=Gordon;432678][url]www.mersenne.ca[/url] just returns the Apache test page....[/QUOTE]It's working fine here...

Dubslow 2016-04-27 22:49

Works fine for me

Gordon 2016-04-27 22:50

[QUOTE=James Heinrich;432679]It's working fine here...[/QUOTE]

Seems your server is picky about which VPN proxy is connecting to it, if I put my location as in the US (173.234.43.118) the connection fails...temporarily disable the VPN it works :exclaim:

This behaviour has changed since yesterday.


All times are UTC. The time now is 19:50.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.