mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Data (https://www.mersenneforum.org/forumdisplay.php?f=21)
-   -   Newer milestone thread (https://www.mersenneforum.org/showthread.php?t=13871)

chalsall 2016-12-21 20:58

[QUOTE=Madpoo;449713]I'm having some trouble figuring out exactly how I should go about adding a predicted expiration time to account for machines that haven't reported in.[/QUOTE]

What about a simple min() function across all the clauses?

[URL="http://www.mersenne.org/assignments/?exp_lo=70247057&exp_hi=&execm=1&exdchk=1&exp1=1&extf=1"]70247057 is a good example.[/URL] It was probably a Cat 1 when assigned -- thus it has to report in at least once every 30 days.

I understand that this is a "front end vs. back end" issue. The right hand doesn't know what the left hand is doing. But it confuses the users.

Madpoo 2016-12-22 01:23

[QUOTE=chalsall;449720]What about a simple min() function across all the clauses?[/QUOTE]

Maybe... right now it's a CASE/WHEN and it's pretty straightforward.

To use min() I'd need to reorganize it, plus it wouldn't perform as well because it would have to figure out the expiration in *all* cases instead of doing them in order with a fall-through condition.

Right now I can't just add another when/then to the CASE because the expiration for "hasn't checked in for XX days" is going to be different than the expiration for other reasons.

Not insurmountable, I just need to re-think how it's being done and make sure however it's done is still performant. In the end, I might just have to do as you say, figure out the expiration for each case and pick whichever one would be first. In the case of the "hasn't reported in for xx days" I'd also make sure it only returns a value at all if it's been more than 15-20 days or whatever, otherwise something that checked in the day before will say "29 days to expire" based on that clause alone. :smile:

Actually it may be easier to use my existing code and then also figure out the expiration based on "last update if it's been >= 20 days" and figure out which one is smaller just between those two things.

Ugh... It'd be ugly but should still work fast enough to show days-to-expire for lists of many assignments at once. It's one of those things that you think 'it's fast enough' but when you iterate it hundreds of times you realize just how bad it is. :smile:

axn 2016-12-22 03:24

[QUOTE=Madpoo;449728]In the case of the "hasn't reported in for xx days" I'd also make sure it only returns a value at all if it's been more than 15-20 days or whatever, otherwise something that checked in the day before will say "29 days to expire" based on that clause alone. :smile:[/QUOTE]

I'd suggest something smaller like 10 days, which would be a good indicator that something has f'ed up, and therefore we should start the count down.

Madpoo 2016-12-22 04:03

[QUOTE=axn;449733]I'd suggest something smaller like 10 days, which would be a good indicator that something has f'ed up, and therefore we should start the count down.[/QUOTE]

Meh... I mocked it up and went with 20 days as the threshold for using that expiration (if lower than any other clause).

You can see it in action here:
[URL="http://www.mersenne.org/assignments/?exp_lo=70312159"]Old version[/URL]
[URL="http://www.mersenne.org/assignments/default.mock.php?exp_lo=70312159"]New version[/URL]

chalsall 2016-12-22 17:51

[QUOTE=Madpoo;449736]You can see it in action here:
[URL="http://www.mersenne.org/assignments/?exp_lo=70312159"]Old version[/URL]
[URL="http://www.mersenne.org/assignments/default.mock.php?exp_lo=70312159"]New version[/URL][/QUOTE]

Looks good.

And not trying to tell you how to chew gum, but this is a perfect situation for caching calculations / knowledge. To the best of my understanding Primenet's back-end only does these calculations once per day (somewhere around midnight UTC) during the expiry process.

Leverage on that to update a table which your front-end can use. Separately, when a client reports in update the related records, or mark the records as "dirty" and update them using a low priority process.

Thus your SELECT statement would only be a min() across previously calculated values. Should scale well for large queries.

petrw1 2016-12-22 20:31

[QUOTE=Madpoo;449566]
I'd been eyeing those two slowpokes, thinking about poaching them myself, but, meh... someone else might anyway.
[/QUOTE]

Could a requirement to get Cat 0/1 be that you need to provide an e-Mail address (assumed valid) so that when they stop reporting (especially so close to the end) they could be NUDGED?

No response with a reasonably long time frame might be hint.

But more accurately you might get a response that says "OOPS Sorry, I moved. PC Died. I quit.,, etc." Then you would know if they are truly abandoned.

chalsall 2016-12-23 00:11

[QUOTE=petrw1;449761]But more accurately you might get a response that says "OOPS Sorry, I moved. PC Died. I quit.,, etc." Then you would know if they are truly abandoned.[/QUOTE]

Never send a human to do a machines' job.

Madpoo 2016-12-23 15:58

[QUOTE=chalsall;449754]Looks good.

And not trying to tell you how to chew gum, but this is a perfect situation for caching calculations / knowledge. To the best of my understanding Primenet's back-end only does these calculations once per day (somewhere around midnight UTC) during the expiry process.

Leverage on that to update a table which your front-end can use. Separately, when a client reports in update the related records, or mark the records as "dirty" and update them using a low priority process.

Thus your SELECT statement would only be a min() across previously calculated values. Should scale well for large queries.[/QUOTE]

Personally, I like having the real-time "days to expire" especially for the case when a machine checked in for the first time after getting the assignment.

There's no nightly task that stores the expire-time for each assignment, it just looks at each one and makes the decision right there "keep or toss", just a binary decision.

Pre-computing the stats could be done if it needed to be, and maybe something more often than daily, but we'll see how it goes. I just timed the current function with my new one that includes the extra step of checking the "last checked in" clause, and for 1000 rows it takes an extra 2 seconds. Oh, 2 seconds you say... that's not bad. Well, it's the difference between 5 seconds and 7 seconds total, so... 40% slower. It also made me think my original function with CASE statements is still slow. (EDIT: It takes 1 second to return that many rows without including any expiry info at all).

In fact, using a function over a lot of rows has apparently always been a performance issue, at least in MSSQL. I didn't get all the technical details, but something about having poor execution plans in that case, or something.

Well, I may take an extra gander at the whole thing and see if I can't get this running any faster, otherwise I may go with plan B of precomputing the whole thing in a sproc, where I can use inline TSQL instead of a function for faster runs over large # of rows.

chalsall 2016-12-23 19:35

[QUOTE=Madpoo;449802]There's no nightly task that stores the expire-time for each assignment, it just looks at each one and makes the decision right there "keep or toss", just a binary decision.[/QUOTE]

A perfect opportunity to cache knowledge. An INSERT on duplicate UPDATE statement once a day into a table is probably less expensive than all the queries asking about the lowest "n" candidates throughout the day.

And I hear you about wanting "real-time" results. This could be achieved by updating the same table's related records and fields every time a machine checks in.

Entirely up to you of course. :smile:

Madpoo 2016-12-26 20:19

Those slow-moving exponents in the 69M range (that will expire imminently)... I just looked back at their full histories (both by the same user).

It's weird... sometimes that user will check in a status showing 4.5% increase from the previous day, but then it could go weeks (checking in daily) with zero progress. Then it'll update and show a marginal (0.1 %) increase from the day before, then weeks again of nothing, and another 4.5% jump out of nowhere.

Very strange. It must not be running full time, or there are some other CPU hungry things running (a LOT) that keep Prime95 from doing any work, but it's still running enough to check in and say "nothing changed".

Anyway, as of right now, those two from that user (69110411 and 69110441) are [I]saying[/I] they'll be done in in 0-1 days, and they'll expire in 0-3 days. But when I look at their average daily progress, they're projected to finish in 10.6 and 1.8 days (respectively).

In other words, both of them will expire before they're projected to finish, but like I mention, their progress has been so sporadic, their average daily rate of progress is misleading when it comes to that final stretch. They might or might not have a burst of energy and finish in time.

It's sometimes strange to peek at the daily progress and see things like that. I don't know what's happening that it's still checking in daily for weeks (sometimes months!) at a time without the needle moving a single blip, but it happens, followed by odd bursts of "getting stuff done".

rudy235 2016-12-26 22:08

[QUOTE=Madpoo;449957]Those slow-moving exponents in the 69M range (that will expire imminently)... I just looked back at their full histories (both by the same user).

It's weird... sometimes that user will check in a status showing 4.5% increase from the previous day, but then it could go weeks (checking in daily) with zero progress. Then it'll update and show a marginal (0.1 %) increase from the day before, then weeks again of nothing, and another 4.5 %...

[stuff deleted]

It's sometimes strange to peek at the daily progress and see things like that. I don't know what's happening that it's still checking in daily for weeks (sometimes months!) at a time without the needle moving a single blip, but it happens, followed by odd bursts of "getting stuff done".[/QUOTE]

Seems clear to me that if by 0 days to expire it means before 00 hours 2016 12 27 that 691110441 will expire. The other 691110411 might finish in time but does not seem probable. It is indeed a pity that being so close to 100% they fail to produce, but hopefully they might qualify for the 2nd LL verification.


All times are UTC. The time now is 23:11.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.