[QUOTE=swl551;321829]Is this the calm before some storm.[/QUOTE]

Boom BOOOOOOOOOOOOMM!! :smile:


...
Hmm, Did I miss something?:loco:
...

[QUOTE=kracker;321830]Boom BOOOOOOOOOOOOMM!! :smile:


...
Hmm, Did I miss something?:loco:
...[/QUOTE]

I don't think you missed anything. I'm just wondering where everyone is... So quiet...

[QUOTE=swl551;321831]I don't think you missed anything. I'm just wondering where everyone is... So quiet...[/QUOTE]

Hmm, yeah, but this whole section of the forum is sorta quiet, Haha.

What we neeed is to discover a new priiime!

[QUOTE=kracker;321833]Hmm, yeah, but this whole section of the forum is sorta quiet, Haha.

What we neeed is to discover a new priiime![/QUOTE]
As you didn't specify which type of prime, may I suggest this thread:
[URL="http://www.mersenneforum.org/showthread.php?t=13180&page=49"]http://www.mersenneforum.org/showthread.php?t=13180&page=49[/URL]

[SIZE="1"]Yet I guess you mean a new [B]Mersenne [/B]prime.[/SIZE]

[QUOTE=sonjohan;321877]As you didn't specify which type of prime, may I suggest this thread:
[URL]http://www.mersenneforum.org/showthread.php?t=13180&page=49[/URL]

[SIZE=1]Yet I guess you mean a new [B]Mersenne [/B]prime.[/SIZE][/QUOTE]

Haha, yes a mersenne prime.... :razz:
Sorry, I knew, my brain wasn't working at all then.

After several thousand exponents with no problems, I have a duplicate that I can't narrow down. G72 says 401 TF assignments, MISFIT says 402. Since the duplicate detector isn't finding anything, it must be one that was already processed. The numbers matched after yeserdays submission, so the duplicate just showed up.

@Scott, can you make MISFIT compare current work to the results (sent) folder? Or maybe hold on to completed results in a table to compare until the user resets? Without the ability to compare current work to completed with there is no easy way to find the duplicate.

[QUOTE=flashjh;321940]After several thousand exponents with no problems, I have a duplicate that I can't narrow down. G72 says 401 TF assignments, MISFIT says 402. Since the duplicate detector isn't finding anything, it must be one that was already processed. The numbers matched after yeserdays submission, so the duplicate just showed up.

@Scott, can you make MISFIT compare current work to the results (sent) folder? Or maybe hold on to completed results in a table to compare until the user resets? Without the ability to compare current work to completed with there is no easy way to find the duplicate.[/QUOTE]

Finally a challenge!!!.. I'll come up with something....

Are you sure it is not a multi-range assignment that you manually split out into distinct phases?

[QUOTE=swl551;321949]Finally a challenge!!!.. I'll come up with something....

Are you sure it is not a multi-range assignment that you manually split out into distinct phases?[/QUOTE]
Yes, I only did those two a while back, everything else is a copy/paste from G72.

[QUOTE=flashjh;321975]Yes, I only did those two a while back, everything else is a copy/paste from G72.[/QUOTE]


Don't delete your GIOM_SENT files. My utility will do a one-time migration of all the data in those files into a database file. GIOM export will be modified to load rows into this database. The log files will remain in case you want to re-run an export/upload.

The database file will just be a CSV containing the key data elememnts. You'll have to use advanced tools like Excel to do "stuff" with the data. (search, order, graph etc)

[QUOTE=swl551;321983]Don't delete your GIOM_SENT files. My utility will do a one-time migration of all the data in those files into a database file. GIOM export will be modified to load rows into this database. The log files will remain in case you want to re-run an export/upload.[/QUOTE]
I have all the files in the GIOM_SENT since MISFIT started. Actually, I have all my results files since I started G72, but that's besides the point ;)
[QUOTE]The database file will just be a CSV containing the key data elememnts. You'll have to use advanced tools like Excel to do "stuff" with the data. (search, order, graph etc)[/QUOTE]

Perfect. Will it check for duplicates every time or only when a button is clicked?

BTW - Does MISFIT collect results immediately before submitting results when on a schedule?

[QUOTE]I'm just wondering where everyone is... So quiet...[/QUOTE]

It's so quiet because it's working well...

Thanks again for your time spent on this... I have so little interaction anymore, sometimes I [I]almost[/I] forget about all the systems that I have running :smile:

[QUOTE=flashjh;321987]I have all the files in the GIOM_SENT since MISFIT started. Actually, I have all my results files since I started G72, but that's besides the point ;)


Perfect. Will it check for duplicates every time or only when a button is clicked?

BTW - Does MISFIT collect results immediately before submitting results when on a schedule?



It's so quiet because it's working well...

Thanks again for your time spent on this... I have so little interaction anymore, sometimes I [I]almost[/I] forget about all the systems that I have running :smile:[/QUOTE]

I'm trying to avoid having MISFIT do any analysis on the exported data which is why I'm leaning to .CSV. A format that other tools can manipulate with ease. For now consolidated storage of all results is the first goal. After that is built we'll look at deeper functionality native to MISFIT.

Hmm, before it escapes my mind, is there to balance *only* specific workers?
For example only, I have 2 projects I run, first proj. on workers 1 and 2, the second on workers 3 and 4.
Is there a way so I can balance without some of them "leaking"
Or simply, balance only specific workers as... said above? :smile:

Thanks!

[QUOTE=kracker;322010]Hmm, before it escapes my mind, is there to balance *only* specific workers?
For example only, I have 2 projects I run, first proj. on workers 1 and 2, the second on workers 3 and 4.
Is there a way so I can balance without some of them "leaking"
Or simply, balance only specific workers as... said above? :smile:

Thanks![/QUOTE]

Sorry kracker there is no way to do that now and setting up [B]balance groups[/B] would be a lot of code. Maybe Santa will be nice to you this year.

utility to migrate all GIOM_SENT txt files into ONE .csv file
 

The utility is done: MigrateGIOMSENT2csv


Here is what you see on startup of the utility
[CODE]
GIOM_SENT file aggregator v 1.0
Run this from main MISFIT directory where GIOM_SENT is a subdirectory.
Output file will be UPLOADED_RESULTS.CSV
Overwritten everytime this utility is executed.
PRESS ANY KEY TO CONTINUE
[/CODE]


get from [URL]http://www.mersenneforum.org/misfit/[/URL]

Future enhancement to MISFIT will build on this.

[QUOTE=kracker;322010]Hmm, before it escapes my mind, is there to balance *only* specific workers?
For example only, I have 2 projects I run, first proj. on workers 1 and 2, the second on workers 3 and 4.
Is there a way so I can balance without some of them "leaking"
Or simply, balance only specific workers as... said above? :smile:

Thanks![/QUOTE]
Just run two instances of MISFIT. You can rename the exe so it allows two instances to run.

Set up each to manage the grouping you want to "relate"

[QUOTE=JMLX;322027]Just run two instances of MISFIT. You can rename the exe so it allows two instances to run.

Set up each to manage the grouping you want to "relate"[/QUOTE]

I believe(?) MISFIT doesn't allow more than one instance. Hmm. maybe misfit1.exe?:razz:

Anyone else interested in "balance groups"
 

[QUOTE=kracker;322010]Hmm, before it escapes my mind, is there to balance *only* specific workers?
For example only, I have 2 projects I run, first proj. on workers 1 and 2, the second on workers 3 and 4.
Is there a way so I can balance without some of them "leaking"
Or simply, balance only specific workers as... said above? :smile:

Thanks![/QUOTE]

Kracker has defined the scenario where workers/instances would be grouped together and the balancing of work would occur inside that group. Does anyone else have interest in this feature or additional ideas on how it could work?

[QUOTE=swl551;322261]Kracker has defined the scenario where workers/instances would be grouped together and the balancing of work would occur inside that group. Does anyone else have interest in this feature or additional ideas on how it could work?[/QUOTE]I've thought about it before, and I could use it sometimes. I thought maybe you could put checkmarks/checkbox next to each worker in the list and only the checked workers would get balanced. That would work well for a once-in-a-while scenario.



Edit: Just finished analysis of the data looking for duplicates. The "duplicate" from the 17th, well, I can't find it. Maybe it was a glitch? Here are the results from all the duplicates I found:
[CODE]
FF EXPONENT FR TO UPLOADED
0 59508727 72 73 10/24/2012 6:53
0 59508727 72 73 10/24/2012 17:44
0 59733463 71 73 10/29/2012 16:06
0 59733463 71 73 10/30/2012 8:28
0 59794937 72 73 10/31/2012 19:20
0 59794937 72 73 11/1/2012 20:38
0 59881543 71 73 11/4/2012 22:34
0 59942557 71 73 11/5/2012 11:19
0 59881543 71 73 11/5/2012 13:26
0 59943703 71 73 11/5/2012 20:35
0 59945363 71 73 11/6/2012 3:54
0 59942557 71 73 11/6/2012 5:58
0 59947549 72 73 11/6/2012 9:06
0 59949437 72 73 11/6/2012 14:20
0 59952029 72 73 11/6/2012 19:15
0 59953687 72 73 11/7/2012 0:27
0 59955251 71 73 11/7/2012 8:48
0 59943703 71 73 11/7/2012 9:51
0 59956709 71 73 11/7/2012 17:14
0 59958421 71 73 11/8/2012 0:42
0 59959829 71 73 11/8/2012 9:07
0 59930539 71 73 11/8/2012 9:07
0 59962057 72 73 11/8/2012 14:19
0 59945363 71 73 11/8/2012 14:19
0 59964173 71 73 11/8/2012 22:45
0 59930539 71 73 11/8/2012 23:48
0 59965519 71 73 11/9/2012 7:05
0 59947549 72 73 11/9/2012 9:11
0 59966947 72 73 11/9/2012 12:22
0 59939107 71 73 11/9/2012 14:30
0 59968343 71 73 11/9/2012 20:46
0 59969753 71 73 11/10/2012 4:04
0 59939107 71 73 11/10/2012 4:04
0 59949437 72 73 11/10/2012 4:04
0 59937413 71 73 11/10/2012 6:11
0 59971621 71 73 11/10/2012 12:29
0 59973031 71 73 11/10/2012 20:49
0 59937413 71 73 11/10/2012 23:04
0 59952029 72 73 11/10/2012 23:04
0 59974489 71 73 11/11/2012 4:25
0 59975681 71 73 11/11/2012 12:54
0 59953687 72 73 11/11/2012 18:11
0 59977237 71 73 11/11/2012 20:20
0 59951587 71 73 11/12/2012 0:35
0 59979133 71 73 11/12/2012 4:46
0 59980721 71 73 11/12/2012 12:04
0 59951587 71 73 11/12/2012 15:13
0 59983277 71 73 11/12/2012 20:31
0 59985383 71 73 11/13/2012 7:20
0 59955251 71 73 11/13/2012 7:20
0 59992157 71 73 11/13/2012 19:50
0 59993377 71 73 11/14/2012 4:33
0 59994661 72 73 11/14/2012 9:43
0 59990641 71 73 11/19/2012 3:37
0 59981377 71 73 11/19/2012 3:37
0 59990641 71 73 11/19/2012 4:39
0 59981377 71 73 11/19/2012 4:39
0 59969753 71 73 11/20/2012 16:30
0 59974489 71 73 11/20/2012 16:30
0 59971621 71 73 11/20/2012 16:30
0 59966947 72 73 11/20/2012 16:30
0 59977237 71 73 11/20/2012 16:30
0 59965519 71 73 11/20/2012 16:30
0 59959829 71 73 11/20/2012 16:30
0 59964173 71 73 11/20/2012 16:30
0 59980721 71 73 11/20/2012 16:30
0 59975681 71 73 11/20/2012 16:30
0 59985383 71 73 11/20/2012 16:30
0 59994661 72 73 11/20/2012 16:30
0 59968343 71 73 11/20/2012 16:30
0 59956709 71 73 11/21/2012 11:53
0 59958421 71 73 11/21/2012 11:53
0 59993377 71 73 11/21/2012 11:53
0 59979133 71 73 11/21/2012 11:53
0 59983277 71 73 11/21/2012 11:53
0 59962057 72 73 11/21/2012 11:53
0 59973031 71 73 11/21/2012 11:53
0 59992157 71 73 11/21/2012 11:53[/CODE]
From the analysis, I can see that the problem was getting worse for some reason (and I was wasting a lot of time!). After the new version came out and I started uploading once per day, the problem [U]immediately disappeared[/U] and has not happened since. Also of note, none of the duplicates were exponents with factors. I wish I knew why it appeared that there was a duplicate the other day? I don't have any .html log files for the old duplicates since the version on the Nov 21st started those. The logs from 17th - 18th don't show the duplicate either, so who knows?

As for the .csv tool, very helpful. Since it was the first time I put it to use, I also compiled the .txt files myself and did a separate analysis on the duplicates - both versions were a perfect match.

Edit2: The FLASH report is perfect!

@FLASHJH
 

First, guess who I named the "FLASH" report after...:showoff:

Anyway I don't think we'll ever get the exact answer here. The most likely culprit was over-active work balancing, but even under controlled testing we couldn't make it produce duplicates.

I think the key here is that the program has been improved to provide prevention, detection, forensics and analysis tools.

Since the first pubic release to the latest, [U]hundreds[/U] of hours of work has been spent improving things.

If there are any future oddities hopefully we'll have everything in place to get accurate answers.

Direct Select Balance Work feature
 

See screenshot on how balancing work between "selected" instances will be implemented.

Considering the minimal interest for "instance grouping", and the complexity of building an "instance group" manager, I think the ability to "select and balance" sort of meets in the middle and provides the desired outcome.

[QUOTE=swl551;322368]See screenshot on how balancing work between "selected" instances will be implemented.

Considering the minimal interest for "instance grouping", and the complexity of building an "instance group" manager, I think the ability to "select and balance" sort of meets in the middle and provides the desired outcome.[/QUOTE]

I think that will be a nice feature.

[QUOTE=swl551;322368]See screenshot on how balancing work between "selected" instances will be implemented.

Considering the minimal interest for "instance grouping", and the complexity of building an "instance group" manager, I think the ability to "select and balance" sort of meets in the middle and provides the desired outcome.[/QUOTE]

Yay! That will work very nicely for me atleast :smile:

That looks great! :smile:

2.2.6 now available
 

VERSION 2.2.6
1. New feature allowing [U]Direct Selection[/U] of instances to balance work between. Just click the instances in the grid you want to balance and then right click. Use CTRL key to select non contigious blocks, and SHIFT for contigious blocks. (standard windows keys). This feature is independent of auto-balancing or clicking the "Balance All" button. Depending on your needs you may use all, none, or just one of the three.

2. Minor tweaks.


Get from [URL]http://www.mersenneforum.org/misfit/[/URL]

Special delivery for Mr. Kracker
 

[QUOTE=kracker;320774]In the leader board, is it possible to display specific categories? ex. (rank in TF, P-1, LL etc)[/QUOTE]


VERSION 2.2.7
1. Leaderboard now allows selection of GIMPS_OVERALL and GIMPS_TF statistics. I am only adding [B]TF[/B] since it is the only statistic, other than ALL, that is relevant to MISFIT.


Get from [URL]http://www.mersenneforum.org/misfit[/URL]

[QUOTE=swl551;322603]VERSION 2.2.7
1. Leaderboard now allows selection of GIMPS_OVERALL and GIMPS_TF statistics. I am only adding [B]TF[/B] since it is the only statistic, other than ALL, that is relevant to MISFIT.


Get from [URL]http://www.mersenneforum.org/misfit[/URL][/QUOTE]

Wha.. Noooo... Why did you select my pathetic position?? :no::rant:

...

Thanks, as always looking good. :smile:

[QUOTE=kracker;322606]Wha.. Noooo... Why did you select my pathetic position?? :no::rant:

...

Thanks, as always looking good. :smile:[/QUOTE]

If I picked FLASHJH [U]everyone[/U] would feel bad....

[QUOTE=swl551;322607]If I picked FLASHJH [U]everyone[/U] would feel bad....[/QUOTE]

Everyone knows he has a whole house full of gpu's crammed he doesn't live there! So, not everyone1! :smile:

Except nucleon. Maybe.

[QUOTE=kracker;322616]
Except nucleon. Maybe.[/QUOTE]

Yes, except he's shut down for the summer. Wait till ~March...

@Scott: I figured out what's happening with the difference between G72 and MISFIT.

I have StopAfterFactor=0 so that each range gets fully tested. Since a typical instance takes about 6-8 hours to complete in this range, sometimes a factor is found and reported in the results.txt file but the range isn't done when the time comes for MISFIT to submit.

When the exponent is complete a line like the following is put in the results.txt file:
[CODE]found 1 factor for M60530593 from 2^71 to 2^73 [mfaktc 0.19 barrett76_mul32][/CODE]
MISFIT uses that line to calculate factors found and adds 1 (or whatever was found) to the completed even though the actual factor has already been submitted and G72 removed it from the assignments.

I suppose it would require quite a lot of changes to the results code and how MISFIT handles a factor if a range isn't complete. It does explain why I haven't been able to find the duplicate(s) over the last few days (which has varied from 0 to 2).

I don't think it's that big of a deal. Even if another factor is found in the same range after MISFIT submits, PrimeNet will still accept more factors (though I'm not sure if G72 will pickup the extra factor, that isn't a big deal either).

[QUOTE=flashjh;322701]@Scott: I figured out what's happening with the difference between G72 and MISFIT.

I have StopAfterFactor=0 so that each range gets fully tested. Since a typical instance takes about 6-8 hours to complete in this range, sometimes a factor is found and reported in the results.txt file but the range isn't done when the time comes for MISFIT to submit.......[/QUOTE]

Well that explains why the change to uploading once per day had an impact. This scenario was no where on my radar!

...
MISFIT would have to look into Results.Txt and make comparisons to exponents in WorkToDo.txt to know if it "OK" to export certain results. The increased overhead of this processing would add time to the export function. Until MFAKTC has file locking I think this is off the table.


However it may not be worth it anyway since it really isn't harming anything.

Very interesting though....

[QUOTE=swl551;322714]Very interesting though....[/QUOTE]
Agreed, and it only affects someone with StopAfterFactor=0 anyhow.

[QUOTE=swl551;322309]Since the first [B][U]pubic[/U][/B] release to the latest, [U]hundreds[/U] of hours of work has been spent improving things. [/QUOTE]

Calling Mr. Eddy, Mr. David Eddy...:smile:

[QUOTE=NBtarheel_33;322822]Calling Mr. Eddy, Mr. David Eddy...:smile:[/QUOTE]
:sos:

[QUOTE=NBtarheel_33;322822]Calling Mr. Eddy, Mr. David Eddy...:smile:[/QUOTE]

I put that in there on purpose to see if anyone was paying attention! :unsure:

2.3.x feature... discussion
 

In version 2.3.x the focus is on preventing IO with GIMPS during peak times. I have defined peak time (in my code) as 5 mins before the hour to 10 mins after the hour.

During this restricted window anything done via button clicks will get a dialog error indicating cannot perform task during peak time.

For automation it is more complicated.
Schedule exports/uploads can no longer be scheduled during that time frame. Existing schedule will be challenged during startup and you'd have to reschedule.

Time interval based exports/uploads will auto delay to past the restricted window if the event occurs within the restricted window.. this will cause the timer to drift past the window and future uploads won't fall in the window at all.

GIOM retries (if GIMPS is down) will also auto delay to past the restricted window if the event occurs within the restricted window. So if you have a schedule upload at 3:30 and it fails due to some primenet problem it retries every 30 mins. this puts the next retry right at peak time (4:00). In this case GIOM will delay that retry to after peak time.

In all automation scenarios the delay past peak time is randomized between 0 and 5 minutes. This delay is to prevent the [B]thousands[/B] of MISFIT instances from uploading results at the same time after the peak window has passed.

So is everyone OK with this?

screenshot shows interval based upload occuring at peak time and then being delayed.

[QUOTE=swl551;322946]So is everyone OK with this?[/QUOTE]Works for me.

[QUOTE=swl551;322946]I have defined peak time (in my code) as 5 mins before the hour to 10 mins after the hour.[/QUOTE]
How does the program determine this? Does it use local time, server time or perhaps a standard timezone like UTC? I ask this because not all time zones are an integral multiple away from UTC.

[QUOTE=axn;323026]How does the program determine this? Does it use local time, server time or perhaps a standard timezone like UTC? I ask this because not all time zones are an integral multiple away from UTC.[/QUOTE]

I think UTC, but I am not sure.

[QUOTE=axn;323026]How does the program determine this? Does it use local time, server time or perhaps a standard timezone like UTC? I ask this because not all time zones are an integral multiple away from UTC.[/QUOTE]


Good point! I was just using PC's local time, but now will convert it to UTC.

2.3.0 beta-1 is out there.
 

Changed peak time definition to

[FONT=Consolas][SIZE=2][COLOR=#a31515][FONT=Consolas][SIZE=2][COLOR=#a31515][FONT=Consolas][SIZE=2][COLOR=#a31515]2 mins before to 10 mins after the hour (UTC).
[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT]

Auto-Assigning work from MISFITworkToDo
 

Suggestion from flashjh... Screenshot highlights the concept. Open for comments.

[QUOTE=swl551;323437]Suggestion from flashjh... Screenshot highlights the concept. Open for comments.[/QUOTE]

Looks good, but I don't know about the 'assing-out' during event ;)

[QUOTE=flashjh;323438]Looks good, but I don't know about the 'assing-out' during event ;)[/QUOTE]
Time for new glasses!

[QUOTE=swl551;323439]Time for new glasses![/QUOTE]

I see you had deleted the picture when I posted that, if I had waited just a few minutes, I wouldn't have ever seen it :smile:

Goddamnit. I wanted to see that! :bangheadonwall:

The reason I requested the balance work after assign is to also ensure the duplicate check is done and any duplicates are removed. If for some reason the balancer determines that it doesn't need to do anything, can you still have the duplicate remover run?

[QUOTE=flashjh;323447]The reason I requested the balance work after assign is to also ensure the duplicate check is done and any duplicates are removed. If for some reason the balancer determines that it doesn't need to do anything, can you still have the duplicate remover run?[/QUOTE]

The BalanceWork function can be passed a "difference" value of 0 which will cause it to run all the way through even though nothing may be out of balance. That will cause the duplicate tests to be performed and duplicates ejected from the work plan.

2.3.0 beta 2 is out there
 

New feature since beta 1

Ability to stage work in bulk and have it drained only when GHZDaysToDo drops below threshold. In addition the number of factor rows to assign out in one event can be controlled. In the process thread AutoAssignWork occurs before AutoBalanceWork so balancing of newly assigned work can occur in harmony with assigning work if you have AutoBalanceWork turned on. The last feature is the ability to execute a custom program prior to assigning of work. The call to the custom program is a blocking call and the exit code is retrieved. If the custom program fails to start an exception is thrown and the AutoAssignWork will *not* complete. If the exit code is non-zero the MISFIT alert panel is shown with the error and AutoAssignWork will complete. The value of this system is to keep your workToDo lean and provide an external method of loading MISFITworkToDo.txt

Designed by FLASHJH

Get from [URL]http://www.mersenneforum.org/misfit[/URL]

[QUOTE=swl551;323783]Designed by FLASHJH[/QUOTE]

Implemented by swl551 :smile:

I'll work on testing later tonight. Thanks for working on this.

The archaic 1.6
 

This is all we had 6 months ago... (screenshot of first public release)

Proving that necessity is the mother of invention. Mostly proving that FLASHJH wants to dominate the world!

Thanks to everyone who has helped guide MISFIT along.

[QUOTE=swl551;323803]This is all we had 6 months ago... (screenshot of first public release)

Proving that necessity is the mother of invention. Mostly proving that FLASHJH wants to dominate the world!

Thanks to everyone who has helped guide MISFIT along.[/QUOTE]

Wow, has it already been half a year? Nice! :smile:

Pinky: "Gee, Brain, what do you want to do tonight?" The Brain: "The same thing we do every night, Pinky—try to [STRIKE]take over the world![/STRIKE] find the next Mersenne prime!"

The final feature of 2.3... AutoWorkFetch from GIMPS
 

Finally ready to code up the last major feature. Everything is in place and the final build out should be simple.

See screenshot, focus on the NEW section and feel free to comment.

Thanks

Scott

Set it and forget it!
 

Great, now I can just set my trial factoring to run and pretty much forget it's there unless I want to see how far up we've come in the stats. (or I get a stalled instance email)

Love it!
:smile:

I have it configured, but I don't think it's running the external file. How will I know if it's running or that it is trying to run the program? Is there a way to force it to run so I can test it out and make sure it's working before I leave it to run automatically?

[QUOTE=flashjh;323885]I have it configured, but I don't think it's running the external file. How will I know if it's running or that it is trying to run the program? Is there a way to force it to run so I can test it out and make sure it's working before I leave it to run automatically?[/QUOTE]

If the process does not start, meaning it cannot execute the external program, MISFIT will throw an exception and the Alert Panel will be displayed. There is no way to force it to execute beyond setting the conditions. Lower the Update Interval to 5 mins and at least you won't have to wait long to see what happens.

[CODE]
if (Convert.ToInt32(this.GHZDaysWorkToDo) < globals.cfg.settingAssignWorkWhenGHZdaysToDoDropsBelow)
{
try
{
if (globals.cfg.settingPreWorkAssignCommand.Length > 0)
{
Process P = new Process();
P.StartInfo.RedirectStandardError = true;
P.StartInfo.UseShellExecute = false;
P.StartInfo.FileName = globals.cfg.settingPreWorkAssignCommand;
P.Start();
// Do not wait for the child process to exit before
// reading to the end of its redirected error stream.

// Read the error stream first and then wait.
string error = P.StandardError.ReadToEnd();

P.WaitForExit();

if (P.ExitCode != 0)
PopSafeErrorForm(P.StartInfo.FileName + " returned error " + P.ExitCode.ToString());
}

MergeWork(false, globals.cfg.settingMaxAssignmentsPerAutoAssignEvent);
}
catch (Exception E)
{
PopSafeErrorForm("Error during AssignWorkViaAutomation\r\n" + E.Message);
}
}
[/CODE]

[QUOTE=swl551;323828]auto fetch from GIMPS [/QUOTE]
[thinking] It may be the time for me to start testing this toy... [/thinking]

Can I fetch a TF assignment from GIMPS, modify the bitlevels in the worktodo file (to make mfaktc to TF few bits higher, this is what the interface snip shows in your picture), do all the work, and report the result only once, at the end? (I mean, for each exponent, don't report the intermediary work until the 73rd bit is done, otherwise I lose the assignment after primenet's bitlevel is done). Or stop auto reporting and do the report manually, at the end, I won't mind to write a batch to do that. If we can do this, then the gpu72 won't be necessary anymore. Otherwise, I may wait until your program can cope with chalsall's toys. I mean auto fetch from gpu72.

[QUOTE=LaurV;323901][thinking] It may be the time for me to start testing this toy... [/thinking]

Can I fetch a TF assignment from GIMPS, modify the bitlevels in the worktodo file (to make mfaktc to TF few bits higher, this is what the interface snip shows in your picture), do all the work, and report the result only once, at the end? (I mean, for each exponent, don't report the intermediary work until the 73rd bit is done, otherwise I lose the assignment after primenet's bitlevel is done). Or stop auto reporting and do the report manually, at the end, I won't mind to write a batch to do that. If we can do this, then the gpu72 won't be necessary anymore. Otherwise, I may wait until your program can cope with chalsall's toys. I mean auto fetch from gpu72.[/QUOTE]
It can do all the things you described and more. It has been able to manually fetch work from GIMPS for a while now with the click of a button. Fully automated fetching is in the works and almost done.

A note on the reporting of completed assignments. If you have Stages=1 in your mfaktX.ini MISFIT may report intermediate results depending on the timing of the results upload. You can turn off auto-results uploading and do it manually if you prefer. Also I could code to not report intermediate results, but it requires some overhead in determining what to export. Not a big deal except that it increase chance of IO collision with results, workToDo and mfaktC. MISFIT and MFAKTO both implemented the .LCK file-locking system to prevent IO collisions, MFAKTC has not. When MFAKTC does implement file-locking the risk of collision will be gone.

Reporting of intermediate results has never caused me a problem since the next upload will complete the range and the chance of someone getting the same exponent from GIMPS is very low in the few hours between uploads. If that not tolerable or ideal consider it a limitations for now. FLASHJH may have additional insight on this and he is the heavy hitter in the MISFIT community.

See additional screenshot covering results uploading and stalled process detection.

[QUOTE=swl551;323886]If the process does not start, meaning it cannot execute the external program, MISFIT will throw an exception and the Alert Panel will be displayed.[/QUOTE]
Ok. I was clicking update stat and it didn't run, but I never tried setting update to 5 min. I posted at 1852(L) and at 1946(L) it ran fine, I just didn't get to see it. As long as it's running, I'm happy. Actually, it tried to run a few more times but had nothing to do, so it returned 1 and MISFIT reported that it returned 255. Is 255 expected for a 1 (or do you just report 255 for any non-zero return)?

Seems to be perfect, thanks again.

[QUOTE=flashjh;323929]Ok. I was clicking update stat and it didn't run, but I never tried setting update to 5 min. I posted at 1852(L) and at 1946(L) it ran fine, I just didn't get to see it. As long as it's running, I'm happy. Actually, it tried to run a few more times but had nothing to do, so it returned 1 and MISFIT reported that it returned 255. Is 255 expected for a 1 (or do you just report 255 for any non-zero return)?

Seems to be perfect, thanks again.[/QUOTE]

I return what the OS returns to me as the exit code. I do not override it with my own value.
[CODE]
if (P.ExitCode != 0)
PopSafeErrorForm(P.StartInfo.FileName + " returned error " + P.ExitCode.ToString());
[/CODE]

Cool. No big deal as long as we're not trying to use the exit code, but I'll look at the exit codes later.

Auto Fetching work from GIMPS is ready (closed beta)
 

2.3.0 beta 3 with Full Work Fetch is ready for closed beta. Please contact me directly for a copy.

[QUOTE=LaurV;323901]If we can do this, then the gpu72 won't be necessary anymore.[/QUOTE]

Well, the point of GPU272 was to distribute lower-than-accessible-from-PrimeNet expos for TF by quick workers. Auto-assignment was never the goal. I'd thus argue that even with autofetch-from-GIMPS, it would still be a good thing to mix in some GPU272 exponents in there.

Of course, is MISFIT could auto fetch from GPU272, that would be supremely awesome... :smile:

[QUOTE=Dubslow;323971]
Of course, is MISFIT could auto fetch from GPU272, that would be supremely awesome... :smile:[/QUOTE]

++1

[QUOTE=Dubslow;323971]....
Of course, is MISFIT could auto fetch from GPU272, that would be supremely awesome... :smile:[/QUOTE]

In red circle of the screenshot is where you put your GPU72 fetch utility and as long as it appends data to MISFITworkToDo.txt it will be fully integrated.

Beyond that there is nothing more I can do.

Well... as a suggestion for chalsall, if you merely implemented the same API as PrimeNet (perhaps with a few less restrictions w.r.t. things like HWID, GUID, etc.), then it should be almost trivial for swl551 to implement GPU272 autofetch (perhaps almost as simple as `sed -e s/http:\/\/v5.mersenne.org\//http:\/\/gpu72.com\//g` :smile:). And it should be fairly easy (relatively) for you to implement such an interface, since you've already done the interface part (not the backend) for the Prime95 proxy.

[QUOTE=Dubslow;323985]Well... as a suggestion for chalsall, if you merely implemented the same API as PrimeNet (perhaps with a few less restrictions w.r.t. things like HWID, GUID, etc.), then it should be almost trivial for swl551 to implement GPU272 autofetch (perhaps almost as simple as `sed -e s/http:\/\/v5.mersenne.org\//http:\/\/gpu72.com\//g` :smile:). And it should be fairly easy (relatively) for you to implement such an interface, since you've already done the interface part (not the backend) for the Prime95 proxy.[/QUOTE]

Currently I execute HTTP gets to mersenne.org and parse out the response strings. I'd do the same for GPU72. Really they should have web services for this stuff.

Can you point me to documentation on [QUOTE]sed -e s/http:\/\/v5.mersenne.org\//http:\/\/gpu72.com\//g[/QUOTE] as I have no idea what it is.

[QUOTE=swl551;323983]In red circle of the screenshot is where you put your GPU72 fetch utility and as long as it appends data to MISFITworkToDo.txt it will be fully integrated.

Beyond that there is nothing more I can do.[/QUOTE]
sed ... very familiar indeed ...

Edit: attached a manual to sed; there are several implementations but it's not too hard to make it do what you want

Edit2: @Dubslow
[QUOTE=Dubslow;323985]sed -e s/http:\/\/v5.mersenne.org\//http:\/\/gpu72.com\//g[/QUOTE]Were you implying that swl551 could just substitue all the mersenne.org with gpu72.com in his source?

[QUOTE=flashjh;323996]
Edit2: @Dubslow
Were you implying that swl551 could just substitue all the mersenne.org with gpu72.com in his source?[/QUOTE]

Something close to it. If chalsall implemented exactly the same interface as PrimeNet (or something very close to it), then what swl551 needs to do should be about that simple.

PS You can just google around for a sed manual (e.g. google `man sed`), there's many places to find it.

[QUOTE=Dubslow;324001]Something close to it. If chalsall implemented exactly the same interface as PrimeNet (or something very close to it), then what swl551 needs to do should be about that simple.

PS You can just google around for a sed manual (e.g. google `man sed`), there's many places to find it.[/QUOTE]


Remember my current methodolgy for fetching and reporting to gimps is HTTP GETs through the public web site pages.

I already wrote the code to execute gpu72 login and POST with parameters to get work from its assignment page. It just never moved forward and died on the vine. (fully functional)

I figured out what was wrong with the exit codes. Windows can't read %errorlevel% directly, but with 'Exit' from a batch file, Windows can read the ExitCode as you coded it. However, I was using Exit /B # which didn't work. What I should have been using was just Exit #. So now MISFIT is showing the correct result and only displays the # if > 0.

For the final version, can you make it an [I]option[/I] to display the Exit code if > 0? It's helpful to see the ExitCode while in development and for troubleshooting, but once I get everything working I won't need to see it anymore and it's actually kinda reduntant to continually see the error. The only thing I would actually need to know is if the external program failed to run at all, so if you can keep that as an option, that would be nice (though if it's been working fine all along, I wouldn't expect that to be an issue either).

Edit: You know I was just thinking that it would be a nice option to have MISFIT email the results of the external run each time. Is that something you could add also?

[QUOTE=flashjh;324004]I figured out what was wrong with the exit codes. Windows can't read %errorlevel% directly, but with 'Exit' from a batch file, Windows can read the ExitCode as you coded it. However, I was using Exit /B # which didn't work. What I should have been using was just Exit #. So now MISFIT is showing the correct result and only displays the # if > 0.

For the final version, can you make it an [I]option[/I] to display the Exit code if > 0? It's helpful to see the ExitCode while in development and for troubleshooting, but once I get everything working I won't need to see it anymore and it's actually kinda reduntant to continually see the error. The only thing I would actually need to know is if the external program failed to run at all, so if you can keep that as an option, that would be nice (though if it's been working fine all along, I wouldn't expect that to be an issue either).[/QUOTE]
"Alert on Non-Zero Exit Code".... in the next build.

Edit: You know I was just thinking that it would be a nice option to have MISFIT email the results of the external run each time. Is that something you could add also?[/QUOTE]

When the time comes for me to change employers you can expect a knock on your door!

[QUOTE=swl551;324006][QUOTE]Edit: You know I was just thinking that it would be a nice option to have MISFIT email the results of the external run each time. Is that something you could add also?[/QUOTE]

When the time comes for me to change employers you can expect a knock on your door![/QUOTE]
Just asking :smile:. If you get tired of the 'request' requests, just let me know.
[QUOTE=swl551;324005]"Alert on Non-Zero Exit Code".... in the next build.[/QUOTE]
Awesome!

[QUOTE=flashjh;324004].... You know I was just thinking that it would be a nice option to have MISFIT email the results of the external run each time. Is that something you could add also?[/QUOTE]


Please specify an example email content to be sure I understand the request.

[QUOTE=swl551;324008]Please specify an example email content to be sure I understand the request.[/QUOTE]
I'm thinking something simple like:

MISFIT pre-assign command exit code: ###.

### is the return code which is already displayed. I'm going to expand my pre-assign program to offer several return codes so this will help me in case something doesn't work, but it will also be nice to see a '0' return code when it does run. Since this is almost completely automated now, this will give me an indication that everything is progressing as expected.

2.3.0 beta 3 with Full Work Fetch
 

[QUOTE=swl551;323966]2.3.0 beta 3 with Full Work Fetch is ready for closed beta. Please contact me directly for a copy.[/QUOTE]

Re-posting as it got lost among other conversations.

screenshot of config
[url]http://mersenneforum.org/attachment.php?attachmentid=9106&d=1357591606[/url]

[QUOTE=flashjh;324011]I'm thinking something simple like:

MISFIT pre-assign command exit code: ###.

### is the return code which is already displayed. I'm going to expand my pre-assign program to offer several return codes so this will help me in case something doesn't work, but it will also be nice to see a '0' return code when it does run. Since this is almost completely automated now, this will give me an indication that everything is progressing as expected.[/QUOTE]


FlashJH,
Please test 2.3.0 beta 4 with these changes.

No showing of non zero error codes at all. Only show if fatal error calling external appliation. Results and Errors can be emailed. See new configuration.

[URL]http://mersenneforum.org/misfit/[/URL]

[QUOTE=swl551;324046]FlashJH,
Please test 2.3.0 beta 4 with these changes.

No showing of non zero error codes at all. Only show if fatal error calling external appliation. Results and Errors can be emailed. See new configuration.

[URL]http://mersenneforum.org/misfit/[/URL][/QUOTE]
I'll get it tested. Thanks!

[QUOTE=Dubslow;323971]Well, the point of GPU272 was to distribute lower-than-accessible-from-PrimeNet expos for TF by quick workers. Auto-assignment was never the goal. I'd thus argue that even with autofetch-from-GIMPS, it would still be a good thing to mix in some GPU272 exponents in there.

Of course, is MISFIT could auto fetch from GPU272, that would be supremely awesome... :smile:[/QUOTE]
No. That came later, with the spiders. The initial goal was to get TF assignments to 67-68 bits from PrimeNet and hold them (as different type of assignment) until the newer, faster GPUs, would TF them to 72 bits, taking the same time (as they were 10 times faster) as P95 would need to TF them to 68 or so, therefore eliminating part of the expos, and saving a lot of precious LL/DC time.

++1 for fetching expos from GPU272 part.

[QUOTE=LaurV;324121]No. That came later, with the spiders. The initial goal was to get TF assignments to 67-68 bits from PrimeNet and hold them (as different type of assignment) until the newer, faster GPUs, would TF them to 72 bits, taking the same time (as they were 10 times faster) as P95 would need to TF them to 68 or so, therefore eliminating part of the expos, and saving a lot of precious LL/DC time.[/QUOTE]
And the reason this needed to be done is because PrimeNet's TF reservation system handed out expos way beyond the wavefront. The spiders were the [URL="http://www.mersenneforum.org/showthread.php?p=320307#post320307"]whole point[/URL] of GPU272.

Gpu72 Work Fetcher
 

Pretty cool Ah?

If you want this please file a petition with Chalsall. I'll even release the full source code (c#) for this utility.

[QUOTE=swl551;324141]Pretty cool Ah?

If you want this please file a petition with Chalsall. I'll even release the full source code (c#) for this utility.[/QUOTE]

Cool! :smile:

I have 95% complete a Primenet like API for fetching work (and reporting estimated completion) which I need get back to Real Soon Now. (Life has gotten in the way.) But this is a worthwhile stop-gap menthodology until then. Thanks for doing this.

[QUOTE=swl551;324141]Pretty cool Ah?

If you want this please file a petition with Chalsall. I'll even release the full source code (c#) for this utility.[/QUOTE]

:max: Looks awesome!!

[QUOTE=chalsall;324145]Cool! :smile:

I have 95% complete a Primenet like API for fetching work (and reporting estimated completion) which I need get back to Real Soon Now. (Life has gotten in the way.) But this is a worthwhile stop-gap menthodology until then. Thanks for doing this.[/QUOTE]

Well, his PrimeNet autofetch also just uses the standard-"public"-frontend-fancy-formatted-pages of PrimeNet, so he doesn't actually use the API (yet).



Also, if I may suggest, why not make all the source code available? There's no reason not to.

2.3.0 officially released
 

VERSION 2.3.0
1. version 2.3.0 the focus is on preventing IO with GIMPS during peak times. I have defined peak time (in my code) as 2 mins before the hour to 10 mins after the hour (UTC).

During this restricted window anything done via button clicks will get a dialog error indicating cannot perform task during peak time.

For automation it is more complicated.
Schedule exports/uploads can no longer be scheduled during that time frame. Existing schedule will be challenged during start-up and you'd have to reschedule.

Time interval based exports/uploads will auto delay to past the restricted window if the event occurs within the restricted window.. this will cause the timer to drift past the window and future uploads won't fall in the window at all.

GIOM retries (if GIMPS is down) will also auto delay to past the restricted window if the event occurs within the restricted window. So if you have a schedule upload at 3:30 and it fails due to some primenet problem it retries every 30 mins. this puts the next retry right at peak time (4:00). In this case GIOM will delay that retry to after peak time.

In all automation scenarios the delay past peak time is randomized between 0 and 5 minutes. This delay is to prevent the thousands of MISFIT instances from uploading results at the same time after the peak window has passed.


2. Ability to stage work in bulk and have it drained only when GHZDaysToDo drops below threshold. In addition the number of factor rows to assign out in one event can be controlled. In the process thread AutoAssignWork occurs before AutoBalanceWork so balancing of newly assigned work can occur in harmony with assigning work if you have AutoBalanceWork turned on. The last feature is the ability to execute a custom program prior to assigning of work. The call to the custom program is a blocking call an the exit code is retrieved. If the custom program fails to start an exception is thrown and the AutoAssignWork will *not* complete. The return code can be emailed to you to help monitor any issues when you are away from your computer.

3. Ability to automatically fetch work from GIMPS. On an hourly basis MISFIT will check the number of rows left in MISFITworkToDo.txt and if below threshold will fetch work directly from GIMPS.


4. Minor tweaks and fixes no one will ever notice.

See screenshot showing configuration for the new features.

Get from [url]http://mersenneforum.org/misfit/[/url]

[QUOTE=swl551;324795]I put 2 hours in there figuring it was an extremely long duration. Why is 2 hours max interval too short?

Post your follow up in the MISFIT forum instead of on this thread.

thx[/QUOTE]
I set it to the maximum because it creates (a tiny) overhead and I rarely check it.
I remember a post in the 'You might be addicted to GIMPS...' which addressed this tiny overhead.:smile:

I do however understand that it is needed to be low enough as the manual button 'update stats' does not trigger the auto-fetching process.

[QUOTE=sonjohan;324804]I set it to the maximum because it creates (a tiny) overhead and I rarely check it.
I remember a post in the 'You might be addicted to GIMPS...' which addressed this tiny overhead.:smile:

I do however understand that it is needed to be low enough as the manual button 'update stats' does not trigger the auto-fetching process.[/QUOTE]


Remember the auto-fetching process only fetches if the workToDo needs to be refilled. File-IO times are sub second if the files is local to the PC.
I think you are worried about a problem that does not really exist. However there is a work-around. edit the MISFITconfig.txt manually and set the value higher. MISFIT config editor will overwrite that value if you use it later, but that is a work around.

Enjoy

Scott

Adding GPU72 fetching into MISFIT
 

Since everything has gone so well with the [B]stand-alone[/B] GPU72 fetching tool I've decided to integrate the fetching code directly into MISFIT. The screen shots show the new WORK FETCH config screen where GPU72 is selected and where GIMPS is selected.


Feel free to comment.

Looks like a great addition! The G72 option won't reduce also, correct? How will the G72 username/password work?

[QUOTE=flashjh;325241]Looks like a great addition! The G72 option won't reduce also, correct? How will the G72 username/password work?[/QUOTE]

GPU72 [B]pledge[/B] cannot be overridden like I allow with work fetched from GIMPS. In other words it specifically tells what GPU72 should assign to you so there is no reason to modify the assignments.

GPU72 credentials managed on the Security tab and the password is fully encrypted.

Looking great! :smile:

Along with the code you're adding for G72 fetching, can you work in some error handling changes? Since everything is automated now, I hardly check on the systems anymore and twice now I've had the workers run out of work because an error window was displayed. I did get an email about it, but I wasn't able to fix the problem for quite a long time. I know it probably would not be too big an issue if I got a lot of assignments at once, but I'm trying to get less now that fetching is automatic.

So, I was wondering, can you MISFIT try to clear non-critical errors on its own (maybe after 15 minutes or so). Like file-lock issues, you could run a file lock cleanup or network path not found, it could clear the error and try again in a while. If you need more info, let me know.

[QUOTE=flashjh;325374]So, I was wondering, can you MISFIT try to clear non-critical errors on its own (maybe after 15 minutes or so). Like file-lock issues, you could run a file lock cleanup or network path not found, it could clear the error and try again in a while. If you need more info, let me know.[/QUOTE]

Somewhat related...

A relatively new participant (no names -- we've communicated privately) had configured MISFIT to try to get work only below 60.5M from GPU72 every ten minutes, not realizing that there are only 33 left which we don't already own -- and only one or two become available every week and they're usually only available at about 0100 UTC.

Would it be possible for MISFIT to be a bit smarter when it needs work, but the parameters the user has set don't return any assignments? Offering the user two options ("Desired", "Fallback"), and/or simply falling back to the defaults ("What Makes Sense", "0", "100000000"), would be logical.

Secondly, could you please set a UserAgent string in the POST request? Then I could do what Primenet does for automatic Prime95/mprime fetch requests, and provide substitutions myself.

(Just to be clear... Writing spiders is hard work. Thanks for your work so far. :smile:)

Edit: Human languages can be so subtle and ambiguous... To be explicit the issue I raise immediately above is not directly related to flashjh's observations and request next immediately above....

I can do the following very easily
[QUOTE=chalsall;325375]Somewhat related...

....Offering the user two options ("Desired", "Fallback"), and/or simply falling back to the defaults ("What Makes Sense", "0", "100000000"), would be logical.

Secondly, could you please set a UserAgent string in the POST request? Then I could do what Primenet does for automatic Prime95/mprime fetch requests, and provide substitutions myself.

[/QUOTE]
It should be noted that the existing version could be wrapped in a bat file that checks the errorlevel (result) and initiates another call pointing to a different config file thereby providing a secondary request. However the coding of primary/secondary configuration scenario is very simple so I'll knock that out.


What do you want the userAgent string to be: "MISFITWorkFetcher" ?

[QUOTE=flashjh;325374]Along with the code you're adding for G72 fetching, can you work in some error handling changes? Since everything is automated now, I hardly check on the systems anymore and twice now I've had the workers run out of work because an error window was displayed. I did get an email about it, but I wasn't able to fix the problem for quite a long time. I know it probably would not be too big an issue if I got a lot of assignments at once, but I'm trying to get less now that fetching is automatic.

So, I was wondering, can you MISFIT try to clear non-critical errors on its own (maybe after 15 minutes or so). Like file-lock issues, you could run a file lock cleanup or network path not found, it could clear the error and try again in a while. If you need more info, let me know.[/QUOTE]

You should be seeing the non-blocking MISFIT alert panel when automation detects an error condition instead of a modal dialog box.

However a file-lock cleanup can be automated just by looking at the age of the lock file. If > 10 mins..... delete.