Automatic fetch of Trial Factoring work for GPU mfakt*
 

Ok, now the fetching of P95 work is solved. I need a toy to fetch TF assignments from GPU272. In fact, what I need the toy/spider/minion/terrorist to do is:

1. read a command (or ini, cfg, etc) file, the only command I am interested now is "stop=0". If "stop" is 1, then exit. [edit: other commands I can implement by myself later]

2. check if "worktodo.txt" exists in the current folder, and if so, check its size. If the size is not zero, skip step 3. [edit: this is because mfaktc does not delete the file when he finishes the assignment, the file is still there, but its size is zero, as no assignment line inside. A nonzero file would mean that the previous assignment was not finished, but accidentally interrupted; edit2: or that the mfaktc is still running on it, see below]

3. if "stop" is zero [edit: obviously is zero, otherwise we won't be here!], fetch a [U][B]single[/B][/U] LL-TF assignment from GPU272 (customizable range/bitdepth would be fine, and I don't mind editing it from time to time, but fixed range and 72/73 bits is also fine, this given by the GPU72 server interface, he can give me one expo he likes, and the bitdepth he likes, I don't mind, something like yoyo's script and factorDB single composite fetching) and write it to worktodo.txt

4. call mfaktc with some parameters, in a specified folder and wait for it to finish (like >>start "mfaktc4" /D mf4 /I /LOW mf4\mfaktc4.exe -d 1) [edit2: if step3 was skipped, see above, there is deliberately no check if any mfaktc is running, more instances can be launched, and this is responsibility of the user. Let's keep it simple here]

5. report the result to whatever (primenet, gpu72, james' site, etc) (obviously it would need my PrimeNet user name and password, so better not be an exe file or black box. Batch/perl/source preferred).

6. repeat from 1 inclusive.

(the reason of 1 is kind of "graceful shutdown", I edit the ini file and put a "1" instead of 0, he would finish the current assignment and exit. Next time I would need to edit the ini again, to make it 0, but I don't mind).

Is anything like this (or close to it) available?
If so, I plan to test it with 4 or 6 copies/instances for a week.

I can do it by myself, except I don't know how to fetch the TF exponents (I assume I need some support/handshake for it from chalsall's side, and obviously it will need my GPU272 user and password, so not any bot can fetch thousands of assignments and never finish any) and I don't know how to report the results yet (this part can be solved, as there is some spider available, which I saved somewhere, but never used it, I can dig into it if it is a must). However, it would be nice to get a ready-made toy.

[QUOTE=LaurV;307921]I can do it by myself, except I don't know how to fetch the TF exponents (I assume I need some support/handshake for it from chalsall's side, and obviously it will need my GPU272 user and password, so not any bot can fetch thousands of assignments and never finish any) and I don't know how to report the results yet (this part can be solved, as there is some spider available, which I saved somewhere, but never used it, I can dig into it if it is a must). However, it would be nice to get a ready-made toy.[/QUOTE]

I have been thinking about this for quite some time -- it's time I start writing code for it. It will be in Perl.

I think I've figured out a graceful way of handing "trust" between the clients and the server. If after implementing and testing it is agreed it's workable, this might be a way of having mfakt* talk to Primenet in a trusted fashion which doesn't require secret code.

[QUOTE=chalsall;307926]
I think I've figured out a graceful way of handing "trust" between the clients and the server. If after implementing and testing it is agreed it's workable, this might be a way of having mfakt* talk to Primenet in a trusted fashion which doesn't require secret code.[/QUOTE]
What about Prime95? Is there something special about mfakt*/tf that allows this method to be open source?

The "special" rely in the things you can do with it. This is not like the bank account. You can't "steal" my work or destroy it, if you know my password. You can see what hardware I have, or some "secrets" which anyhow I boast them everyday on the forum. The only bad thing you can do with my gpu272/p95 password is to report results on my name. Be my guest. :razz:

[edit: moreover, P95 does not need your PrimeNet password, there is no place in P95 to put your PrimeNet password, you don't need it to fetch work and report results, and Chris's proxy doesn't know it, and does not need it. You can put in any of your computers the username LaurV and get work on my name, and increase my credit when you report the results. You wont need my password for it :razz:]

[edit2: and to answer your question, the "secret" part in P95 try to forbid you to report fake results (i.e. it deals with assignment keys, shifts, whatever). There is no such issue on mfaktc, a factor is a factor, you can't report a fake factor, and there is nothing to deal with fake "no factor" reports. One can report fake "no factors" and get thousands of TF credit. The point is that those expos will have to be LL-ed anyhow. When I began mfakt-ing I raised this issue on the forum, but people said I am overreacting :smile:]

[QUOTE=Dubslow;307928]What about Prime95? Is there something special about mfakt*/tf that allows this method to be open source?[/QUOTE]

No. It's nothing special -- instead of "secret code" generating the secret hash (sh), I plan to have a "shared secret" to generate it. Basically each Computer on GPU72 will have a unique MD5 string which will be fetched by each user and placed in the fetching spider's config file(s). This will then be used to generate a hash which is sent with each message.

The only reason it wouldn't work well with Prime95/mprime is it would mean someone couldn't simply download the program and start running it.

That would be similar to the BOINC's two keys:
[CODE]User ID
[SIZE=1](Used in community functions) [/SIZE] [COLOR=blue]11..99 [/COLOR]
Account key
[SIZE=1](Provides full access to your account)[/SIZE] [COLOR=blue]fcb...666 [/COLOR]
Weak account key
[SIZE=1](Provides limited access to your account)[/SIZE] [COLOR=blue]147..._...777[/COLOR]
[/CODE]

If this made it's way to mfakto, that would be amazing.

[QUOTE=KyleAskine;307936]If this made it's way to mfakto, that would be amazing.[/QUOTE]

That's the plan. External spiders are simply short-term "duct tape".

"This is only temporary, unless it works..." - Red Green

Assistance requested...
 

Hey all.

OK, I've begun implementing the "fetching spider". I had hoped I would be able to provide Perl code which would work under both Linux and Windows. Unfortunately I've run into a problem I need some help with.

For context, I had planned on having the spider fetch work, build the worktodo.txt file with a single candidate, and then fork() a child which would exec() mfakt*. I want to do this so I can have the fetching spider grab another candidate in parallel, return completed work in parallel, monitor mfakt*, provide the output of mfakt* to the user, be able to kill() mfakt* when needed, etc.

Now I know that Windows doesn't actually have a fork() function, but the good people at ActiveState implemented "forked pseudo-processes". Details at [url]http://perldoc.perl.org/perlfork.html[/url]

Unfortunately, while the example work-around code for pipes works fine under Linux, it doesn't seem to work under Strawberry Perl on Windows. I'm wondering if those who use Perl under Windows could give the following code a try, and let me know if it works for them:

[CODE]#!/usr/bin/perl -w

TestFork(1);
TestFork(2);
exit;


sub TestFork {
($Test) = @_;

if (pipe_from_fork('BAR')) {
# parent
while (<BAR>) { print; }
close BAR;
print "\n\nTest completed. Was output seen from the child?\n\n\n";
} else {
# child

if ($Test == 1) {
for ($Cnt = 0; $Cnt<5; $Cnt++) {
print "We're here! Cnt: ${Cnt}\n";
sleep(1);
}
exit(0);
} else {
exec("dir");
}
}
}


sub pipe_from_fork ($) {
my $parent = shift;
pipe $parent, my $child or die;

my $pid = fork();
die "fork() failed: $!" unless defined $pid;

if ($pid) {
close $child;
} else {
close $parent;
open(STDOUT, ">&=" . fileno($child)) or die;
}
$pid;
}[/CODE]

The output should be something like this. The "We're here!" bit is the child process talking to the parent. The "fetch_test.pl" bit is the output of the exec()ed "dir" command, and might be different for you (if it works) as a function of what else is in the directory where you run the program.[CODE]We're here! Cnt: 0
We're here! Cnt: 1
We're here! Cnt: 2
We're here! Cnt: 3
We're here! Cnt: 4


Test completed. Was output seen from the child?


fetch_test.pl


Test completed. Was output seen from the child?[/CODE]

I'm particularly interested in those who run ActiveState's Perl, and/or cygwin's.

If I can't find a solution which works in all Perl environments, then I'll simply have to have different versions -- one for Linux, and one for Windows. But this is supposed to work!!!

(Have I mentioned I hate Windows? How could a modern operating system not provide the simple command "tail"?)

I don't understand your forking problem... (for-king, that is for me, isn't it?).

I use to start multiple windows processes in batch files using... start command. For example, next batch is "forking" 6 windows with 6 copies of mfaktc, any of them having its own folder, files, options, window title, etc, all run low priority, you can even set process affinity, etc.

[CODE]
start "mfaktc1" /D mf1 /I /LOW mf1\mfaktc1.exe -d 0
start "mfaktc2" /D mf2 /I /LOW mf2\mfaktc2.exe -d 0
start "mfaktc3" /D mf3 /I /LOW mf3\mfaktc3.exe -d 1
start "mfaktc4" /D mf4 /I /LOW mf4\mfaktc4.exe -d 1
start "mfaktc5" /D mf5 /I /LOW mf5\mfaktc5.exe -d 2
start "mfaktc6" /D mf6 /I /LOW mf6\mfaktc6.exe -d 2
[/CODE]

this certainly works in win7-64, what I am using on this comp, and i have variants for xp, vista, etc.

You implement the server side (i.e. accessing a link would pop me up one single line with the expo and bit levels, same as FactorDB pops a composite to factor when you access the right link) and let us deal with mfakting that expo and reporting the results. Well, if you want to do all in one, is even better. :razz:

[QUOTE=LaurV;308270]I don't understand your forking problem... (for-king, that is for me, isn't it?).

I use to start multiple windows processes in batch files using... start command. For example, next batch is "forking" 6 windows with 6 copies of mfaktc, any of them having its own folder, files, options, window title, etc, all run low priority, you can even set process affinity, etc.

[CODE]
start "mfaktc1" /D mf1 /I /LOW mf1\mfaktc1.exe -d 0
start "mfaktc2" /D mf2 /I /LOW mf2\mfaktc2.exe -d 0
start "mfaktc3" /D mf3 /I /LOW mf3\mfaktc3.exe -d 1
start "mfaktc4" /D mf4 /I /LOW mf4\mfaktc4.exe -d 1
start "mfaktc5" /D mf5 /I /LOW mf5\mfaktc5.exe -d 2
start "mfaktc6" /D mf6 /I /LOW mf6\mfaktc6.exe -d 2
[/CODE]
[/QUOTE]

1. Does this work under Linux?

2. Do you have any control of the "started" instances? Can you kill() them? Can you monitor their STDOUT or STDERR output?

Please forgive me for my pedantic nature, but if you're going to write software which will be used by others, it is in your best interest to make sure it will always work correctly.

There's a reason "Have you tried turning it off and on again?" is not funny to some....

[QUOTE=chalsall;308266]
I'm particularly interested in those who run ActiveState's Perl, and/or cygwin's.
[/QUOTE]
Me ActiveState.
[CODE]
This is perl 5, version 14, subversion 2 (v5.14.2) built for MSWin32-x64-multi-thread
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2011, Larry Wall

Binary build 1402 [295342] provided by ActiveState http://www.ActiveState.com
Built Oct 7 2011 15:19:36
[/CODE][CODE]
C:\Computing>perl -w fork.pl
Entering TestFork...
Entering pipe_from_fork...[/CODE]No output at all in your version. I added some prints, find attached.

[QUOTE=Brain;308275]Me ActiveState.
[CODE]
C:\Computing>perl -w fork.pl
Entering TestFork...
Entering pipe_from_fork...[/CODE]No output at all in your version. I added some prints, find attached.[/QUOTE]

Thank you very much Brain. Important data.

So it is dying (or freezing) in pipe_from_fork.

Could I ask you to change the pipe_from_fork subroutine to this, and then run again?

[CODE]sub pipe_from_fork ($) {
my $parent = shift;
print "pipe_from_fork - 1 -- $$\n";
pipe $parent, my $child or die;
print "pipe_from_fork - 2 -- $$\n";

my $pid = fork();
print "pipe_from_fork - 3 -- $$\n";
die "fork() failed: $!" unless defined $pid;
print "pipe_from_fork - 4 -- $$ -- PID: $pid\n";

if ($pid) {
print "pipe_from_fork - 5 -- $$ -- PID: $pid\n";
close $child;
print "pipe_from_fork - 5.1 -- $$ -- PID: $pid\n";
} else {
print "pipe_from_fork - 6 -- $$\n";
close $parent;
print "pipe_from_fork - 6.1 -- $$\n";
open(STDOUT, ">&=" . fileno($child)) or die;
print "pipe_from_fork - 6.2 -- $$\n";
}
print "pipe_from_fork - 7 -- $$\n";
$pid;
}
[/CODE]

Ready.
[CODE]Entering TestFork...
Entering pipe_from_fork...
pipe_from_fork - 1 -- 3440
pipe_from_fork - 2 -- 3440
pipe_from_fork - 3 -- 3440
pipe_from_fork - 4 -- 3440
pipe_from_fork - 5 -- 3440
pipe_from_fork - 7 -- 3440
pipe_from_fork - 3 -- -5324
pipe_from_fork - 4 -- -5324
pipe_from_fork - 6 -- -5324[/CODE]Not terminating on its own.

[QUOTE=Brain;308277]Ready.
[CODE]Entering TestFork...
Entering pipe_from_fork...
pipe_from_fork - 1 -- 3440
pipe_from_fork - 2 -- 3440
pipe_from_fork - 3 -- 3440
pipe_from_fork - 4 -- 3440
pipe_from_fork - 5 -- 3440
pipe_from_fork - 7 -- 3440
pipe_from_fork - 3 -- -5324
pipe_from_fork - 4 -- -5324
pipe_from_fork - 6 -- -5324[/CODE]Not terminating on its own.[/QUOTE]

OK. Thanks.

One last code change (above) which I'd love if you could run.

I suspect that 6 will return, but 6.1 might not, and 6.2 won't. I don't understand why yet.

Thanks for your help with this.

Nice experiment here. Strawberry, 5.16, 64 bits.

I added few prints into it:
[CODE]
#!/usr/bin/perl -w

TestFork(1);
TestFork(2);
exit;


sub TestFork {
($Test) = @_;

print "one ${Test}\n";
if (pipe_from_fork('BAR')) {
# parent
print "two ${Test}\n";
while (<BAR>) { print; }
print "three ${Test}\n";
close BAR;
print "\n\nTest completed. Was output seen from the child?\n\n\n";
} else {
# child
print "four ${Test}\n";
if ($Test == 1) {
print "five ${Test}\n";
for ($Cnt = 0; $Cnt<5; $Cnt++) {
print "We're here! Cnt: ${Cnt}\n";
sleep(1);
}
print "six ${Test}\n";
exit(0);
} else {
print "seven ${Test}\n";
exec("dir");
print "eight ${Test}\n";
}
}
}


sub pipe_from_fork ($) {
print "inside 1\n";
my $parent = shift;
pipe $parent, my $child or die;

print "inside 2\n";
my $pid = fork();
die "fork() failed: $!" unless defined $pid;

print "inside 3\n";
if ($pid) {
close $child;
print "inside 4\n";
} else {
close $parent;
open(STDOUT, ">&=" . fileno($child)) or die;
print "inside 5\n";
}
print "outside\n";
$pid;
}
[/CODE]

The output:
[CODE]
>perl chalsall.pl
Statement unlikely to be reached at chalsall.pl line 33.
(Maybe you meant system() when you said exec()?)
one 1
inside 1
inside 2
inside 3
inside 4
[COLOR=RoyalBlue]outside
inside 3[/COLOR]
two 1
Terminating on signal SIGINT(2)
[/CODE]

(it does not terminate by itself, need ctrl+c).
The blue lines, "outside" and the second "inside 3" lines, are sometime exchanged, if i run it many times, this means he is really multitasking there, the blue "inside 3" is print by the forked process which is faster sometimes. What else is doing, no idea...

[QUOTE=chalsall;308278]OK. Thanks.

One last code change (above) which I'd love if you could run.

I suspect that 6 will return, but 6.1 might not, and 6.2 won't. I don't understand why yet.

Thanks for your help with this.[/QUOTE]
Not terminating...[CODE]
Entering TestFork...
Entering pipe_from_fork...
pipe_from_fork - 1 -- 2344
pipe_from_fork - 2 -- 2344
pipe_from_fork - 3 -- 2344
pipe_from_fork - 4 -- 2344 -- PID: -6040
pipe_from_fork - 5 -- 2344 -- PID: -6040
pipe_from_fork - 3 -- -6040
pipe_from_fork - 4 -- -6040 -- PID: 0
pipe_from_fork - 5.1 -- 2344 -- PID: -6040
pipe_from_fork - 7 -- 2344
pipe_from_fork - 6 -- -6040
pipe_from_fork - 6.1 -- -6040[/CODE]

with my main and your function:

[CODE]one 1
pipe_from_fork - 1 -- 2652
pipe_from_fork - 2 -- 2652
pipe_from_fork - 3 -- 2652
pipe_from_fork - 4 -- 2652 -- PID: -4404
pipe_from_fork - 5 -- 2652 -- PID: -4404
pipe_from_fork - 3 -- -4404
pipe_from_fork - 5.1 -- 2652 -- PID: -4404
pipe_from_fork - 4 -- -4404 -- PID: 0
pipe_from_fork - 7 -- 2652
pipe_from_fork - 6 -- -4404
two 1
pipe_from_fork - 6.1 -- -4404
Terminating on signal SIGINT(2)[/CODE]

I'm running ActivePerl 5.10.1 Build 1006 and get the same result as Brain:[code]C:\Aliquot\Data>perl fork.pl
Entering TestFork...
Entering pipe_from_fork...
pipe_from_fork - 1 -- 5272
pipe_from_fork - 2 -- 5272
pipe_from_fork - 3 -- 5272
pipe_from_fork - 4 -- 5272 -- PID: -4680
pipe_from_fork - 5 -- 5272 -- PID: -4680
pipe_from_fork - 5.1 -- 5272 -- PID: -4680
pipe_from_fork - 7 -- 5272
pipe_from_fork - 3 -- -4680
pipe_from_fork - 4 -- -4680 -- PID: 0
pipe_from_fork - 6 -- -4680
pipe_from_fork - 6.1 -- -4680[/code]Also with no termination....

[QUOTE=Brain;308280]Not terminating...[/QUOTE]

OK. Thanks Brain and LaurV and schickel.

So the Perl statement "open(STDOUT, ">&=" . fileno($child)) or die;" doesn't work under Windows. Even though it is supposed to.

Hmmmm....

How about using the piped version of open? Here's something I whipped up a while back to run mfaktc from Perl. Works on activestate and cygwin.

[code]
open (MFAKTC, "$mfaktc_bin -tf $this_exp $this_depth $this_depth_end |");
my $factor_found = 0;
my $ok_end = 0;
while ($line = <MFAKTC>)
{
print LOG $line;
if ($line =~ /found \d+ factor\(s\) for M/)
{
$factor_found = 1;
}
elsif ($line =~ /cleared assignment/)
{
$ok_end = 1;
}
}
close MFAKTC;
[/code]

Might make it more difficult to run multiple copies of mfaktc, though

[QUOTE=kjaget;308289]How about using the piped version of open? Here's something I whipped up a while back to run mfaktc from Perl. Works on activestate and cygwin.[/QUOTE]

According to the documentation, that's not supposed to work.

But clearly the documentation is wrong. I'll drill down on your example shortly.

Thanks Kevin. :smile:

Have you investigated the various flavours of *nix utils for Windows out there? A quick search turned up some Q&As on StackOverflow regarding "tail" for Windows....

[QUOTE=schickel;308305]Have you investigated the various flavours of *nix utils for Windows out there? A quick search turned up some Q&As on StackOverflow regarding "tail" for Windows....[/QUOTE]

Yeah... My rant had to do with the fact that WinBlows doesn't include the command-line command "tail" by default. A fundamental command for anyone serious.

Anyone can install cygwin (for example) and get the tail command in the shell. But come on, why is this not available under the "Power Shell" which now allows "ls"? The power shell was supposed to show that Microsoft was serious about power users.

At the end of the day my opinion remains that M$ is a toy operating system -- meant for those who are not serious, but have a lot of money to spend....

[QUOTE=chalsall;308307]
At the end of the day my opinion remains that M$ is a toy operating system -- meant for those who are not serious, but have a lot of money to spend....[/QUOTE]

Until Adobe puts out a 'nix version of Photoshop I'm locked into Windows. I can't afford a Mac of similar power to my PC.

Still, I have plenty of frustrations with Windows going way back.:bangheadonwall::rakes:

[QUOTE=kladner;308312]Until Adobe puts out a 'nix version of Photoshop I'm locked into Windows. I can't afford a Mac of similar power to my PC.[/QUOTE]

Yeah. I hear you loud and clear. I have several clients in a similar situation.

The ironic thing is MacOS is actually based on an Open Source Unix-like kernel. Apple spent a bit of time and money writing a flashy UI, but the core is Unix.

There is no technical reason why Adobe couldn't make their software available for Linux. If they use the excuse "there are so many distributions to choose from", the answer is "Choose one -- Ubuntu would be fine".

Want to bet when this will happen?

[QUOTE=chalsall;308307]Yeah... My rant had to do with the fact that WinBlows doesn't include the command-line command "tail" by default. A fundamental command for anyone serious.[/quote]What I was thinking was finding the best/easiest of the publicly available stand alones out there, then you could either include it in the distribution of the future mfakt* Windows packages or make it available on the download page.

[QUOTE=schickel;308316]What I was thinking was finding the best/easiest of the publicly available stand alones out there, then you could either include it in the distribution of the future mfakt* Windows packages or make it available on the download page.[/QUOTE]

I don't need "tail" for my spider. That was just a tangential rant.

I want a fork() function which works no matter what version of Perl anyone has installed in any environment the Perl code finds itself.

I don't think that a small amount of code should come with a huge amount of baggage. Please forgive me, but in the Unix world it is possible to share a few lines of Perl which works without having to include an installer which downloads MBs of supporting libraries just to run.

[QUOTE=chalsall;308315]Yeah. I hear you loud and clear. I have several clients in a similar situation.

The ironic thing is MacOS is actually based on an Open Source Unix-like kernel. Apple spent a bit of time and money writing a flashy UI, but the core is Unix.

There is no technical reason why Adobe couldn't make their software available for Linux. If they use the excuse "there are so many distributions to choose from", the answer is "Choose one -- Ubuntu would be fine".

Want to bet when this will happen?[/QUOTE]

I wouldn't bet on it, but I would venture that it might happen about the same time that the waters around Barbados freeze a meter deep.:razz:

[QUOTE=kladner;308312]Until Adobe puts out a 'nix version of Photoshop I'm locked into Windows. I can't afford a Mac of similar power to my PC.

Still, I have plenty of frustrations with Windows going way back.:bangheadonwall::rakes:[/QUOTE]

[URL="http://www.gimp.org/"]GIMP[/URL] or [URL="http://inkscape.org/"]Inkscape[/URL]

[QUOTE=chalsall;308319]I don't need "tail" for my spider. That was just a tangential rant.

I want a fork() function which works no matter what version of Perl anyone has installed in any environment the Perl code finds itself.

I don't think that a small amount of code should come with a huge amount of baggage. Please forgive me, but in the Unix world it is possible to share a few lines of Perl which works without having to include an installer which downloads MBs of supporting libraries just to run.[/QUOTE]Sorry, I was obviously concentrating on the wrong issue there.

Getting back to something LaurV said about START, the answers are 1) no, not from the command line, and 2) limited control; yes, XP or higher has TASKKILL, which can kill other processes from the command line; and no, not when starting from a CMD box.

START spawns processes off and turns them loose; GUI programs open normally, console programs open with an independent console (CMD) window.

Unfortunately, if you want to get down as close as controlling the in/out pipes of a console program, you need to write to the Win32/64 API.

[SIZE="1"]Either that, or develop a GUI version of the various programs where the MBs of supporting libraries are already there deep in the bowels of the Windows installation.....[/SIZE]

[QUOTE=schickel;308340]Unfortunately, if you want to get down as close as controlling the in/out pipes of a console program, you need to write to the Win32/64 API.[/QUOTE]

Sigh... Thanks schickel. That was what I was afraid of.

Kinda strange that getting "down and dirty" and reading/writing the in/out/error pipes of a console program under Windows would be so very difficult.

We do this kind of thing every day under Unix, and we don't even really think about it....

[QUOTE=chalsall;308344]Sigh... Thanks schickel. That was what I was afraid of.

Kinda strange that getting "down and dirty" and reading/writing the in/out/error pipes of a console program under Windows would be so very difficult.

We do this kind of thing every day under Unix, and we don't even really think about it....[/QUOTE]Unfortunately it's not really that hard [i]if you write it in non-portable C/C++ code[/i]. So for a portable Perl-based solution, that's not really an option.

You seem to be going a little complicated here, what about a program that was like the submission spider, set to run every X hours and just checked the size of the worktodo.txt file? I always refill my worktodo on mfakt* when it drops below 1K (cause I try to check it daily but sometimes every other day), but if I had an automatic thing like this I could have it check to see if it was below say 200 bytes (approx 7 assignments) and if so, then grab 1-10 more (depending on the time between program runs) and append worktodo.

[QUOTE=bcp19;308550]You seem to be going a little complicated here, what about a program that was like the submission spider, set to run every X hours and just checked the size of the worktodo.txt file?[/QUOTE]

Yeah... I can get a bit obsessive about solving problems... However, I have figured out a cross-platform solution. The same Perl program works under both Linux and Windows, and it is designed such that it can be a fetching spider, a submission spider and a mfakt* slave driver (or any combination of same) all at the same time. :smile:

I want to run through a few more error-recovery tests on my own systems, but I'll soon need a couple of beta testers for each OS type. If anyone is interested in giving this a try, please PM me with a machine name (or machine names) to associate with the fetching spider -- this is needed for the "shared secret" technique.

Any number of instances can be associated with each machine, so if you (for example) run four instances of mfaktc on one machine, only one machine association is needed -- each instance will appear separately on GPU72, much like the threads/CPUs for Prime95/mprime.

Any news?

[QUOTE=chalsall;308566]Yeah... I can get a bit obsessive about solving problems... However, I have figured out a cross-platform solution. The same Perl program works under both Linux and Windows, and it is designed such that it can be a fetching spider, a submission spider and a mfakt* slave driver (or any combination of same) all at the same time. :smile:

I want to run through a few more error-recovery tests on my own systems, but I'll soon need a couple of beta testers for each OS type. If anyone is interested in giving this a try, please PM me with a machine name (or machine names) to associate with the fetching spider -- this is needed for the "shared secret" technique.

Any number of instances can be associated with each machine, so if you (for example) run four instances of mfaktc on one machine, only one machine association is needed -- each instance will appear separately on GPU72, much like the threads/CPUs for Prime95/mprime.[/QUOTE]

I stumbled over this thread just now ... sorry for being quite late.

How about two additional hooks in mfakt*: one after it read a worktodo entry (or at least tried to ...), and one after it wrote a results entry.

This could be something like two ini-variables that define something being started by mfakt* in those two cases, for example:

FETCH_COMMAND=perl spider.pl fetch w1.txt
SUBMIT_COMMAND=perl spider.pl submit res.txt

Of course, mfakt* could automatically append the names of the worktodo and results files for the current instance.

I guess this could have simplified your spider, but now that you've finished it already ...

Anyway, to add hooks like that is not difficult, and might turn out useful once the "temporary duct tape" is removed (i.e. you tell me how to fetch stuff programmatically ;-).

Anyway, I also use your submit spider (after adding proxy support), and have not (yet) tried to move it into my C(++) code. It's so much easier in perl ...

Hmm, maybe I could check out how prime95 is doing the communication and "learn" from that ..

I'm willing to port the perl code to a .NET console app.
 

I'm willing to transpose it to a .NET console app as I was already planning this integration for my mfaktXapp.

I'll use his Perl code as the blueprint for its conversion to c#

Since this domain is Chalsall's I have no objection to turning over the source code to him when I'm done. I'll maintain it until I'm fired, die or you find a better solution.

Let me know.

[QUOTE=Bdot;313399]I guess this could have simplified your spider, but now that you've finished it already ... [/QUOTE]
Finish? Where? I may be missed the announcement, and am still waiting for it. And I don't understand why is so difficult, just provide a mechanism (http link? like factorDB?) to get an assignment from the gpu72 server, and let us worry about putting that in a fork or in a graphic interface... We just need to be able to tell to the server form time to time "give me something to do" and "this is my result". When WE desire, and in a SAFE way (like my assignments should me MINE not YOURS). Excluding crawling to the assignment page by hand, with mouse clicks and typing.

[QUOTE=kladner;308312]Until Adobe puts out a 'nix version of Photoshop I'm locked into Windows. I can't afford a Mac of similar power to my PC.

Still, I have plenty of frustrations with Windows going way back.:bangheadonwall::rakes:[/QUOTE]

Is there any mac of a similar power to a (generic) PC? :razz:

LaurV, everyday frustrated by Windows and women. We can't live with them. We can't live without them...

[QUOTE=LaurV;313130]Any news?[/QUOTE]
Yes. For windows you must call the Kernel32.dll GenerateConsoleCtrlEvent


First you have to AttachConsole
Second GenerateConsoleCtrlEvent



Windows 8 seriously mucked up this interaction. (see my code comments on what broke with Windows 8.)



Exact code from MISFITs' SendCtrlCode project. For windows you could launch SendCtrlCode.exe from Perl and pass in the PID. It may not be how you wanted to solve the problem, but it works.

I hope it helps with any future projects.

[QUOTE=LaurV;313448]Is there any mac of a similar power to a (generic) PC? :razz:

LaurV, everyday frustrated by Windows and women. We can't live with them. We can't live without them...[/QUOTE]

Long after the question.....does this come close to similarity?
[url]http://www.macmall.com/p/Apple-Mac-Pro/product~dpno~9232259~pdp.hgcjafi[/url]

[QUOTE=kladner;334973]Long after the question.....does this come close to similarity?
[url]http://www.macmall.com/p/Apple-Mac-Pro/product~dpno~9232259~pdp.hgcjafi[/url][/QUOTE]

The Xeon E5645 is only $574.99 at [URL="http://www.newegg.com/Product/Product.aspx?Item=N82E16819117256"]newegg[/URL]... the whole server/computer is $3,627.99... wow looks like something is really expensive there or.... Usual high mac-like prices.

Also, it is technically *not* 12 core, it is 6 core with HT. :edit:

[QUOTE=kracker;334982]The Xeon E5645 is only $574.99 at [URL="http://www.newegg.com/Product/Product.aspx?Item=N82E16819117256"]newegg[/URL]... the whole server/computer is $3,627.99... wow looks like something is really expensive there or.... Usual high mac-like prices.

Also, it is technically *not* 12 core, it is 6 core with HT. :edit:[/QUOTE]
Look again.

[QUOTE]2.4GHz 12-Core (2x 6-Core) Intel Xeon[/QUOTE]

[QUOTE] The single-processor Mac Pro offers four slots that support up to 32GB of DDR3 ECC SDRAM, while the dual-processor Mac Pro offers eight slots that support up to 64GB.[/QUOTE]

[QUOTE][B]Up to 24 virtual cores.[/B]The Intel Xeon processors support Hyper-Threading, which allows two threads to run simultaneously on each core. So, for example, a 12-core Mac Pro presents 24 virtual cores that are recognized by OS X.[/QUOTE]


And yes. It is a Mac and costs accordingly.

:threadhijacked:

[QUOTE=kladner;335003]Look again.
.[/QUOTE]

Ah, stupid me.:davieddy:

[QUOTE=swl551;335006]:threadhijacked:[/QUOTE]
Oops. Sorry.

Been writing and running a Python script for these automatic updates for the past week, so here's a shameless plug (as already mentioned in the mfakto thread):

[URL]https://github.com/teknohog/primetools[/URL]

[QUOTE=TeknoHog;351284]Been writing and running a Python script for these automatic updates for the past week, so here's a shameless plug (as already mentioned in the mfakto thread):

[URL]https://github.com/teknohog/primetools[/URL][/QUOTE]

Nice! A man after my own heart. Python and github. I submit manually but I like the simplicity of your program. A suggestion: [url]http://stackoverflow.com/questions/3217673/why-use-argparse-rather-than-optparse[/url]
Otherwise, looks very nice.

[QUOTE=garo;351581]A suggestion: [URL]http://stackoverflow.com/questions/3217673/why-use-argparse-rather-than-optparse[/URL]
Otherwise, looks very nice.[/QUOTE]

Thanks! :cool: I'll have to look into that as a long-term Python development goal, but it's probably not a core issue here.

Of course. Just a suggestion for future projects.

It looks good and works fine.

I just wish it could get assignments from gpu72.com ([URL]https://www.gpu72.com/account/getassignments/lltf/[/URL]) instead of mersenne.org. Unfortunately I cannot use the proxy method because I need a proxy to get to the internet.

[QUOTE=Bdot;351861]It looks good and works fine.

I just wish it could get assignments from gpu72.com ([URL]https://www.gpu72.com/account/getassignments/lltf/[/URL]) instead of mersenne.org.[/QUOTE]

I'm working on this, just need to clean up a few kinks before release.

Edit: [url]https://github.com/teknohog/primetools/blob/master/mfloop72.py[/url]

[QUOTE=TeknoHog;351980]I'm working on this, just need to clean up a few kinks before release.[/QUOTE]

Coolness. Please let me know if I can be of any assistance.

[QUOTE=TeknoHog;351980]I'm working on this, just need to clean up a few kinks before release.

Edit: [URL]https://github.com/teknohog/primetools/blob/master/mfloop72.py[/URL][/QUOTE]

Scratch that, the gpu72.com functionality is now merged into mfloop.py, with a fallback to primenet.

[QUOTE=chalsall;352000]Coolness. Please let me know if I can be of any assistance.[/QUOTE]

Cheers! I think I'm fine for now :)

Teknohog merged a pull request of mine to his primetools/mfloop.py script. New functionality includes:[LIST][*]Buffering GHz-days of work instead of a simple count of assignments[*]Fetching LLTF or DCTF[*]Supporting the GPU72 "option". The default is "what makes sense" but you can choose any of the options, including letting GPU72 decide[/LIST]

Heads up to primetools/mfloop.py users: the way the script detected that the results were successfully submitted broke with the changes, but I've fixed it in my fork: [url]http://github.com/MarkRose/primetools[/url]

I've submitted a pull request to teknohog.

The HTML mfloop.py used to test for a successful login changed today, so there's another update on my fork. I've submitted another pull request to TeknoHog.

Not sure if this is the correct thread for my post, but it's at least very close.

In any event, I'm using MISFIT 2.11.0 to run a single instance of MFAKTC 0.21 on a RTX2080. Both programs work great, but I have one question for the group.

Why does MISFIT receive TF work so far ahead of the wave front when compared to the manual assignments received from: [url]https://www.mersenne.org/manual_gpu_assignment/?[/url]

As an example, the MISFIT assignments are in the M210000000 range and the manual assignments are in the M111000000 range. It would seem to me that the manual assignment range makes more sense, but maybe I'm missing something.

I have a copy of misfit running on my work box, and the numbers it pulls are all in the 100 to 105M range. All 101s atm.

[QUOTE=Aramis Wyler;559197]I have a copy of misfit running on my work box, and the numbers it pulls are all in the 100 to 105M range. All 101s atm.[/QUOTE]

Interesting, thanks for the reply. Maybe I missed a setting somewhere in misfit. I’ll look things over again.

[QUOTE=jwnutter;559202]Interesting, thanks for the reply. Maybe I missed a setting somewhere in misfit. I’ll look things over again.[/QUOTE]

Keep in mind that MISFIT can get work from both Primenet and GPU72.

As part of your debugging drill-down, look to see which is the assignment source.

[QUOTE=chalsall;559213]Keep in mind that MISFIT can get work from both Primenet and GPU72.

As part of your debugging drill-down, look to see which is the assignment source.[/QUOTE]

MISFIT Work Source = GIMPS. This may account for at least part of the difference. If so, my original question might best be directed to George and team. I'm currently TF'ing about 50 ~210M exponents a day from a bit level of ~72 to 76. I guess any reliable results are good results, but I found it odd that the thousands of exponents I've tested over the last few months were all way out in front of the current wave. Less TF work to be done later I suppose. :smile:

I'd have to know what MISFIT is asking the manual pages for. The gpu_manual_assignments page defaults to a range far below the 200 millions.

[QUOTE=Prime95;559221]I'd have to know what MISFIT is asking the manual pages for. The gpu_manual_assignments page defaults to a range far below the 200 millions.[/QUOTE]

I'm not sure if this is helpful, but here's what's shown in MISFITConfig.txt.

[CODE]LaunchURLonExport:www.mersenne.org
LaunchURL:www.mersenne.org
AutoUpdateStatsMins:10
SetExtProcessWorkingDirectoryToGivenPath:True
StartExtProcessDelay:1
GimpsStatsCacheTimeout:30
ShowConfirmationDialogs:False
ShowExportResultsInExternalFileViewer:True
EnableSchedules:True
GIMPSUserID:jwnutter
GIMPSPassword:xxxxxxxx
AutoBalanceDifferenceTrigger:0
SMTPServer:
SMTPPort:25
SMTPUserID:
SMTPPassword:xxxxxxxx
SMTPUseTLS:False
SMTPRequiresAuthentication:False
SMTPFromAddress:
SMTPRecipients:
MinimizeToSysTray:False
MinsOldBeforeProcessStallAlarm:10
StalledProcessSendEmailAlert:False
NotifyWhenGHZdaysToDoDropsBelow:1750
SendEmailWhenLowGHZdaysIsDetected:False
ShowWorkingFactorInPhases:True
AssignWorkWhenGHZdaysToDoDropsBelow:2975
MaxGhDzPerAutoAssignEvent:2275
WorkFetchExternalCommand:
WorkFetchExternalCommandArguments:
WorkFetchExternalCommandEmailResults:False
WorkFetchMISFITWorkToDoHoursBetweenChecks:1
WorkFetchWhenRemainingMISFITWorkToDoDropsBelowGhDz:4375
WorkFetchAssignmentsToFetch:16
WorkFetchBitLevelTo:76
WorkFetchReplaceIdentifierWithDate:False
WorkFetchMode:1
GPU72UserID:
GPU72Password:xxxxxxxxxx
WorkFetchGPU72WorkType:0
WorkFetchGPU72Preference:0
WorkFetchGPU72ExponentLow:0
WorkFetchGPU72ExponentHigh:100000000
WorkFetchGPU72FallBack:True
WorkFetchGPU72GHZdays:3500
EmailGlobalWorkStatsBeforeAutoUpload:False
PurgeLogsDaysOld:0
DuplicateAssignmentsDetectionIncludeBitLevel:False
ProductivityShowAverages:False
EmailSubjectExtenralFetchResults:MISFIT fetch results
EmailSubjectLowGHZDsToDo:MISFIT detected the work queue is low!
EmailSubjectStalledProcess:MISFIT detected a stalled process!
EmailSubjectStats:MISFIT statistics report
StartWithStallDetectionSuspended:False
EmailSubjectAlertPanelRemainsOpen:MISFIT ALERT PANEL is open!
SendEmailWhenAlertPanelRemainsOpen:False
EmailSubjectGIOMPanelRemainsOpen:MISFIT GIOM PANEL is open!
SendEmailWhenGIOMPanelRemainsOpen:False
HoldBackPartialResults:True
UseWorkToDoAdd:False
Lookin:C:\xxxxxx
StartExtProcessName:C:\xxxxxx
Scheduler:*,00:55,UPLOAD_RESULTS
Scheduler:*,04:55,UPLOAD_RESULTS
Scheduler:*,08:55,UPLOAD_RESULTS
Scheduler:*,12:55,UPLOAD_RESULTS
Scheduler:*,16:55,UPLOAD_RESULTS
Scheduler:*,20:55,UPLOAD_RESULTS
[/CODE]

[QUOTE=jwnutter;559226]I'm not sure if this is helpful, but here's what's shown in MISFITConfig.txt. ][/QUOTE]

I really know nothing about how MISFIT works. There are parameters in there that seem to indicate it is asking GPU72 for TF work below 100M. There probably is none available. I don't know what GPU72's fallback is in that case.

[QUOTE=Prime95;559221]I'd have to know what MISFIT is asking the manual pages for. The gpu_manual_assignments page defaults to a range far below the 200 millions.[/QUOTE]Indeed, for "TF for double-checks" it start at the first unverified exponent instead of the lowest CAT2, and for "TF for first-time LLs" it starts at the first untested exponent instead of the first CAT 2.

Jacob

[QUOTE=Prime95;559228]There are parameters in there that seem to indicate it is asking GPU72 for TF work below 100M. There probably is none available. I don't know what GPU72's fallback is in that case.[/QUOTE]

It /should/ automatically increase the "High" variable (as well as the Pledge, if it's below 74).

[QUOTE=chalsall;559253]It /should/ automatically increase the "High" variable (as well as the Pledge, if it's below 74).[/QUOTE]

I can confirm that it does this. I do not use [I]Misfit[/I] because I do not trust it. It left me in a jam a couple of years ago. However, I have the below is a [U]looping[/U] batch file:

[QUOTE]call c:\gpu72\gpu72workfetch c:\gpu72\gpu72config.txt
[/QUOTE]The rest is done by the fetch program based on the parameters in [I]gpu72config.txt[/I]. The fetched work goes into the [I]mfaktc[/I] folder in the [I]worktodo.txt[/I] file. I do a manual submission every 12 hours, more or less. Thisl works very well and I do not have to worry about it running out of work. :grin:

Thanks to all for the responses. I will PM user: swl551 to see if he has time to review/comment on this thread.

[QUOTE=jwnutter;559183]Not sure if this is the correct thread for my post, but it's at least very close.

In any event, I'm using MISFIT 2.11.0 to run a single instance of MFAKTC 0.21 on a RTX2080. Both programs work great, but I have one question for the group.

Why does MISFIT receive TF work so far ahead of the wave front when compared to the manual assignments received from: [url]https://www.mersenne.org/manual_gpu_assignment/?[/url]

As an example, the MISFIT assignments are in the M210000000 range and the manual assignments are in the M111000000 range. It would seem to me that the manual assignment range makes more sense, but maybe I'm missing something.[/QUOTE]

HI,
The answer is so you don't run out of work. You might be disconnected from the internet for an unexpected duration. If you didn't have enough queued work you would DIE.

MISFIT allows the disabling of automatic fetching allowing you to manually get freaky with assignments.

[QUOTE=swl551;560177]HI,
The answer is so you don't run out of work. You might be disconnected from the internet for an unexpected duration. If you didn't have enough queued work you would DIE.

MISFIT allows the disabling of automatic fetching allowing you to manually get freaky with assignments.[/QUOTE]

But the question wasn't why so much work, but why this work, that is so ahead of where it's needed the most.

[QUOTE=Viliam Furik;560201]But the question wasn't why so much work, but why this work, that is so ahead of where it's needed the most.[/QUOTE]

Why is it getting TF work in that range... Let's start with a screenshot of your fetching configuration...

[QUOTE=swl551;560255]Why is it getting TF work in that range... Let's start with a screenshot of your fetching configuration...[/QUOTE]

Scott... The reason is the user has Primenet set as their source.

This isn't so much a problem with your code, but rather, what Primenet is giving based on what it's being asked for by your software.

[QUOTE=swl551;560255]Why is it getting TF work in that range... Let's start with a screenshot of your fetching configuration...[/QUOTE]

Is MISFIT using the manual assignments web page, the GPU assignments web page, or a custom interface written just for MISFIT.

I think the server is thinking MISFIT is a prime95 client where TF limits are much lower. The work distribution map is showing first available prime95 TF exponent is over 200 million. GPUs need to TF 3 or 4 bit levels deeper than prime95.

[QUOTE=Prime95;560259]Is MISFIT using the manual assignments web page, the GPU assignments web page, or a custom interface written just for MISFIT.[/QUOTE]

An important question. Scott, can you tell us all what URL MISFIT "hits" for GIMPS assignments?

All I can say with certainty is GPU72 never held, and thus didn't assign, what this user was given.

Read: not my problem. But happy to help debug.

[QUOTE=chalsall;560261]An important question. Scott, can you tell us all what URL MISFIT "hits" for GIMPS assignments?

All I can say with certainty is GPU72 never held, and thus didn't assign, what this user was given.

Read: not my problem. But happy to help debug.[/QUOTE]

"source code says"

[CODE] public string GetWorkGIMPS(string userID, string password,int cores, int assignmentsPerCore, GIMPSPreferredWorkType workType, int? expStart, int? expEnd)
{
string url = "https://www.mersenne.org/<redacted>?uid=" + userID +
//"&user_password=" + password +
"&cores=" + cores +
"&num_to_get=" + assignmentsPerCore +
"&pref=" + (int)workType +
"&exp_lo=" + expStart +
"&exp_hi=" + expEnd +
"&B1=Get+Assignments";

Debug.WriteLine(url);

return PerformGetWithNOCookies(url);
}[/CODE]

So MISFIT has its own secret web page.

Looking at that web page, it is applying prime95's rules for TF upper bounds. I'll see what I can come up with.

[QUOTE=Prime95;560471]So MISFIT has its own secret web page.

Looking at that web page, it is applying prime95's rules for TF upper bounds. I'll see what I can come up with.[/QUOTE]

I've attempted a fix. Let me know if it works or not.

Sorry folks, I got real busy and forgot to check back in on this thread.

Yes, the MISFIT work source is set to GIMPS on this specific client.

I can also report that on 10/25/2020 this client started turning in TF results for exponents in the 115M range (vs the previously mentioned 215M range). Based on my specific MISFIT assignment settings, I believe MISFIT is caching ~2-3 days of TF work FYI.

Thanks a lot for the help. This change alone has increased the TF throughput on this client by about 4%.

I recently upgraded a bunch of systems to new distributions and Python 2 is obviously no longer an option. Tonight I got around to updating mfloop.py to use Python 3.

I also updated mfloop.py to use the [URL="https://www.mersenne.org/manual_gpu_assignment/"]manual gpu assignment page[/URL] instead of the [URL="https://www.mersenne.org/manual_assignment/"]manual assignment page[/URL] when falling back to PrimeNet when not fetching anything from GPU72.

You can fetch it from [URL="https://github.com/MarkRose/primetools"]https://github.com/MarkRose/primetools[/URL].

I also see that teknohog has recently updated the primary repository as well to work with Python 3 and made some other changes: [URL="https://github.com/teknohog/primetools"]https://github.com/teknohog/primetools[/URL]. I may rebase off him soon.