What happens if work is reported "out of order"?

Example: Say you are trial factoring from 71 to 74, and when the spider tries to communicate that 71-72 is done, it fails, and then a while later it reports, successfully, that 72-73 and 73-74 is done. Later you want to report the 71-72 part. What happens?

[QUOTE=Xyzzy;365936]Example: Say you are trial factoring from 71 to 74, and when the spider tries to communicate that 71-72 is done, it fails, and then a while later it reports, successfully, that 72-73 and 73-74 is done. Later you want to report the 71-72 part. What happens?[/QUOTE]

Excellent point!

Although I have never encountered this myself, in such a case additional human interaction is required, since a "not required" message from Primenet is not considered a communications error and so the results.txt file is not moved to "not_completely_submitted_${now}.txt".

The commands:

[code]cat not_completely_submitted_* > temp.txt
grep [EXPONENT_IN_QUESTION] sent/*.txt >> temp.txt
sort temp.txt >> results.txt
rm not_completely_submitted_* temp.txt[/CODE]

Should work.

We tested the spider and it works well.

We will remove the factor found lines and deal (somehow) with those manually because PrimeNet shorted us, which we suppose is like playing the lottery. (See image attached.)

We have had great trouble in the past submitting our "mega-dumps" but we are lazy and that seemed like the only way to submit work without error. We will now use the spider, perhaps once a day.

Question: What happens if mfaktc writes to results.txt during the time the spider is working? Do we need to ensure that mfaktc is in the middle of a long run to be safe?

PS - We had to manually run "sudo apt-get install libwww-perl" to get the spider to work. This was not an easily figured out solution on our end because there are billions (?) of Perl packages. One would think a package like that would be installed by default.

Thanks for the script!

:tu:

[QUOTE=Xyzzy;365943]We will remove the factor found lines and deal (somehow) with those manually because PrimeNet shorted us, which we suppose is like playing the lottery. (See image attached.)[/QUOTE]

Yes; this is a well documented "bug" with Primenet. George and James are aware of it, and hopefully it can be fixed sometime soon... (?)

[QUOTE=Xyzzy;365943]We will now use the spider, perhaps once a day.[/QUOTE]

I, personally, use it twice an hour via a crontab entry:[CODE]
9,39 * * * * ~/prime/gpu/tf/submit_spider ~/prime/gpu/tf/ >/dev/null[/CODE]

[QUOTE=Xyzzy;365943]Question: What happens if mfaktc writes to results.txt during the time the spider is working? Do we need to ensure that mfaktc is in the middle of a long run to be safe?[/QUOTE]

Nope, you don't need to worry about this. The spider is intended to be (mostly) fully automatic. The code in the script:[CODE]# Move the results file so it won't get clobbered.
rename("results.txt","submitting_${Now}.txt");

# Sleep for two seconds to ensure any other users of results.txt finish up.
sleep(2);[/CODE]

... manages the situation where mfakt[c|o] and the spider happen to be working on the file at the same instant. It's known as a "reader - writer" relationship; only one program writes, only one reads. (To get "geeky", this leverages on the fact that a file can be renamed by one program, but if another program already has an open "file-handle" for said file any changes to it are preserved.)

[QUOTE=Xyzzy;365943]PS - We had to manually run "sudo apt-get install libwww-perl" to get the spider to work. This was not an easily figured out solution on our end because there are billions (?) of Perl packages. One would think a package like that would be installed by default.[/QUOTE]

Interesting observation. What distro are you using? I'll make a note on the spider's documentation page.

[QUOTE]I, personally, use it twice an hour via a crontab entry:[/QUOTE]How do you deal with factors found if it is automated? (Should the script be modified to pull out factor found lines and put them into a different file? Or maybe rewrite them back into results.txt?)

[QUOTE]Interesting observation. What distro are you using? I'll make a note on the spider's documentation page.[/QUOTE]Ubuntu 12.04 (LTS)

[QUOTE=Xyzzy;365947]How do you deal with factors found if it is automated? (Should the script be modified to pull out factor found lines and put them into a different file? Or maybe rewrite them back into results.txt?)[/QUOTE]

I, personally, don't particularly care about "proper" credits. And I would argue that George / Scott (and now James) should fix a bug which has been communally known for several years now, rather than having other programs and humans do additional work. Trivial to fix on Primenet (one regex test).

[QUOTE=Xyzzy;365947]Ubuntu 12.04 (LTS)[/QUOTE]

Thanks for that information.

The only reason we concern ourself with the factor thingie is because you have a report that shows the expected number of factors for the amount of work we have done.

[url]http://www.gpu72.com/reports/worker/7e6a2e592a37a719fac4f765eb0f6ca8/[/url]

Since it is trivial to spoof results to gain false credit, we like to think that the expected factors found metric would allow others to see that our work is legit.

:confused:

[QUOTE=Xyzzy;365953]Since it is trivial to spoof results to gain false credit, we like to think that the expected factors found metric would allow others to see that our work is legit.[/QUOTE]

GPU72 calculates this metric based on its own assignments. It doesn't matter if Primenet erroneously records the factor-found result as being found via P-1.

And, there will often be a large deviation of "Expected" vs "Found". Some are "lucky", some are "unlucky". If it's more than 10% out we get suspicious (and I sometimes waste my own GPU cycles double checking TFing results -- I have never found a cheater), but your particular results do not appear to be non-nominal.

Well, the spider is now running automatically, and it works.

:bow:

Now we have to find something else to "fix".

:max:

[QUOTE=chalsall;365946]Yes; this is a well documented "bug" with Primenet. George and James are aware of it, and hopefully it can be fixed sometime soon... (?)[/QUOTE]I have been (still am) on vacation for the last long while. I'll be back in the frozen land I call home in about a week though, and I'll try and keep Primenet results submission fixing near the top of the huge pile of catch-up I have to sort through then.

New primenet pages vs. submit_spider
 

It appears submit_spider does not do well with the new primenet mechanism. Until I have time to fix the script I probably won't submit my results ...
at least not regularly - I hope there is still some page to manually submit things

[QUOTE=Bdot;366739]It appears submit_spider does not do well with the new primenet mechanism. Until I have time to fix the script I probably won't submit my results ...
at least not regularly - I hope there is still some page to manually submit things[/QUOTE]

Hmmmm... I'll take a look at this tomorrow afternoon. Have a deliverable for tomorrow morning.

But, yes, the "Manual Submission" page still exists and works.

[QUOTE=chalsall;366745]Hmmmm... I'll take a look at this tomorrow afternoon. Have a deliverable for tomorrow morning.[/QUOTE]

Thanks to Bdot (who got to the problem before I did), the [URL="https://www.gpu72.com/software/submit_spider"]Submission Spider[/URL] for Linux has been updated to handle the POST log-in function of Primenet.

Those who still use this script please upgrade, and automatic submission of results will once again work.

:bow:

Please see [URL="http://www.mersenneforum.org/showpost.php?p=374601&postcount=203"]this post[/URL] regarding a problem I've been having recently. It's a combination of submit_spider not catching an error (0 GHz credit assigned by server) and a server-side problem.

[QUOTE=brilong;374602]Please see [URL="http://www.mersenneforum.org/showpost.php?p=374601&postcount=203"]this post[/URL] regarding a problem I've been having recently. It's a combination of submit_spider not catching an error (0 GHz credit assigned by server) and a server-side problem.[/QUOTE]

Once again I'm having trouble with submit_spider and this time it's not a server timeout issue. I was able to submit the results manually.

Please see [URL]http://www.mersenneforum.org/showpost.php?p=383419&postcount=754]this post[/URL] for more details.

Hey Brian et al.

OK, sorry for the delay on this; extremely busy on "Real Work" at the moment...

Two things: first, there was the original bug which resulted in a bad result from Primenet not being correctly detected; this has now been fixed.

Secondly, sometime in the last week or so the Primenet admins changed the submission form from being a "GET" (or "PUT") to being a "POST" form. This broke the submission spider even further.

I have hopefully rectified both issues. Please give version 0.24 of [URL="https://www.gpu72.com/software/submit_spider"]Submission Spider[/URL] a try, and let me know if you see any additional issues.

I'm running it currently on a couple of EC2 instances, but would appreciate additional testing/feedback.

[QUOTE=brilong;383420]Once again I'm having trouble with submit_spider and this time it's not a server timeout issue. I was able to submit the results manually.[/QUOTE]