Automatic Submission Spider for Workers...

[chalsall]

Originally Posted by flashjh

I've had the same problem, but I have my system setup to rename the 'not submitted' file back to results.txt so it just sends it again later. Since the server started having problems I noticed the submission spider fails around once a day - it is still working thought with the rename batch file.

Yeah... This is the bug that oswald brought to my attention above. The spider doesn't properly handle the situation where it successfully logs into PrimeNet, but then doesn't get a sane response when actually submitting results.

Your (flashjh's) solution is relatively reasonable, but it still has a race condition (moving the file back could overwrite new results written by mfakt*). A better solution would be to have the spider place the file somewhere where it knows about it, and will deal with it the next time it runs.

Bringing this spider up to 0.3 has been on my todo list for quite some time; I really must bump up its priority....

[flashjh]

Originally Posted by chalsall

Your (flashjh's) solution is relatively reasonable, but it still has a race condition (moving the file back could overwrite new results written by mfakt*). A better solution would be to have the spider place the file somewhere where it knows about it, and will deal with it the next time it runs.

That's a good idea; I'll modify the setup to incorporate the not_submitted file(s) at submission time. I know the race condition is legitimate, but no matter how the system is setup there is always a chance a result could get lost during submission. My main system collects all the results files and then submits from there because I find it easier to have one system submit and archive the results. Either way, thanks for working on this. Someday all of this will be part of P95 or a separate GUI and we won't have to worry about not_submitted files :)

[bcp19]

Originally Posted by chalsall

Yeah... This is the bug that oswald brought to my attention above. The spider doesn't properly handle the situation where it successfully logs into PrimeNet, but then doesn't get a sane response when actually submitting results.

Your (flashjh's) solution is relatively reasonable, but it still has a race condition (moving the file back could overwrite new results written by mfakt*). A better solution would be to have the spider place the file somewhere where it knows about it, and will deal with it the next time it runs.

Bringing this spider up to 0.3 has been on my todo list for quite some time; I really must bump up its priority....

Renaming the file is a good idea (considering I have been getting 5-6 not submitted files a day lately) and would not be a problem as the spider is in a separate folder from the 3 mfaktc folders, so there is no chance a new result could be lost unless the spider hung and took over an hour to fail and the next task started to bring in the new results (which I think is impossible, since it cannot restart a task that has not finished?).

[flashjh]

Originally Posted by bcp19

Renaming the file is a good idea (considering I have been getting 5-6 not submitted files a day lately) and would not be a problem as the spider is in a separate folder from the 3 mfaktc folders, so there is no chance a new result could be lost unless the spider hung and took over an hour to fail and the next task started to bring in the new results (which I think is impossible, since it cannot restart a task that has not finished?).

You can also have the task terminate if it runs for XX amount of time.

[chalsall]

Originally Posted by bcp19

(which I think is impossible, since it cannot restart a task that has not finished?).

Now you start to get into the tricky bit about cooperating tasks...

Depending on the environment, it is certainly possible for a new process to be started running the same code which you might not expect to be running concurrently.

It's a very difficult problem, which manifests surprisingly often. This probably explains why "Hello, IT. Have you tried turning it off and on again?" (quoting Roy Trenneman) is so commonly heard....

[bcp19]

Originally Posted by chalsall

Now you start to get into the tricky bit about cooperating tasks...

Depending on the environment, it is certainly possible for a new process to be started running the same code which you might not expect to be running concurrently.

It's a very difficult problem, which manifests surprisingly often. This probably explains why "Hello, IT. Have you tried turning it off and on again?" (quoting Roy Trenneman) is so commonly heard....

I set up a task in windows to repeat hourly, it runs a batch file, then submit.pl, then another batch file. If the submit.pl hung for over an hour (which I have never seen happen, but it surely is possible), then the scheduled task never completed, so I don't see windows starting another of the same task without the prior one being complete.

[chalsall]

Originally Posted by bcp19

I set up a task in windows to repeat hourly, it runs a batch file, then submit.pl, then another batch file. If the submit.pl hung for over an hour (which I have never seen happen, but it surely is possible), then the scheduled task never completed, so I don't see windows starting another of the same task without the prior one being complete.

Do you know that for a fact, or are you guessing? Have you tested this?

Under *NIX, if you list a program in the CRONTAB it will do it (again) when you've told it to do it.

Damn computers!!! They do exactly what you've told them to do....

[bcp19]

Originally Posted by chalsall

Do you know that for a fact, or are you guessing? Have you tested this?

Under *NIX, if you list a program in the CRONTAB it will do it (again) when you've told it to do it.

Damn computers!!! They do exactly what you've told them to do....

It is set up "If the task is already running, then: Do not start a new instance."

[chalsall]

Originally Posted by bcp19

It is set up "If the task is already running, then: Do not start a new instance."

Please forgive me. I don't do Windows.

But what does that actually mean? What if the task is "zombied"? Is it still "running"?

Or, let's say, you have a run-away task. Happens often. Should the operating system let it do it's thing?

What if the task was "killed" before it had a chance to clean up after itself (read: left files it owned in an undefined state). It would therefore not be running, but may have left garbage behind.

I say again... Can be difficult....

[bcp19]

Originally Posted by chalsall

Please forgive me. I don't do Windows.

But what does that actually mean? What if the task is "zombied"? Is it still "running"?

Or, let's say, you have a run-away task. Happens often. Should the operating system let it do it's thing?

What if the task was "killed" before it had a chance to clean up after itself (read: left files it owned in an undefined state). It would therefore not be running, but may have left garbage behind.

I say again... Can be difficult....

I have no real idea, though it is set to be killed if it has been running for longer than 2 hours. Whether this means 'zombied'? Dunno. Never seen it hang, so can it? Dunno. Will worry about it if it happens.

[Dubslow]

So... the reason I did my append hack was so that I could submit results to Mersenne-aries after PrimeNet, but it just now occurred to me, why not automate that too? The form:

<form method="post" enctype="multipart/form-data" action="/index.php"> [snip] 
<br>User Name: <input type="text" size="15" name="anonymous_username" value="Dubslow"> 
and/or computer ID: <input type="text" size="15" name="anonymous_compname" value="GTX 460"> 
<textarea name="results" style="width: 800px; height: 400px;" wrap="off"></textarea><br>
or, upload new results as a file (max 96MB):<br><input type="file" name="results_file" />
<br><input class="submit" type="submit" value="Submit"><br>
</form>

I'm not sure how easy it is to POST a file in Perl (not very easy in Python I don't think), but pasting the lines into a very long string via "results" should be doable if the former isn't possible (perhaps that's rather inefficient, but hey).