Automatic Submission Spider for Workers...
 

Based on a question by Dubslow about how to do HTTP requests using Bash (the context being automating the submission of results until Christenson completes his work.

I don't know how to do script HTTP in Bash, but I do know how to in Perl... So I've put together a quick script which will hopefully be useful to some of you.

Please see: [URL="http://www.gpu72.com/spider/"]Submission Spider[/URL].

Note that this is currently targeted to Unix users, but it should be possible to adapt it to Windows without too much effort. Advise on how to do that would be appreciated (I don't do Windows).

As always, feedback welcome.

For those wanting to try on windows - cygwin tools is your best bet.

I've been using a bash script and lynx (command line browser) to auto submit answers and get more work.

-- Craig

Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)

[QUOTE=Dubslow;282093]Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)[/QUOTE]

OK, but I would strongly recommend you learn Perl if you want to get into spidering -- it's one of Perl's sweet spots.

You [I]can[/I] spider using Bash, gluing together other programs like lynx, wget, sed, awk, et al. But I would argue you can do everything you need just with Perl (and a few modules), and have much more control and better error handling with less overall work. And half of the job of spidering is dealing with errors...

But as always with software, there's more than one way to do things.... :smile:

Learning one thing at a time. Programming will come after programming class. Programming in general is not something that interests me, just something that is useful, and I'm not a very good self motivator, and a CS course is mandatory, which should help jumpstart things.

I was also going to try and use the PrimeNet API.

A minor bug correction to the spider...
unlink("submitted_${Now}.txt");
should be
unlink("submitting_${Now}.txt");

Thanks for recommending cygwin. I was trying to use the unix subsystem that microsoft gives away and it is very limited. cygwin is better for me.

So now I use a bash script to hit the directories and call the submission spider. I use the windows scheduler to call the bash script. Maybe a strange combo, but it only took a 10 minutes to set up.

Anyway, [URL="http://gpu.mersenne.info/software/submit_spider"]submit_spider[/URL] is great and I just feed the wordtodo.txt and relax.

Life is good.

I installed ActivePerl 5.14.2 Build 1402 (64-bit) for Windows, wrote a couple of batch files to combine all my results.txt files and use Windows scheduler to run everthing. I combine results before submission so I have one 'submitted' file that I archive each time -- makes everything neater and easier to double check once in a while.

The submission spider is awesome! Thanks.

If anyone needs Windows advice, PM me.

Jerry

I just ran into a hiccup.
So far it has worked on all submissions I think, but failed on
[CODE]no factor for M45833773 from 2^74 to 2^75 [mfaktc 0.18 barrett79_mul32][/CODE]Submitting to [URL]http://www.mersenne.org/manual_result/[/URL] worked ok.
I've tried again expecting a not needed result, but received the following. It is the same error as before.

I'll reboot and try again just to be sure.

[CODE]20120108_032034 INFO: Submission spider starting...
20120108_032034 INFO: Attempting to log into PrimeNet. This can take a little while...
20120108_032058 INFO: Successfully logged into PrimeNet.
Use of uninitialized value $GHzDays in concatenation (.) or string at /usr/local/bin/submit_spider line 151, <IN> line 1.
20120108_032121 INFO: M49831559 submitted; GHz Days credit.
20120108_032122 INFO: Submission complete. Moving results file to "submitted_20120108_032058.txt"
20120108_032122 INFO: Spider has finished. Exiting.[/CODE]

I'm getting timeouts now. Hold off looking into this until I can verify this.
Thanks,
Roy