Automatic Submission Spider for Workers...

chalsall · 2011-12-13, 18:51

Based on a question by Dubslow about how to do HTTP requests using Bash (the context being automating the submission of results until Christenson completes his work.

I don't know how to do script HTTP in Bash, but I do know how to in Perl... So I've put together a quick script which will hopefully be useful to some of you.

Please see: Submission Spider.

Note that this is currently targeted to Unix users, but it should be possible to adapt it to Windows without too much effort. Advise on how to do that would be appreciated (I don't do Windows).

As always, feedback welcome.

nucleon · 2011-12-13, 21:00

For those wanting to try on windows - cygwin tools is your best bet.

I've been using a bash script and lynx (command line browser) to auto submit answers and get more work.

-- Craig

Dubslow · 2011-12-13, 22:12

Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)

chalsall · 2011-12-13, 22:23

Quote:

Originally Posted by Dubslow

Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)

OK, but I would strongly recommend you learn Perl if you want to get into spidering -- it's one of Perl's sweet spots.

You can spider using Bash, gluing together other programs like lynx, wget, sed, awk, et al. But I would argue you can do everything you need just with Perl (and a few modules), and have much more control and better error handling with less overall work. And half of the job of spidering is dealing with errors...

But as always with software, there's more than one way to do things....

Dubslow · 2011-12-13, 22:38

Learning one thing at a time. Programming will come after programming class. Programming in general is not something that interests me, just something that is useful, and I'm not a very good self motivator, and a CS course is mandatory, which should help jumpstart things.

Dubslow · 2011-12-14, 18:03

I was also going to try and use the PrimeNet API.

frmky · 2011-12-15, 09:00

A minor bug correction to the spider...
unlink("submitted_${Now}.txt");
should be
unlink("submitting_${Now}.txt");

oswald · 2011-12-31, 23:06

Thanks for recommending cygwin. I was trying to use the unix subsystem that microsoft gives away and it is very limited. cygwin is better for me.

So now I use a bash script to hit the directories and call the submission spider. I use the windows scheduler to call the bash script. Maybe a strange combo, but it only took a 10 minutes to set up.

Anyway, submit_spider is great and I just feed the wordtodo.txt and relax.

Life is good.

flashjh · 2012-01-01, 00:18

I installed ActivePerl 5.14.2 Build 1402 (64-bit) for Windows, wrote a couple of batch files to combine all my results.txt files and use Windows scheduler to run everthing. I combine results before submission so I have one 'submitted' file that I archive each time -- makes everything neater and easier to double check once in a while.

The submission spider is awesome! Thanks.

If anyone needs Windows advice, PM me.

Jerry

oswald · 2012-01-08, 04:25

I just ran into a hiccup.
So far it has worked on all submissions I think, but failed on

Code:

no factor for M45833773 from 2^74 to 2^75 [mfaktc 0.18 barrett79_mul32]

Submitting to http://www.mersenne.org/manual_result/ worked ok.
I've tried again expecting a not needed result, but received the following. It is the same error as before.

I'll reboot and try again just to be sure.

Code:

20120108_032034 INFO:  Submission spider starting...
20120108_032034 INFO:  Attempting to log into PrimeNet.  This can take a little while...
20120108_032058 INFO:  Successfully logged into PrimeNet.
Use of uninitialized value $GHzDays in concatenation (.) or string at /usr/local/bin/submit_spider line 151, <IN> line 1.
20120108_032121 INFO:  M49831559 submitted;  GHz Days credit.
20120108_032122 INFO:  Submission complete.  Moving results file to "submitted_20120108_032058.txt"
20120108_032122 INFO:  Spider has finished.  Exiting.

oswald · 2012-01-08, 04:40

I'm getting timeouts now. Hold off looking into this until I can verify this.
Thanks,
Roy

2011-12-13, 18:51	#1
chalsall If I May "Chris Halsall" Sep 2002 Barbados 9730₁₀ Posts	Automatic Submission Spider for Workers... Based on a question by Dubslow about how to do HTTP requests using Bash (the context being automating the submission of results until Christenson completes his work. I don't know how to do script HTTP in Bash, but I do know how to in Perl... So I've put together a quick script which will hopefully be useful to some of you. Please see: Submission Spider. Note that this is currently targeted to Unix users, but it should be possible to adapt it to Windows without too much effort. Advise on how to do that would be appreciated (I don't do Windows). As always, feedback welcome. Last fiddled with by chalsall on 2014-03-29 at 13:36

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
GPU to 72 spider poaching exponents?	ixfd64	GPU to 72	2	2014-07-27 20:38
It Seems The Great Spider ...	Dubslow	Forum Feedback	13	2012-06-01 13:39
Automatic temperature control	TObject	Software	13	2012-04-27 19:33
Went from 8 workers to 4 workers on v26.6 upgrade	dmoran	Software	13	2011-05-23 12:36
Manual submission of automatic assignment result	tichy	PrimeNet	4	2010-12-17 09:57

2011-12-13, 21:00	#2
nucleon Mar 2003 Melbourne 5·103 Posts	For those wanting to try on windows - cygwin tools is your best bet. I've been using a bash script and lynx (command line browser) to auto submit answers and get more work. -- Craig

2011-12-13, 22:12	#3
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 16065₈ Posts	Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)

2011-12-13, 22:38	#5
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 3×29×83 Posts	Learning one thing at a time. Programming will come after programming class. Programming in general is not something that interests me, just something that is useful, and I'm not a very good self motivator, and a CS course is mandatory, which should help jumpstart things.

2011-12-14, 18:03	#6
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 1110000110101₂ Posts	I was also going to try and use the PrimeNet API.

2011-12-15, 09:00	#7
frmky Jul 2003 So Cal 2×3⁴×13 Posts	A minor bug correction to the spider... unlink("submitted_${Now}.txt"); should be unlink("submitting_${Now}.txt");

2011-12-31, 23:06	#8
oswald Apr 2011 in vivo 75₁₀ Posts	Thanks for recommending cygwin. I was trying to use the unix subsystem that microsoft gives away and it is very limited. cygwin is better for me. So now I use a bash script to hit the directories and call the submission spider. I use the windows scheduler to call the bash script. Maybe a strange combo, but it only took a 10 minutes to set up. Anyway, submit_spider is great and I just feed the wordtodo.txt and relax. Life is good.

2012-01-01, 00:18	#9
flashjh "Jerry" Nov 2011 Vancouver, WA 1,123 Posts	I installed ActivePerl 5.14.2 Build 1402 (64-bit) for Windows, wrote a couple of batch files to combine all my results.txt files and use Windows scheduler to run everthing. I combine results before submission so I have one 'submitted' file that I archive each time -- makes everything neater and easier to double check once in a while. The submission spider is awesome! Thanks. If anyone needs Windows advice, PM me. Jerry

2012-01-08, 04:40	#11
oswald Apr 2011 in vivo 1001011₂ Posts	I'm getting timeouts now. Hold off looking into this until I can verify this. Thanks, Roy