A couple of questions about polynomial selection - Page 3

schickel · 2010-11-21, 21:41

Quote:

Originally Posted by aifbowman

Ok, I have tested out my idea of stopping the 10 clients and then restarting with an 11th, specifying the maximum number of clients as 11, but the 11th just starts sieving right from the beginning again. So it is looking like the only way I will be able to use more machines is if I specify different ranges for them. Can I do this through the python script? Also how would I merge the relations this second cluster would find with those from the first cluster? Is it simply a case as copying and pasting the contents of one dat file into the other?

Yes, when you restart a client it starts from its resume file.

To add another client you have to figure out where it should be and either create a resume file for it to start with, or start it, let it create the resume file, stop it, update the resume file, then restart it. The only thing I don't know is where in the cycle the script writes the resume file. The perl script updates the resume file *after* each sieve block, so the first time you start a client, it doesn't have a resume file.

As far as adding the new relations in, if you start the extra machine(s) as "1/2/3 of x", the #1 machine will be collecting the new relations as they go. Just take the file produced by the #1 machine and rename it to a client file from the original set and then move it to where the relations are being dumped for the first batch of machines. Next time the original #1 machine collects relations, presto, it finds the new ones... If the new machines are isloated from each other, just take the relations from all the machines and just rename the file from the #1 machine; the others will look like the *.add files already.

Just be extra careful that you don't overwrite the master relation file on the original master machine.....that would hurt!

aifbowman · 2010-11-21, 23:48

thanks schickel, that's exactly what I needed. You don't happen to know what I need to edit in the python script to specify the range for the second cluster? Or will I need to run GNFS directly with some suffix to the command to specify that?

Also is there a maximum range for sieving i.e. is it possible for me to make the second cluster search a range that is too high to find any relations, because obviously I want to avoid there being any crossover in their two ranges,

Anthony

aifbowman · 2010-11-22, 02:33

Actually, I've worked out the answer to the above question, I just need to add a q0 field to the poly file.

My only remaining query is whether there is a limit to the number of machines the factmsieve python script will allow/work with? I don't want to be too ambitious and and end up wasting precious time by trying too many...

schickel · 2010-11-22, 03:01

Quote:

Originally Posted by aifbowman

Actually, I've worked out the answer to the above question, I just need to add a q0 field to the poly file.

My only remaining query is whether there is a limit to the number of machines the factmsieve python script will allow/work with? I don't want to be too ambitious and and end up wasting precious time by trying too many...

Well, I don't think I'd want to try more than <maxint> machines at one time.....

smh · 2010-11-22, 08:26

Quote:

Originally Posted by aifbowman

Actually, I've worked out the answer to the above question, I just need to add a q0 field to the poly file.

My only remaining query is whether there is a limit to the number of machines the factmsieve python script will allow/work with? I don't want to be too ambitious and and end up wasting precious time by trying too many...

I can't check right now, but i think you actually need te edit the .job file.

fivemack · 2010-11-22, 15:30

Quote:

Originally Posted by aifbowman

Tom- no actually it is not, I don't think he lectures for any first year courses (at least not the ones I have started so far), why do you ask?

Oh, I was expecting this to be a problem from a third-year number theory course, and would have expected Kevin to be teaching that ... I did a number-theory PhD (at Nottingham) about five years ago and remember Kevin as a nice guy that I met at inter-university seminars.

warut · 2010-11-22, 22:31

I think aifbowman's teacher is Daniel R. Moore and the 160-digit number is

Code:

7823445407949925252516952959603663363759565065213243539144686925012567049413798634116058539700325978831844466909241650643702088625525642002090404962143815108959

aifbowman · 2010-11-25, 19:18

fivemack- ah ok, well I know that professor buzzard is taking part in our current first year project, cracking student's public keys if they make them too weak. a few people have already fallen victim to him haha...

warut- yes you are right... are you a member of my class? :p

Ok i've finished sieving, reached 107% of estimated min of relations, but I am slightly worried about this output:

Code:

commencing linear algebra
Thu Nov 25 11:28:00 2010  read 4441680 cycles
Thu Nov 25 11:28:06 2010  cycles contain 11039094 unique relations
Thu Nov 25 11:37:08 2010  read 11039094 relations
Thu Nov 25 11:37:24 2010  using 20 quadratic characters above 536870298
Thu Nov 25 11:38:10 2010  building initial matrix
Thu Nov 25 11:41:03 2010  memory use: 1411.9 MB
Thu Nov 25 11:41:52 2010  read 4441680 cycles
Thu Nov 25 11:41:54 2010  matrix is 4441503 x 4441680 (1273.5 MB) with weight 416806460 (93.84/col)
Thu Nov 25 11:41:54 2010  sparse part has weight 298294192 (67.16/col)
Thu Nov 25 11:42:32 2010  filtering completed in 2 passes
Thu Nov 25 11:42:34 2010  matrix is 4441039 x 4441216 (1273.4 MB) with weight 416789819 (93.85/col)
Thu Nov 25 11:42:34 2010  sparse part has weight 298289378 (67.16/col)
Thu Nov 25 11:46:45 2010  matrix starts at (0, 0)
Thu Nov 25 11:46:47 2010  matrix is 4441039 x 4441216 (1273.4 MB) with weight 416789819 (93.85/col)
Thu Nov 25 11:46:47 2010  sparse part has weight 298289378 (67.16/col)
Thu Nov 25 11:46:47 2010  saving the first 48 matrix rows for later
Thu Nov 25 11:46:48 2010  matrix includes 64 packed rows
Thu Nov 25 11:46:49 2010  matrix is 4440991 x 4441216 (1219.3 MB) with weight 331720561 (74.69/col)
Thu Nov 25 11:46:49 2010  sparse part has weight 292992292 (65.97/col)
Thu Nov 25 11:46:49 2010  using block size 65536 for processor cache size 8192 kB
Thu Nov 25 11:47:10 2010  commencing Lanczos iteration (4 threads)
Thu Nov 25 11:47:10 2010  memory use: 1131.2 MB
Thu Nov 25 11:47:40 2010  linear algebra at 0.0%, ETA 23h34m
Thu Nov 25 11:47:50 2010  checkpointing every 190000 dimensions

What worries me is 1. the matrix is much smaller then the one for the c160 you were talking about earlier fivemack, and 2. the ETA is much shorter as well... any ideas if there is a problem or am I just worrying over nothing?

axn · 2010-11-25, 20:53

Quote:

Originally Posted by aifbowman

What worries me is 1. the matrix is much smaller then the one for the c160 you were talking about earlier fivemack, and 2. the ETA is much shorter as well... any ideas if there is a problem or am I just worrying over nothing?

Looks ok. Oversieving usually results in a smaller matrix / shorter ETA.

aifbowman · 2010-11-25, 21:09

exactly what I wanted to hear :)

fivemack · 2010-11-26, 00:14

Nothing to worry about in that output; I think you must have oversieved a lot, whilst my example C160 was sieved only until I was able to get a matrix (since I had a dozen cores available against your 40 or more). And the i7 is very, very fast at doing matrices; I have a 4.5M matrix done in 33 hours on an i7 without overclocking and with slow memory, so if you've boosted your machine at all then doing the matrix in 24 hours is plausible.

When you've got the result, could you also post

number of relations // number of unique relations
and the lpbr/lpba/mfbr/mfba/alim/rlim/alambda/rlambda lines from your polynomial file

and, if you've got the time and disc space, truncate msieve.dat say 5M relations at a time, running until you get the matrix generated, so that you can get yourself an idea of how the matrix size varies as a function of oversieving.

2010-11-22, 22:31	#29
warut Dec 2009 89 Posts	I think aifbowman's teacher is Daniel R. Moore and the 160-digit number is Code: 7823445407949925252516952959603663363759565065213243539144686925012567049413798634116058539700325978831844466909241650643702088625525642002090404962143815108959

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
A couple newbie questions	evanmiyakawa	Information & Answers	4	2017-11-07 01:37
new here with a couple questions	theshark	Information & Answers	21	2014-08-30 17:36
2^877-1 polynomial selection	fivemack	Factoring	47	2009-06-16 00:24
Polynomial selection	CRGreathouse	Factoring	2	2009-05-25 07:55
A couple questions from a new guy	Optics	Information & Answers	8	2009-04-25 18:23

2010-11-21, 23:48	#24
aifbowman Nov 2010 2×7 Posts	thanks schickel, that's exactly what I needed. You don't happen to know what I need to edit in the python script to specify the range for the second cluster? Or will I need to run GNFS directly with some suffix to the command to specify that? Also is there a maximum range for sieving i.e. is it possible for me to make the second cluster search a range that is too high to find any relations, because obviously I want to avoid there being any crossover in their two ranges, Anthony

2010-11-22, 02:33	#25
aifbowman Nov 2010 E₁₆ Posts	Actually, I've worked out the answer to the above question, I just need to add a q0 field to the poly file. My only remaining query is whether there is a limit to the number of machines the factmsieve python script will allow/work with? I don't want to be too ambitious and and end up wasting precious time by trying too many...

2010-11-25, 21:09	#32
aifbowman Nov 2010 2×7 Posts	exactly what I wanted to hear :)

2010-11-26, 00:14	#33
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 14423₈ Posts	Nothing to worry about in that output; I think you must have oversieved a lot, whilst my example C160 was sieved only until I was able to get a matrix (since I had a dozen cores available against your 40 or more). And the i7 is very, very fast at doing matrices; I have a 4.5M matrix done in 33 hours on an i7 without overclocking and with slow memory, so if you've boosted your machine at all then doing the matrix in 24 hours is plausible. When you've got the result, could you also post number of relations // number of unique relations and the lpbr/lpba/mfbr/mfba/alim/rlim/alambda/rlambda lines from your polynomial file and, if you've got the time and disc space, truncate msieve.dat say 5M relations at a time, running until you get the matrix generated, so that you can get yourself an idea of how the matrix size varies as a function of oversieving.