Distributed finishing for 2,1870L - Page 6

smh · 2011-02-21, 20:16

Quote:

Originally Posted by R.D. Silverman

I will drop another ~10M relations in snail mail to you later today.
You should get them in ~2days.

Our firewall prevents a direct transfer of the data.

Just wondering, can't you send them from home? Even a slow ADSL would probably take only a couple of hours and would save the trouble of copying the files to CD.

Maybe it's time to store relations in a more efficient format or a separate utility to compress them in a more suitable format then *zip?

Batalov · 2011-02-21, 20:36

Yes,
You can get the CD home and from home with the PuTTY psftp.exe (or on linux, sftp) and you are all set, I'll send you the instructions by PM.
(Especially that US post has a day off today, I believe)
--Serge

jrk · 2011-02-21, 21:07

Quote:

Originally Posted by smh

Maybe it's time to store relations in a more efficient format or a separate utility to compress them in a more suitable format then *zip?

One could just store the a,b coordinates for each relation, which would compress down to about 8.7 bytes per relation with bzip2. Or, you could squeeze it down further by recording the lattice basis vectors for each specialq,root and then store the i,j lattice coordinates for each relation instead, which I guess would require about half the storage. "Decompressing" would be slow in either case.

This might make it viable to send through email, if that was the only available method of transmission, though 10M relations will still be quite large.

fivemack · 2011-02-22, 01:42

Quote:

Originally Posted by jrk

One could just store the a,b coordinates for each relation, which would compress down to about 8.7 bytes per relation with bzip2. Or, you could squeeze it down further by recording the lattice basis vectors for each specialq,root and then store the i,j lattice coordinates for each relation instead, which I guess would require about half the storage.

You don't need to record the vectors while you're sieving, you can recover them by rolling lattice-basis reduction ... keep the LLL of the last three x,y pairs, most of the time you'll be able to write the next one in terms of that basis and otherwise you output in clear.

bsquared · 2011-02-22, 18:35

113-114 is done and is uploading.

Batalov · 2011-02-24, 08:00

Quote:

Originally Posted by R.D. Silverman

I am having shoulder surgery on 2/24 to remove some bone spurs
and repair my rotator cuff.

We wish you the best of luck with the procedure, Bob, and a fast recovery!

The ETA for this project is late night Friday, don't worry about it.

Batalov · 2011-02-26, 01:28

The cofactor is a product of a p75 and a p105 and will appear on page 120 as Silverman+mersenneforum snfs. Thanks, everyone!

Batalov · 2011-02-26, 10:24

P.S. Here's the log (with a few filtering attempts at different points, and with a finely articulated cusp at 12.0M²; the good matrix is 7.6M² and a 4^x times less ETA).

jasonp · 2011-02-27, 17:04

I must be getting spoiled, this is the first big run I've seen in a long time that had to perform singleton removal from disk files. Apparently the estimate of RAM needed was just past the switchover point (half of total RAM).

Batalov · 2011-02-27, 21:39

Yeah, it works great (not everyone has 32Gb of RAM

)
Here's a fragment from 3,610+'s log
...
Tue Jan 4 08:12:25 2011 commencing duplicate removal, pass 1
Tue Jan 4 08:42:56 2011 found 10205937 hash collisions in 216381785 relations
Tue Jan 4 08:43:59 2011 commencing duplicate removal, pass 2
Tue Jan 4 08:47:58 2011 found 21 duplicates and 216381765 unique relations
Tue Jan 4 08:47:58 2011 memory use: 756.8 MB
Tue Jan 4 08:47:58 2011 reading ideals above 720000
Tue Jan 4 08:47:58 2011 commencing singleton removal, initial pass
Tue Jan 4 09:31:56 2011 memory use: 5512.0 MB
Tue Jan 4 09:31:56 2011 removing singletons from LP file
Tue Jan 4 09:31:56 2011 start with 216381765 relations and 182327021 ideals
Tue Jan 4 09:35:35 2011 pass 1: found 44887504 singletons
Tue Jan 4 09:37:07 2011 pass 2: found 8615107 singletons
Tue Jan 4 09:38:34 2011 pass 3: found 1627313 singletons
Tue Jan 4 09:40:02 2011 pass 4: found 290576 singletons
Tue Jan 4 09:42:51 2011 pruned dataset has 160961265 relations and 122666568 large ideals
Tue Jan 4 09:42:51 2011 reading all ideals from disk
Tue Jan 4 09:43:57 2011 memory use: 6481.4 MB
Tue Jan 4 09:44:54 2011 keeping 121915663 ideals with weight <= 200, target excess is 867008
Tue Jan 4 09:45:59 2011 commencing in-memory singleton removal
Tue Jan 4 09:46:46 2011 begin with 160961265 relations and 121915663 unique ideals
Tue Jan 4 09:53:03 2011 reduce to 160900260 relations and 121854650 ideals in 7 passes
...
Great stuff (and results are identical to the ones on a large-memory machine) compared to filtering from ideals with, say, weight <= 40 as it were in the past. Many thanks!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Finishing expired LL test	Damian	PrimeNet	2	2017-11-16 00:36
1870L : Yield Rate	R.D. Silverman	Cunningham Tables	44	2010-10-21 14:32
Finishing mprime runs	bill-p	Software	1	2009-12-08 17:45
distributed.net completes OGR-25	ixfd64	Lounge	4	2008-11-22 01:59
LL assignments on slow PCs "die" shortly before finishing?	Andi47	PrimeNet	1	2007-02-28 22:03

2011-02-21, 20:36	#57
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 2⁴·593 Posts	Yes, You can get the CD home and from home with the PuTTY psftp.exe (or on linux, sftp) and you are all set, I'll send you the instructions by PM. (Especially that US post has a day off today, I believe) --Serge

2011-02-22, 18:35	#60
bsquared "Ben" Feb 2007 2²×3×293 Posts	113-114 is done and is uploading.

2011-02-26, 01:28	#62
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 2⁴·593 Posts	The cofactor is a product of a p75 and a p105 and will appear on page 120 as Silverman+mersenneforum snfs. Thanks, everyone!

2011-02-27, 17:04	#64
jasonp Tribal Bullet Oct 2004 3·1,181 Posts	I must be getting spoiled, this is the first big run I've seen in a long time that had to perform singleton removal from disk files. Apparently the estimate of RAM needed was just past the switchover point (half of total RAM).

2011-02-27, 21:39	#65
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 2⁴×593 Posts	Yeah, it works great (not everyone has 32Gb of RAM ) Here's a fragment from 3,610+'s log ... Tue Jan 4 08:12:25 2011 commencing duplicate removal, pass 1 Tue Jan 4 08:42:56 2011 found 10205937 hash collisions in 216381785 relations Tue Jan 4 08:43:59 2011 commencing duplicate removal, pass 2 Tue Jan 4 08:47:58 2011 found 21 duplicates and 216381765 unique relations Tue Jan 4 08:47:58 2011 memory use: 756.8 MB Tue Jan 4 08:47:58 2011 reading ideals above 720000 Tue Jan 4 08:47:58 2011 commencing singleton removal, initial pass Tue Jan 4 09:31:56 2011 memory use: 5512.0 MB Tue Jan 4 09:31:56 2011 removing singletons from LP file Tue Jan 4 09:31:56 2011 start with 216381765 relations and 182327021 ideals Tue Jan 4 09:35:35 2011 pass 1: found 44887504 singletons Tue Jan 4 09:37:07 2011 pass 2: found 8615107 singletons Tue Jan 4 09:38:34 2011 pass 3: found 1627313 singletons Tue Jan 4 09:40:02 2011 pass 4: found 290576 singletons Tue Jan 4 09:42:51 2011 pruned dataset has 160961265 relations and 122666568 large ideals Tue Jan 4 09:42:51 2011 reading all ideals from disk Tue Jan 4 09:43:57 2011 memory use: 6481.4 MB Tue Jan 4 09:44:54 2011 keeping 121915663 ideals with weight <= 200, target excess is 867008 Tue Jan 4 09:45:59 2011 commencing in-memory singleton removal Tue Jan 4 09:46:46 2011 begin with 160961265 relations and 121915663 unique ideals Tue Jan 4 09:53:03 2011 reduce to 160900260 relations and 121854650 ideals in 7 passes ... Great stuff (and results are identical to the ones on a large-memory machine) compared to filtering from ideals with, say, weight <= 40 as it were in the past. Many thanks!