mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Factoring

Reply
 
Thread Tools
Old 2021-02-06, 12:36   #914
Happy5214
 
Happy5214's Avatar
 
"Alexander"
Nov 2008
The Alamo City

26×32 Posts
Default

That seems promising. Assuming you're working with Ryan's data, if you can successfully filter it, is it possible to send him the msieve output files so he doesn't have to redo the work?
Happy5214 is offline   Reply With Quote
Old 2021-02-06, 14:23   #915
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

1000101002 Posts
Default

If frmky is able to form a matrix, either he or I can run the LA. I'd also be interested to get the msieve patch so I can try the filtering step myself.
ryanp is offline   Reply With Quote
Old 2021-02-07, 17:39   #916
frmky
 
frmky's Avatar
 
Jul 2003
So Cal

1000001000112 Posts
Default

Code:
Sat Feb  6 23:19:11 2021  commencing 2-way merge
Sat Feb  6 23:28:47 2021  reduce to 334640603 relation sets and 328431256 unique ideals
Sat Feb  6 23:28:47 2021  commencing full merge
Sun Feb  7 04:51:15 2021  memory use: 38947.4 MB
Sun Feb  7 04:52:10 2021  found 147393213 cycles, need 147295456
Sun Feb  7 04:53:37 2021  weight of 147295456 cycles is about 19148588285 (130.00/cycle)
Sun Feb  7 04:53:37 2021  distribution of cycle lengths:
Sun Feb  7 04:53:37 2021  1 relations: 11327174
Sun Feb  7 04:53:37 2021  2 relations: 10934036
Sun Feb  7 04:53:37 2021  3 relations: 11762737
Sun Feb  7 04:53:37 2021  4 relations: 11269419
Sun Feb  7 04:53:37 2021  5 relations: 10840433
Sun Feb  7 04:53:37 2021  6 relations: 10270300
Sun Feb  7 04:53:37 2021  7 relations: 9536741
Sun Feb  7 04:53:37 2021  8 relations: 8923986
Sun Feb  7 04:53:37 2021  9 relations: 8177647
Sun Feb  7 04:53:37 2021  10+ relations: 54252983
Sun Feb  7 04:53:37 2021  heaviest cycle: 28 relations
Sun Feb  7 04:54:54 2021  commencing cycle optimization
Sun Feb  7 05:08:34 2021  start with 1239155413 relations
Sun Feb  7 07:05:08 2021  pruned 42215931 relations
Sun Feb  7 07:05:10 2021  memory use: 34774.5 MB
Sun Feb  7 07:05:11 2021  distribution of cycle lengths:
Sun Feb  7 07:05:11 2021  1 relations: 11327174
Sun Feb  7 07:05:11 2021  2 relations: 11209087
Sun Feb  7 07:05:11 2021  3 relations: 12224988
Sun Feb  7 07:05:11 2021  4 relations: 11643676
Sun Feb  7 07:05:11 2021  5 relations: 11239934
Sun Feb  7 07:05:11 2021  6 relations: 10573210
Sun Feb  7 07:05:11 2021  7 relations: 9830741
Sun Feb  7 07:05:11 2021  8 relations: 9126063
Sun Feb  7 07:05:11 2021  9 relations: 8350544
Sun Feb  7 07:05:11 2021  10+ relations: 51770039
Sun Feb  7 07:05:11 2021  heaviest cycle: 28 relations
Sun Feb  7 07:13:57 2021  RelProcTime: 199990
Sun Feb  7 07:13:57 2021  elapsed time 55:33:18
147.3M matrix! That would take my little 8 nodes about 2.5 months to solve!
Patch is attached. Edit: Had to add .txt to the end to get the forum to accept it.
Attached Files
File Type: txt memory_overflow_patch.diff.txt (9.2 KB, 27 views)

Last fiddled with by frmky on 2021-02-07 at 17:41
frmky is offline   Reply With Quote
Old 2021-02-07, 18:01   #917
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

22·3·23 Posts
Default

Quote:
Originally Posted by frmky View Post
147.3M matrix! That would take my little 8 nodes about 2.5 months to solve!
Any chance you'd be able to run it?

Otherwise, I'll test out the patch and can try sieving some more.
ryanp is offline   Reply With Quote
Old 2021-02-07, 21:52   #918
frmky
 
frmky's Avatar
 
Jul 2003
So Cal

1000001000112 Posts
Default

It's now in the official msieve SVN, so no need for the patch file. Just grab the latest code.
frmky is offline   Reply With Quote
Old 2021-02-08, 03:15   #919
frmky
 
frmky's Avatar
 
Jul 2003
So Cal

2,083 Posts
Default

Code:
commencing linear algebra
read 147295456 cycles
cycles contain 531217644 unique relations
read 531217644 relations
using 8 quadratic characters above 4294917295
building initial matrix
memory use: 68478.7 MB
read 147295456 cycles
matrix is 147295267 x 147295456 (74519.1 MB) with weight 19922810683 (135.26/col)
sparse part has weight 18061791563 (122.62/col)
filtering completed in 2 passes
matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col)
sparse part has weight 18061766673 (122.63/col)
matrix starts at (0, 0)
matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col)
sparse part has weight 18061766673 (122.63/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 147292392 x 147292629 (71275.3 MB) with weight 17993852562 (122.16/col)
sparse part has weight 17211463600 (116.85/col)
using block size 8192 and superblock size 1966080 for processor cache size 20480 kB
commencing Lanczos iteration (16 threads)
memory use: 73902.4 MB
linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)
I'll see you in 4 years...
frmky is offline   Reply With Quote
Old 2021-02-08, 05:57   #920
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

3×1,579 Posts
Default

Quote:
Originally Posted by frmky View Post
Code:
linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)
I'll see you in 4 years...
Holy {censored}!!!!

I mean, wow. 147M dimensions.... But the ETA seems off- shouldn't this take around 4x as long as 73M dimensions? I solved one that size for 2330L in about 3100hr on a single 10-core Ivy Bridge socket, and around 2000hr when run naively (without MPI) on both 10-core sockets.

I'm not volunteering for ~8000 machine-hours, but it seems an 8-socket mini-cluster should be around 10x faster than this!

Thanks for finding and fixing this bug- I'm happy that we're likely to have msieve at our disposal for jobs as big as we can solve matrices for!

Last fiddled with by VBCurtis on 2021-02-08 at 05:57
VBCurtis is offline   Reply With Quote
Old 2021-02-08, 06:49   #921
frmky
 
frmky's Avatar
 
Jul 2003
So Cal

2,083 Posts
Default

I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.

I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.
frmky is offline   Reply With Quote
Old 2021-02-08, 23:13   #922
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

27610 Posts
Default

Quote:
Originally Posted by frmky View Post
I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.

I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.
If you're able to run this on your cluster, that would be greatly appreciated; if not, I also understand. Meanwhile I will continue sieving to see if I can form something resembling a reasonable matrix.
ryanp is offline   Reply With Quote
Old 2021-02-10, 05:36   #923
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

22×3×23 Posts
Default

Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64-core workstation:

Code:
commencing linear algebra
read 147295123 cycles
cycles contain 531218272 unique relations
read 531218272 relations
using 8 quadratic characters above 4294917295
building initial matrix
memory use: 68478.7 MB
read 147295123 cycles
matrix is 147294934 x 147295123 (74519.4 MB) with weight 19922890703 (135.26/col)
sparse part has weight 18061865192 (122.62/col)
filtering completed in 2 passes
matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col)
sparse part has weight 18061839994 (122.63/col)
matrix starts at (0, 0)
matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col)
sparse part has weight 18061839994 (122.63/col)
saving the first 240 matrix rows for later
matrix includes 256 packed rows
matrix is 147291864 x 147292281 (69444.9 MB) with weight 16505103766 (112.06/col)
sparse part has weight 15847879420 (107.59/col)
using block size 8192 and superblock size 608256 for processor cache size 25344 kB
commencing Lanczos iteration (64 threads)
memory use: 98165.2 MB
linear algebra at 0.0%, ETA 4273h 0m292281 dimensions (0.0%, ETA 4273h 0m)    
checkpointing every 40000 dimensions292281 dimensions (0.0%, ETA 4287h45m)    
linear algebra completed 81656 of 147292281 dimensions (0.1%, ETA 4336h46m)
Unfortunately I don't think I can tie up this machine for ~half year, but hopefully some more sieving can bring down the matrix size a bit.
ryanp is offline   Reply With Quote
Old 2021-02-10, 11:00   #924
henryzz
Just call me Henry
 
henryzz's Avatar
 
"David"
Sep 2007
Cambridge (GMT/BST)

5,857 Posts
Default

Quote:
Originally Posted by ryanp View Post
Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64-core workstation:
Is this using MPI? On a multi-cpu system that should help.
henryzz is online now   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Passive Pascal Xyzzy GPU Computing 1 2017-05-17 20:22
Tesla P100 — 5.4 DP TeraFLOPS — Pascal Mark Rose GPU Computing 52 2016-07-02 12:11
Nvidia Pascal, a third of DP firejuggler GPU Computing 12 2016-02-23 06:55
Calculating perfect numbers in Pascal Elhueno Homework Help 5 2008-06-12 16:37
Factorization attempt to a c163 - a new Odd Perfect Number roadblock jchein1 Factoring 30 2005-05-30 14:43

All times are UTC. The time now is 09:41.

Sun Apr 18 09:41:29 UTC 2021 up 10 days, 4:22, 0 users, load averages: 1.54, 1.43, 1.45

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.