mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Factoring (https://www.mersenneforum.org/forumdisplay.php?f=19)
-   -   Pascal's OPN roadblock files (https://www.mersenneforum.org/showthread.php?t=19066)

Happy5214 2021-02-06 12:36

That seems promising. Assuming you're working with Ryan's data, if you can successfully filter it, is it possible to send him the msieve output files so he doesn't have to redo the work?

ryanp 2021-02-06 14:23

If frmky is able to form a matrix, either he or I can run the LA. I'd also be interested to get the msieve patch so I can try the filtering step myself.

frmky 2021-02-07 17:39

1 Attachment(s)
[CODE]Sat Feb 6 23:19:11 2021 commencing 2-way merge
Sat Feb 6 23:28:47 2021 reduce to 334640603 relation sets and 328431256 unique ideals
Sat Feb 6 23:28:47 2021 commencing full merge
Sun Feb 7 04:51:15 2021 memory use: 38947.4 MB
Sun Feb 7 04:52:10 2021 found 147393213 cycles, need 147295456
Sun Feb 7 04:53:37 2021 weight of 147295456 cycles is about 19148588285 (130.00/cycle)
Sun Feb 7 04:53:37 2021 distribution of cycle lengths:
Sun Feb 7 04:53:37 2021 1 relations: 11327174
Sun Feb 7 04:53:37 2021 2 relations: 10934036
Sun Feb 7 04:53:37 2021 3 relations: 11762737
Sun Feb 7 04:53:37 2021 4 relations: 11269419
Sun Feb 7 04:53:37 2021 5 relations: 10840433
Sun Feb 7 04:53:37 2021 6 relations: 10270300
Sun Feb 7 04:53:37 2021 7 relations: 9536741
Sun Feb 7 04:53:37 2021 8 relations: 8923986
Sun Feb 7 04:53:37 2021 9 relations: 8177647
Sun Feb 7 04:53:37 2021 10+ relations: 54252983
Sun Feb 7 04:53:37 2021 heaviest cycle: 28 relations
Sun Feb 7 04:54:54 2021 commencing cycle optimization
Sun Feb 7 05:08:34 2021 start with 1239155413 relations
Sun Feb 7 07:05:08 2021 pruned 42215931 relations
Sun Feb 7 07:05:10 2021 memory use: 34774.5 MB
Sun Feb 7 07:05:11 2021 distribution of cycle lengths:
Sun Feb 7 07:05:11 2021 1 relations: 11327174
Sun Feb 7 07:05:11 2021 2 relations: 11209087
Sun Feb 7 07:05:11 2021 3 relations: 12224988
Sun Feb 7 07:05:11 2021 4 relations: 11643676
Sun Feb 7 07:05:11 2021 5 relations: 11239934
Sun Feb 7 07:05:11 2021 6 relations: 10573210
Sun Feb 7 07:05:11 2021 7 relations: 9830741
Sun Feb 7 07:05:11 2021 8 relations: 9126063
Sun Feb 7 07:05:11 2021 9 relations: 8350544
Sun Feb 7 07:05:11 2021 10+ relations: 51770039
Sun Feb 7 07:05:11 2021 heaviest cycle: 28 relations
Sun Feb 7 07:13:57 2021 RelProcTime: 199990
Sun Feb 7 07:13:57 2021 elapsed time 55:33:18[/CODE]
147.3M matrix! That would take my little 8 nodes about 2.5 months to solve!
Patch is attached. Edit: Had to add .txt to the end to get the forum to accept it.

ryanp 2021-02-07 18:01

[QUOTE=frmky;571096]147.3M matrix! That would take my little 8 nodes about 2.5 months to solve![/QUOTE]

Any chance you'd be able to run it? :smile:

Otherwise, I'll test out the patch and can try sieving some more.

frmky 2021-02-07 21:52

It's now in the official msieve SVN, so no need for the patch file. Just grab the latest code.

frmky 2021-02-08 03:15

[CODE]commencing linear algebra
read 147295456 cycles
cycles contain 531217644 unique relations
read 531217644 relations
using 8 quadratic characters above 4294917295
building initial matrix
memory use: 68478.7 MB
read 147295456 cycles
matrix is 147295267 x 147295456 (74519.1 MB) with weight 19922810683 (135.26/col)
sparse part has weight 18061791563 (122.62/col)
filtering completed in 2 passes
matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col)
sparse part has weight 18061766673 (122.63/col)
matrix starts at (0, 0)
matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col)
sparse part has weight 18061766673 (122.63/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 147292392 x 147292629 (71275.3 MB) with weight 17993852562 (122.16/col)
sparse part has weight 17211463600 (116.85/col)
using block size 8192 and superblock size 1966080 for processor cache size 20480 kB
commencing Lanczos iteration (16 threads)
memory use: 73902.4 MB
linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)[/CODE]
I'll see you in 4 years... :lol:

VBCurtis 2021-02-08 05:57

[QUOTE=frmky;571129][CODE]
linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)[/CODE]
I'll see you in 4 years... :lol:[/QUOTE]
Holy {censored}!!!!

I mean, wow. 147M dimensions.... But the ETA seems off- shouldn't this take around 4x as long as 73M dimensions? I solved one that size for 2330L in about 3100hr on a single 10-core Ivy Bridge socket, and around 2000hr when run naively (without MPI) on both 10-core sockets.

I'm not volunteering for ~8000 machine-hours, but it seems an 8-socket mini-cluster should be around 10x faster than this!

Thanks for finding and fixing this bug- I'm happy that we're likely to have msieve at our disposal for jobs as big as we can solve matrices for!

frmky 2021-02-08 06:49

I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.

I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.

ryanp 2021-02-08 23:13

[QUOTE=frmky;571140]I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.

I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.[/QUOTE]

If you're able to run this on your cluster, that would be greatly appreciated; if not, I also understand. Meanwhile I will continue sieving to see if I can form something resembling a reasonable matrix.

ryanp 2021-02-10 05:36

Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64-core workstation:

[CODE]commencing linear algebra
read 147295123 cycles
cycles contain 531218272 unique relations
read 531218272 relations
using 8 quadratic characters above 4294917295
building initial matrix
memory use: 68478.7 MB
read 147295123 cycles
matrix is 147294934 x 147295123 (74519.4 MB) with weight 19922890703 (135.26/col)
sparse part has weight 18061865192 (122.62/col)
filtering completed in 2 passes
matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col)
sparse part has weight 18061839994 (122.63/col)
matrix starts at (0, 0)
matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col)
sparse part has weight 18061839994 (122.63/col)
saving the first 240 matrix rows for later
matrix includes 256 packed rows
matrix is 147291864 x 147292281 (69444.9 MB) with weight 16505103766 (112.06/col)
sparse part has weight 15847879420 (107.59/col)
using block size 8192 and superblock size 608256 for processor cache size 25344 kB
commencing Lanczos iteration (64 threads)
memory use: 98165.2 MB
linear algebra at 0.0%, ETA 4273h 0m292281 dimensions (0.0%, ETA 4273h 0m)
checkpointing every 40000 dimensions292281 dimensions (0.0%, ETA 4287h45m)
linear algebra completed 81656 of 147292281 dimensions (0.1%, ETA 4336h46m)[/CODE]

Unfortunately I don't think I can tie up this machine for ~half year, but hopefully some more sieving can bring down the matrix size a bit.

henryzz 2021-02-10 11:00

[QUOTE=ryanp;571253]Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64-core workstation:[/QUOTE]
Is this using MPI? On a multi-cpu system that should help.


All times are UTC. The time now is 20:15.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.