The big filtering bug strikes again (I think)

[Dubslow]

unconnected, would you be willing to finish the matrix? If not, wombatman? I think it'll be a few days before my buddy downloads the other files of his that are on this server, at which point I'll start over with another job, or more likely, do some ECM on 4788.

[wombatman]

If unconnected already built the matrix, he's welcome to finish it up. Otherwise, I can tackle it.

[wombatman]

Alright, I'm going ahead and downloading everything--it should finish after I head to bed, so I will start it in the morning if I don't hear anything different.

[unconnected]

Originally Posted by wombatman

If unconnected already built the matrix, he's welcome to finish it up. Otherwise, I can tackle it.

Ok, I'll start LA.

[unconnected]

I rebuilt the matrix with TD=130 and got ETA ~90 hours (there was ETA ~83 hours with TD=120). I thought that filtering with larger density produces smaller matrix, so processing time decreases. But not in this case. Logs below.

density 120

Wed Jan 27 11:16:44 2016  commencing linear algebra
Wed Jan 27 11:16:44 2016  skipping matrix build
Wed Jan 27 11:16:44 2016  initialized process (0,0) of 1 x 2 grid
Wed Jan 27 11:16:53 2016  matrix starts at (0, 0)
Wed Jan 27 11:16:55 2016  matrix is 13308537 x 6654365 (3138.7 MB) with weight 901872971 (135.53/col)
Wed Jan 27 11:16:55 2016  sparse part has weight 749600001 (112.65/col)
Wed Jan 27 11:16:55 2016  saving the first 48 matrix rows for later
Wed Jan 27 11:16:57 2016  matrix includes 64 packed rows
Wed Jan 27 11:17:00 2016  matrix is 13308489 x 6654365 (3009.7 MB) with weight 774255725 (116.35/col)
Wed Jan 27 11:17:00 2016  sparse part has weight 722430143 (108.56/col)
Wed Jan 27 11:17:00 2016  using block size 8192 and superblock size 1474560 for processor cache size 15360 kB
Wed Jan 27 11:17:59 2016  commencing Lanczos iteration (12 threads)
Wed Jan 27 11:17:59 2016  memory use: 2624.3 MB
Wed Jan 27 11:18:36 2016  linear algebra at 0.0%, ETA 82h54m

density 130

Sat Jan 30 00:11:30 2016  commencing linear algebra
Sat Jan 30 00:11:30 2016  skipping matrix build
Sat Jan 30 00:11:30 2016  initialized process (0,0) of 1 x 2 grid
Sat Jan 30 00:11:39 2016  matrix starts at (0, 0)
Sat Jan 30 00:11:41 2016  matrix is 12990826 x 6495509 (3278.2 MB) with weight 939459774 (144.63/col)
Sat Jan 30 00:11:41 2016  sparse part has weight 787922785 (121.30/col)
Sat Jan 30 00:11:41 2016  saving the first 48 matrix rows for later
Sat Jan 30 00:11:43 2016  matrix includes 64 packed rows
Sat Jan 30 00:11:46 2016  matrix is 12990778 x 6495509 (3145.5 MB) with weight 813366370 (125.22/col)
Sat Jan 30 00:11:46 2016  sparse part has weight 759615083 (116.94/col)
Sat Jan 30 00:11:46 2016  using block size 8192 and superblock size 1474560 for processor cache size 15360 kB
Sat Jan 30 00:12:46 2016  commencing Lanczos iteration (12 threads)
Sat Jan 30 00:12:46 2016  memory use: 2745.8 MB
Sat Jan 30 00:13:27 2016  linear algebra at 0.0%, ETA 90h12m

[VBCurtis]

Looks like the size of the matrix went down, but total weight went up.
If the ETAs are accurate, it looks like ETA scales with the square of matrix weight rather than the square of dimension; a 5% increase in weight has produced a nearly 10% increase in ETA.

It may be that the ETA algorithm isn't as accurate at density 120+, but if you'd let each run an hour to get a more accurate ETA they may have converged.

An interesting msieve optimization would be to set filtering to automatically run 3 times at various target densities, proceeding to solve the matrix with the lowest ETA. I think the cost of 2 extra filtering runs is less than the time gained by a better matrix, but I haven't tried manually doing it to test it (say, target-density 112, 120, 128 for an NFS@home project).

Interesting post, sir.

[Dubslow]

In my experience, the ETA changes substantially (25% or more) after the first one or two percent. That first guess ETA might not be accurate.

[unconnected]

Originally Posted by Dubslow

In my experience, the ETA changes substantially (25% or more) after the first one or two percent. That first guess ETA might not be accurate.

Maybe other processes affected your machine? I use console server where even system daemons like syslogd/snmpd/ntpd/etc were stopped and ETA always pretty accurate.

From the second run right now:

linear algebra completed 237509 of 12991003 dimensions (1.8%, ETA 88h26m)

After LA completes, I'll experiment with saved datasets from several C16x and one C170 GNFS jobs against various TDs.

[WraithX]

Originally Posted by VBCurtis

Looks like the size of the matrix went down, but total weight went up.
If the ETAs are accurate, it looks like ETA scales with the square of matrix weight rather than the square of dimension; a 5% increase in weight has produced a nearly 10% increase in ETA.

It may be that the ETA algorithm isn't as accurate at density 120+, but if you'd let each run an hour to get a more accurate ETA they may have converged.

An interesting msieve optimization would be to set filtering to automatically run 3 times at various target densities, proceeding to solve the matrix with the lowest ETA. I think the cost of 2 extra filtering runs is less than the time gained by a better matrix, but I haven't tried manually doing it to test it (say, target-density 112, 120, 128 for an NFS@home project).

Interesting post, sir.

I tried many different TD values for my C210 GNFS run. The ETA was strongly dependent on matrix size, and not weight. But, I believe I also ran into an issue where using 13 threads on 16 cores (dual 8 core processors), the ETA would depend on how well Windows allocated the memory in relation to how the msieve threads were distributed across the two processors. Here is the summary of the different TD values I tried:

TD 70 - ETA 4002h57m: matrix is 88382727 x 88382952 (25970.7 MB) with weight 6847042373 (77.47/col) sparse part has weight 5747458016 (65.03/col)
TD 80 - ETA 4640h 7m: matrix is 83909604 x 83909829 (27454.4 MB) with weight 7346597637 (87.55/col) sparse part has weight 6190094189 (73.77/col)
TD 85 - ETA 4513h 4m: matrix is 81992464 x 81992689 (28181.3 MB) with weight 7584940314 (92.51/col) sparse part has weight 6403641518 (78.10/col)
TD 90 - ETA 3602h20m: matrix is 80299721 x 80299946 (28917.8 MB) with weight 7824093700 (97.44/col) sparse part has weight 6617039915 (82.40/col)
TD 95 - ETA 3668h10m: matrix is 78767972 x 78768197 (29656.9 MB) with weight 8060707185 (102.33/col) sparse part has weight 6829171047 (86.70/col)
TD100 - ETA 4282h56m: matrix is 77349463 x 77349687 (30375.4 MB) with weight 8288226977 (107.15/col) sparse part has weight 7034545610 (90.94/col)
TD110 - ETA 3929h22m: matrix is 74870112 x 74870338 (31818.0 MB) with weight 8739992621 (116.74/col) sparse part has weight 7442460412 (99.40/col)
TD120 - ETA 3811h56m: matrix is 72742805 x 72743029 (33234.9 MB) with weight 9176491366 (126.15/col) sparse part has weight 7839402666 (107.77/col)
TD125 - ETA 3141h 6m: matrix is 71776154 x 71776379 (33924.4 MB) with weight 9386761820 (130.78/col) sparse part has weight 8031767895 (111.90/col)
TD125 - ETA 3750h36m: matrix is 71776154 x 71776379 (33924.4 MB) with weight 9386761820 (130.78/col) sparse part has weight 8031767895 (111.90/col)
TD125 - ETA 3054h11m: matrix is 71776154 x 71776379 (33924.4 MB) with weight 9386761820 (130.78/col) sparse part has weight 8031767895 (111.90/col)
TD128: Failed
TD130: Failed
TD150: Failed

This wasn't the order I tried the various TD values, this is just sorted by TD size. You can see some more details in the attached log file. However, with the TD125 size, you can see how the high ETA makes it looks like it had poorly allocated the memory for the job in relation to the msieve process. At least, that is my interpretation of the above numbers.

Also, the ETA was very accurate for the TD125 job that I let finish. It had estimated 3054 hours and it took 3052 hours.

[unconnected]

Originally Posted by unconnected

I use console server where even system daemons like syslogd/snmpd/ntpd/etc were stopped and ETA always pretty accurate.

Sat Jan 30 00:13:27 2016  linear algebra at 0.0%, ETA 90h12m
Sat Jan 30 00:13:39 2016  checkpointing every 150000 dimensions
Tue Feb  2 18:22:35 2016  lanczos halted after 205436 iterations (dim = 12990776)
Tue Feb  2 18:23:01 2016  recovered 32 nontrivial dependencies
Tue Feb  2 18:23:02 2016  BLanczosTime: 324692
Tue Feb  2 18:23:02 2016  elapsed time 90:11:39