[QUOTE=EdH;226099]My current copy of Msieve does not appear to want to stop polynomial selection. On Monday I was running Aliqueit against a c91 with gnfs_cutoff set to 89 and use_msieve_polyfind = true. After five hours of endless polynomials, I stopped it and changed gnfs_cutoff to 95. The c91 subsequently finished in two hours via SIQS.

Today I had a c99 turn up and had set gnfs_cutoff to 95 and use_msieve_polyfind = false. This time the polynomial selection ran for 2.5 hours before I shut it down.[/QUOTE]
This gets asked a lot.

If I can remember correctly msieve uses CPU time for the overall search timeout, not walltime. So if there are other tasks running on your machine, it may take longer for msieve to reach its timeout.

Might that be the case?

There's a similar situation to this when a GPU is used, but it appears you aren't using one.

It might be time to rethink how the timeout is used.

I have seen this several times but not really sure what the .dat.lp file is used for? Sometimes after doing filtering (-nc1) I get both .dat.cyc and .dat.lp files. Other times I just get a .dat.cyc. How come this file is sometimes there and sometimes not?

Jeff.

Unlike the other NFS phases, the poly selection does not have a set amount of work to do. The current code uses the elapsed CPU time to bound the limit for the overall search, and the wallclock time to bound the search for each set of coefficients within that. So if you have other CPU-intensive processes running at the same time, msieve will not get much CPU time and the search will drag on forever, but coefficients will change every so often (about every 5 seconds given the size of your inputs).

Originally the code used wallclock time for everything, but that does the wrong thing when your machine goes into standby mode (i.e. overnight). On wakeup, a lot of time has passed so the search stops immediately. I guess this is a case where everyone wants different behavior: "run at full speed when the CPU is idle, but stop after a given time, and make the time longer when the CPU is busy, but ignore all of that when CPU is in standby". That should be perfect :)

Jeff: the .lp file is used to list the relations that survive the initial filtering stages, along with only their large primes. It saves a lot of time to avoid having to parse each relation over and over again only to throw most of it away. The .lp file is created whenever the filtering runs, and is deleted if filtering succeeds but not otherwise IIRC.

[QUOTE=jasonp;226129]Jeff: the .lp file is used to list the relations that survive the initial filtering stages, along with only their large primes. It saves a lot of time to avoid having to parse each relation over and over again only to throw most of it away. The .lp file is created whenever the filtering runs, and is deleted if filtering succeeds but not otherwise IIRC.[/QUOTE]

Ok thanks, so I guess these were files left over from my stopping a run or having a problem since 99% of the time an .lp file is there. So I can delete the .lp file and just need to send over the .dat and .cyc to build a matrix. Good to know.

Jeff.

Thanks. I was running aliqueit without the idle priority switch, but don't know what else may have been running. A quick check right now shows msieve using ~84% and the system monitor using ~12% with everything else asleep. I have the screensaver set to just go blank. I will see how it turns out later today (and of course, close that "wasteful" system monitor). Maybe this machine (or this Fedora) is less friendly to msieve.:sad:

With my latest RSALS job [[B]599_83_minus1 - SNFS(233)[/B]], I tried using LARGEBLOCKS and TARGET_DENSITY=80 to see how things changed.

I can conclude that for a job of this size, the LARGEBLOCKS=1 made a major difference and increasing TARGET_DENSITY from 70 to 80 made a minor difference.

These tests were performed on a single CPU core of a dual-4-core Xeon E5405 system (2.0GHz). I let msieve run for 30 minutes with -nc2 to generate the matrix, then I stopped to create a check-point. Restarted the largeblocks and normal versions with -ncr from the same check point and let it run for 10 minutes before I took the ETA time. So they are all tested pretty close to the same dimension being completed.

[CODE]TARGET_DENSITY = 70
matrix is 7416958 x 7417185 (2127.1 MB) with weight 525245035 (70.81/col)
Normal : memory use: 1738.3 MB (0.1%, ETA 339h 9m)
Largeblocks: memory use: 2446.3 MB (0.1%, ETA 284h41m)

TARGET_DENSITY = 80
matrix is 7081574 x 7081799 (2265.5 MB) with weight 567627391 (80.15/col)
Normal : memory use: 1866.1 MB (0.1%, ETA 334h18m)
Largeblocks: memory use: 2631.1 MB (0.1%, ETA 279h43m)[/CODE]

In both target density cases, going from normal block size to large block size increased RAM usage a significant amount but decreased the ETA by 16% of the original time.

In both block size cases, increasing the target density from 70 to 80 increased RAM usage slightly but only decreased ETA by 2%.

Does LARGEBLOCKS work in the MPI version or is that a completely different beast because of the way you split things up?

[QUOTE=Jeff Gilchrist;226182]Does LARGEBLOCKS work in the MPI version or is that a completely different beast because of the way you split things up?[/QUOTE]
It works with MPI as well, and there memory isn't an issue at all since each computer only holds its part of the matrix so memory usage per computer is significantly reduced.

[QUOTE=jasonp;226129]I guess this is a case where everyone wants different behavior: "run at full speed when the CPU is idle, but stop after a given time, and make the time longer when the CPU is busy, but ignore all of that when CPU is in standby". That should be perfect :)[/QUOTE]
Yep, that sounds adequately impossible.

OK, this is a c99. It's been poly searching since 10:29 this morning. It is the only thing running and is at normal priority. It is now after 17:00 and it is still clocking along writing:
[code]
. . .
coeff 232920-232920 9283222 10211544 10211545 11232699
------- 9283222-10211544 10211545-11232699
poly 0 p 10145257 q 10635401 coeff 107898876443057
poly 0 p 9609937 q 10689203 coeff 102722567410211
poly 0 p 9831181 q 10892749 coeff 107088587006569
poly 0 p 9974929 q 10667741 coeff 106409959065389
poly 0 p 9655237 q 10525837 coeff 101629450858369
poly 0 p 9501293 q 10954979 coeff 104086465287847
poly 0 p 9673541 q 11200567 coeff 108349144097747
poly 0 p 9450689 q 11165287 coeff 105519655032743
poly 0 p 9639349 q 10307983 coeff 99362245623067
poly 0 p 9447233 q 10556119 coeff 99726115768727
poly 0 p 10106077 q 10282307 coeff 103913786279639
poly 0 p 10091177 q 10413431 coeff 105083775398287
coeff 232980-232980 9283372 10211709 10211710 11232881
------- 9283372-10211709 10211710-11232881
poly 0 p 10096169 q 10743461 coeff 108467797900909
. . .
[/code]Each of these blocks is taking about six wall clock seconds.

I'll let it continue until I shut it down about five or six hours from now and see if it ever moves on, but I think there is something amiss. . .

More thoughts?

If you want to enforce a hard deadline, you can add '-d X' to the command line and msieve will interrupt itself after X wall-clock minutes.

[QUOTE=jasonp;226129]"run at full speed when the CPU is idle, but stop after a given time, and make the time longer when the CPU is busy, but ignore all of that when CPU is in standby". That should be perfect :)[/QUOTE]

LOL. But why do a variable search in the first place? Can't the search be parameterised so that it does a fixed amount of work (modulo the use of randomness in the search space)? Ideally speaking, the amount of work (not time) to be expended is a function on the amount of work to be expended on the other phases (sieving, LA, etc.) -- why should having a faster CPU or GPU cause me to do a deeper search?