Raman's plans and questions

[Raman]

2,1778L so far

[Raman]

7,320+ right now only
by now itself, as of now:

[xilman]

Originally Posted by Raman

By the way, I am adding up the Euclid-Millen sequence, analogous -1 sequence, Woodall numbers factors, Leyland numbers factors, to the list of the above distributed computing projects, that I missed out already only. So, did I miss out anymore of the important distributed computing projects cases, that are already being existing up only, thus?

You appear to have missed the Cullen numbers, and the generalized Cullen and Woodall numbers. The first are a*2^a+1 and the others are a*b^a + and a*b^a - 1.

Paul

[FactorEyes]

:%s/\<up\>//g

[Batalov]

Originally Posted by Raman

Matrix for 2,1778L is 6175062 x 6175310
and that for 7,320+ is 5801778 x 5802310
just at the threshold of fitting up within 2 GB of memory.
...

Just above the threshold.
This shows that the process is severerly swapping:

Fri Jan 1 14:37:22 2010 commencing Lanczos iteration (2 threads)
Fri Jan 1 14:37:23 2010 memory use: 1810.4 MB
Fri Jan 1 16:20:17 2010 linear algebra at 0.0%, ETA 10987h25m

This is more than a year (while it should take just a few days on a machine with 3Gb+); not to mention that after a month the hard drive will die a horrible clicking death.

[Raman]

Originally Posted by Batalov

Just above the threshold.
This shows that the process is severerly swapping:

Fri Jan 1 14:37:22 2010 commencing Lanczos iteration (2 threads)
Fri Jan 1 14:37:23 2010 memory use: 1810.4 MB
Fri Jan 1 16:20:17 2010 linear algebra at 0.0%, ETA 10987h25m

This is more than a year (while it should take just a few days on a machine with 3Gb+); not to mention that after a month the hard drive will die a horrible clicking death.

No, that this was an initially poor estimate of the matrix timing. Exactly after 24 hours, the matrix has completed upto 16.5%, and then that I must have added up that the ETA was hardly only 138 hours, at that time. It will take only upto 5 to 6 days to get finished off.

I must have modified the initial ETA value within the msieve log file to something sort of reasonable values. It uses only upto 90% of the machine's memory, that the machine is not slowed down at all.

[Raman]

This morning I saw that linear algebra had been interrupted for both 2,1778L and 7,320+
Then, when I restarted up msieve, it says that "Checkpoint recovery failed"
How to avoid this fault with msieve in this case only?

PS: Checkpoint file for 2,1778L is 179.1 MB
that for 7,320+ is 164.5 MB

Linear Algebra for 2,1778L was at 58%
that for 7,320+ was at 38%

[Batalov]

If you have an earlier saved .chk file, then use it.
If you don't, nothing much to do other than start again with -nc2.

Corollary: backup .dat, .cyc, .mat safely once; backup .chk files periodically (every day or so; they are smaller).

[Raman]

Originally Posted by Batalov

If you have an earlier saved .chk file, then use it.
If you don't, nothing much to do other than start again with -nc2.

Corollary: backup .dat, .cyc, .mat safely once; backup .chk files periodically (every day or so; they are smaller).

I will run the linear algebra for 2,1778L and 7,320+ starting from scratch on 8 core cluster with 8 GB RAM at my college. No disturbance will be there from others at all. 10,339+ and 10,351+ a few days after that only.

PATIENCE please: I have just got an account today, it will take tomorrow to figure out how to run jobs on that cluster. I tried out today. Job is not executing properly as my user id/password is not properly registered within the YP server. Should start up tomorrow only. Once jobs are executing properly in the compute cluster, I can start doing harder jobs, for example 2,935-

MOST IMPORTANT: For running harder jobs, for sieving, I need to try my best to automate the jobs, in order to get best utilization of resources, and then complete up with the sieving as quickly as possible.

I don't use the perl script at all. What is the command line for GGNFS lattice siever to resume the jobs from the previous checkpoint? I noticed that the latest GGNFS binary had this feature as well:

gnfs-lasieve4I15e -k -o spairs.out -v -n0 -r number.job -R
does not resume properly at all. It rather starts over from the unchanged q0 value from the old job file itself, does not change the value of q0 to the value in .last_spq0 file at all. Also that the qintsize value should be changed to q1-q0 value. What is the proper command to do this only?

Resuming with -f 1 -c 0
What is -f 1 -c 0? How to alter these values in order to ensure that the GGNFS lattice siever restarts correctly exactly from the previous checkpoint special-q value only?

Actually, that I think of writing a small shell script within crontab file, which checks out if the GGNFS job is running every hour, if not resume the binary only. Something like this:

#minute hour date month day
0 * * * *
x = `pgrep gnfs | wc -l`
if test $x -eq 0
then
./gnfs-lasieve4I15e <arguments for proper resuming of job only>
# The GGNFS lattice siever command line itself
fi

BINGO!

[fivemack]

The resume functionality is written on the assumption that you use -f 7300000 -c 100000 on the command-line to define the sieving range, rather than q0 and qintsize in the jobfile.

gnfs-lasieve4I15e -a job -f 7300000 -c 100000 -o output -R

with 'job' not having any definition of q in it

has worked pretty reliably for me. It figures out the last q by reading 'output' rather than by looking at any other file.

[Raman]

What the hell is this?

Thu Jan  7 19:24:59 2010  commencing linear algebra
Thu Jan  7 19:30:47 2010  read 6175310 cycles
Thu Jan  7 19:31:20 2010  cycles contain 17029975 unique relations
Thu Jan  7 20:14:28 2010  read 17029975 relations
Thu Jan  7 20:15:14 2010  using 20 quadratic characters above 536870684
Thu Jan  7 20:17:01 2010  building initial matrix
Thu Jan  7 20:29:06 2010  memory use: 2194.7 MB
Thu Jan  7 20:30:03 2010  read 6175310 cycles
Thu Jan  7 20:30:09 2010  matrix is 6175110 x 6175310 (1819.0 MB) with weight 529991724 (85.82/col)
Thu Jan  7 20:30:09 2010  sparse part has weight 415095106 (67.22/col)
Thu Jan  7 20:32:43 2010  filtering completed in 1 passes
Thu Jan  7 20:32:44 2010  matrix is 6175110 x 6175310 (1819.0 MB) with weight 529991724 (85.82/col)
Thu Jan  7 20:32:44 2010  sparse part has weight 415095106 (67.22/col)
Thu Jan  7 20:33:25 2010  read 6175310 cycles
Thu Jan  7 20:33:31 2010  matrix is 6175110 x 6175310 (1819.0 MB) with weight 529991724 (85.82/col)
Thu Jan  7 20:33:31 2010  sparse part has weight 415095106 (67.22/col)
Thu Jan  7 20:33:31 2010  saving the first 48 matrix rows for later
Thu Jan  7 20:33:34 2010  matrix is 6175062 x 6175310 (1747.2 MB) with weight 424246901 (68.70/col)
Thu Jan  7 20:33:34 2010  sparse part has weight 396272778 (64.17/col)
Thu Jan  7 20:33:34 2010  matrix includes 64 packed rows
Thu Jan  7 20:33:34 2010  using block size 65536 for processor cache size 4096 kB
Thu Jan  7 20:35:19 2010  commencing Lanczos iteration (8 threads)
Thu Jan  7 20:35:19 2010  memory use: 2093.1 MB
Thu Jan  7 20:36:11 2010  linear algebra at 0.0%, ETA 110h43m
Mon Jan 11 11:56:40 2010  lanczos halted after 97652 iterations (dim = 6175059)
Mon Jan 11 11:57:03 2010  recovered 34 nontrivial dependencies
Mon Jan 11 11:57:03 2010  BLanczosTime: 318724
Mon Jan 11 11:57:03 2010  elapsed time 88:32:11
Mon Jan 11 11:57:04 2010  
Mon Jan 11 11:57:04 2010  
Mon Jan 11 11:57:04 2010  Msieve v. 1.43
Mon Jan 11 11:57:04 2010  random seeds: 6b6302aa 275b3b48
Mon Jan 11 11:57:04 2010  factoring 187456795062290175781588001552911615336516481836838901307910624762737156478204777835354105902206650861355396494419638960993048427307433062550898806001787475198429902286327892727708624452163107713035095622371850541 (213 digits)
Mon Jan 11 11:57:07 2010  no P-1/P+1/ECM available, skipping
Mon Jan 11 11:57:07 2010  commencing number field sieve (213-digit input)
Mon Jan 11 11:57:07 2010  R0: -170141183460469231731687303715884105729
Mon Jan 11 11:57:07 2010  R1:  9223372036854775808
Mon Jan 11 11:57:07 2010  A0:  8
Mon Jan 11 11:57:07 2010  A1:  32
Mon Jan 11 11:57:07 2010  A2:  16
Mon Jan 11 11:57:07 2010  A3: -20
Mon Jan 11 11:57:07 2010  A4: -10
Mon Jan 11 11:57:07 2010  A5:  2
Mon Jan 11 11:57:07 2010  A6:  1
Mon Jan 11 11:57:07 2010  skew 1.00, size 3.281311e-11, alpha 2.151949, combined = 1.437125e-12
Mon Jan 11 11:57:07 2010  
Mon Jan 11 11:57:07 2010  commencing square root phase
Mon Jan 11 11:57:07 2010  reading relations for dependency 1
Mon Jan 11 11:57:10 2010  read 3088087 cycles
Mon Jan 11 11:57:26 2010  cycles contain 8515254 unique relations
Mon Jan 11 12:39:15 2010  read 8515254 relations
Mon Jan 11 12:40:25 2010  multiplying 8515254 relations
Segmentation fault

I have now access to 8 core, 8 GB RAM compute cluster at my college, which is suitable for running linear algebra jobs. Linear Algebra went smoothly under 8 core system, but square root failed....

I transferred the dependency file to my local file system at my department, and the square root phase says "Algebraic side is not a square!", "Number of relations is not even", for 10,339+ and 7,320+

Since the process is running in a remote node within the cluster, I do not know what is happening with the process until it actually gets completed fully.

Using the old binary for msieve-1.42 to run the square root phase in the cluster, it destroys all the important files, Relations, checkpoint, matrix, large prime, cycle, dependency... safely I have backed up the dependency files for 10,339+ and 7,320+ in my local file system at the department, and transferring back to the cluster by using scp takes a lot of time, nearly half an hour for 13 GB. But, unfortunately I haven't backed up the dependency file for 2,1778L before itself. So, should I have to re-run the linear algebra for 2,1778L starting up right from the scratch?

Why do you do all the latest modifications to msieve and then spoil up the previous code? I wish that I had written my own code to be devoid of these errors, being dependent upon others, but understanding the algorithm is too difficult, especially the notations given within the papers, so much optimizations needed... I don't have that patience for writing 1 man year of code at all...