Raman's plans and questions - Page 17

henryzz · 2010-01-12, 14:10

Quote:

Originally Posted by Raman

Are you really sure enough? I have been actually stunned up by this post only.
I was keenly expecting up for the factors only yesterday morning itself.

--

The rural children of India, do not have as much as primary education as urban children of India
The urban people of India, do not have as much as advanced education as those people of USA

(The rural children of India face so much difficulties even with simple primary school level Mathematics)
(I find so much difficulty in understanding or implementing the advanced algorithms with so much of optimizations)

Hmm..... i thought that bug had been solved.
Maybe worth going back to an older version.

jasonp · 2010-01-12, 14:14

Quote:

Originally Posted by Raman

What do you get the md5sum with your msieve-1.43, 64 bit Linux binary?
357bc5e227119d001cf76d01c1987561 - compute cluster
acf42a2eb06c8e9f8cbf2890921f2d1c - department computer
cbd2952dda80dd67dff846b3b01bce19 - 32 bit Linux binary compiled on department

I assume that the relation files remain unchanged, and work with both binaries of msieve, right? It is not possible to run md5sum on relation or cycle files. They are too huge enough. But the same files copied should have same md5sum only. [B]Will the relation file work with both copies of msieve?

I expect compiled binaries to be different from each other pretty much every time, and don't care about that. Do not assume the computer always works perfectly. Run md5sum on the relation file, cycle file and dependency file; in a previous job we would run this utility on files the size of entire hard drives. The square root has many many integrity checks on everything that goes in, because it is the last chance available to verify that all the inputs from previous stages are valid. If the square root complains about anything, something is inconsistent about the data you gave msieve and you have to figure out what it is. Jeff Gilchrist has already performed a factorization that was spoiled because the files were moved to another machine and corrupted in transit.

You're a lot smarter than I was at age 22; learn some patience and you will be dangerous.

Raman · 2010-01-12, 14:57

Quote:

Originally Posted by henryzz

Hmm..... i thought that bug had been solved.
Maybe worth going back to an older version.

Shall I use msieve-1.38 for all filtering, linear algebra, square root stages
for 2,1778L 7,320+ 10,339+ 10,351+ compiling it up under the compute cluster machine?

msieve-1.42 has the filtering crash; msieve-1.43 has the square root crash.

I am pretty sure that the relation file will be unchanged because it is entirely the same
copy from the other machine, transferred by using the scp command, exactly the
same file size, date of modification, time of modification, etc.

The relation file is produced directly from GGNFS, it is a data file and should be unchanged.

Quote:

Originally Posted by jasonp

You're a lot smarter than I was at age 22; learn some patience and you will be dangerous.

May I know at what age you started up to read and then implement about the factorization
algorithms? I know that you were a member of team for proving that F24 is composite, ten
years ago, before itself only... I assume that right now you are just 26 years old simply...

Msieve version 1.38 is ok?
I will have to get up this archive from my home. Can only start up executing tomorrow morning.

Raman · 2010-01-12, 16:11

Quote:

Originally Posted by Raman

I assume that right now you are just 26 years old simply...

It should be 36 years... Before I made the correction, the editing timer has expired up.

If I run the entire post processing for all these 4 numbers
The good msieve version should not have any crash within the square root phase. The linear algebra will take 4 days to run, so I am enquiring of that, safely before itself.

Since the computations will take place only within an internal node within the compute cluster, until the task finishes off, I don't know the output or anything of what is happening inside at all.

Using msieve-1.42 for the square root corrupted up all the good files within the folder from which the task was started off from. No good result was being produced at all, the log was corrupted even.
@henryzz: The segmentation fault is independent of the binary, so it does not matter due to the binary transferred from the other computer. For your information, the linear algebra worked out successfully from the binary that was being transferred from the old computer. Though the dependency files said everytime that "Algebraic side is not a square!" even with the 64 bit Linux binary that was being compiled up under that compute cluster machine. This is before the segmentation fault stage, so I don't know at all whether or not the crash will occur with the use of this new binary...

fivemack · 2010-01-12, 20:02

I think you may have got the cause and effect wrong there; it is wildly unlikely that a crash of msieve would corrupt the log file, and entirely likely that a problem with, say, the shared filesystem of the computer cluster would corrupt lots of files to the point that the square root fails.

I have often observed mysterious bugs during and after the linear-algebra stage which turn out to be because the disc is full.

It is worth doing md5sum at both ends when transferring a large file between computers of whose reliability you are not extremely confident - in theory scp does the equivalent of this, but md5sum doesn't take all that long and is guaranteed (more likely that you'd win the lottery six times in consecutive weeks than that it would miss a problem) to pick up any corruption.

I appreciate how frustrating this is; I have had initial intermittent faults which break a factorisation with the last three computers I've bought.

jasonp · 2010-01-12, 20:08

Quote:

Originally Posted by Raman

May I know at what age you started up to read and then implement about the factorization
algorithms? I know that you were a member of team for proving that F24 is composite, ten
years ago, before itself only... I assume that right now you are just 26 years old simply...

I started writing the code that would eventually become msieve in early 2003. Before that I helped proofread a draft of Crandall&Pomerance's 'Prime Numbers: A Computational Perspective' back around 2000, but did not seriously get into factoring until a lot later.

Raman · 2010-01-12, 20:18

So, shall I try out the entire post processing of the 4 numbers with
msieve 1.43 binary that is being compiled under the compute cluster itself
or some old, for example msieve 1.41 binary?
It shouldn't fail again this time, at all. If so, another 4 days get wasted up.

I had noticed that the capacity of the task executing (scratch) directory is only 63 GB, while for the files storage directory within the compute cluster, it was 3.1 TB. Is that low capacity of the file system the cause for the corrupt of the files?

jasonp · 2010-01-13, 01:45

Possibly, but you don't have to guess.

Code:

md5sum msieve.dat
md5sum msieve.dat.cyc
md5sum msieve.dat.dep

on both new and old machines. Don't assume, verify! If one of the files is different, just scp again and go straight to the square root, saving 4 days of machine time.

Raman · 2010-01-13, 04:36

All the relations, dependency, matrix, cycles file are exactly the same now by using md5sum, but that original dependency file that was produced by the linear algebra might have got corrupted up while file transfer (or due to lack of disk space after the task failed, while writing? - In this case avoiding segmentation fault during square root phase is the best). If it was corrupted up during the file transfer, if I had known before itself, I would certainly have run up md5sum upon the dependency file immediately after the file transfer was being completed up only. Now that original dependency file that was being produced up by the Linear Algebra is itself being corrupted up.

Thus, is it possible to revert back to the old revision of the dependency file by using subversion commands, etc.? How to do it up so?
Finally I get up that

svn status 7_320P.dat.dep
svn: warning: '7_320P.dat.dep' is not a working copy

jasonp · 2010-01-13, 04:59

Subversion in not an automatically versioned file system like VMS was; you have to add the file into subversion manually before you can get it back.

Raman · 2010-01-13, 08:10

Still, for 7,320+, it is only saying that

Code:

Wed Jan 13 10:25:09 2010  Msieve v. 1.43
Wed Jan 13 10:25:09 2010  random seeds: d2902e91 21339cda
Wed Jan 13 10:25:09 2010  factoring 154758218728434566356200475118402575823277636002771024732774611297531505856368662705022790238791621423554625142974233023501037614707598549788891266955273526560342732155180961801351291833302132131894401 (201 digits)
Wed Jan 13 10:25:12 2010  no P-1/P+1/ECM available, skipping
Wed Jan 13 10:25:12 2010  commencing number field sieve (201-digit input)
Wed Jan 13 10:25:12 2010  R0: -1219760487635835700138573862562971820755615294131238401
Wed Jan 13 10:25:12 2010  R1:  1
Wed Jan 13 10:25:12 2010  A0:  1
Wed Jan 13 10:25:12 2010  A1: -1
Wed Jan 13 10:25:12 2010  A2:  1
Wed Jan 13 10:25:12 2010  A3: -1
Wed Jan 13 10:25:12 2010  A4:  1
Wed Jan 13 10:25:12 2010  skew 1.00, size 5.963718e-22, alpha 1.694464, combined = 4.225816e-13
Wed Jan 13 10:25:12 2010  
Wed Jan 13 10:25:12 2010  commencing square root phase
Wed Jan 13 10:25:12 2010  reading relations for dependency 1
Wed Jan 13 10:25:14 2010  read 2895297 cycles
Wed Jan 13 10:25:28 2010  cycles contain 7794247 unique relations
Wed Jan 13 10:30:09 2010  read 7794247 relations
Wed Jan 13 10:30:10 2010  number of relations is not even
Wed Jan 13 10:30:10 2010  reading relations for dependency 2
Wed Jan 13 10:30:14 2010  read 2898146 cycles
Wed Jan 13 10:30:28 2010  cycles contain 7799198 unique relations
Wed Jan 13 10:33:56 2010  read 7799198 relations
Wed Jan 13 10:34:22 2010  algebraic side is not a square!
Wed Jan 13 10:34:22 2010  reading relations for dependency 3
Wed Jan 13 10:34:26 2010  read 2897249 cycles
Wed Jan 13 10:34:40 2010  cycles contain 7798000 unique relations
Wed Jan 13 10:37:17 2010  read 7798000 relations
Wed Jan 13 10:37:43 2010  algebraic side is not a square!
Wed Jan 13 10:37:43 2010  reading relations for dependency 4
Wed Jan 13 10:37:47 2010  read 2896141 cycles
Wed Jan 13 10:38:01 2010  cycles contain 7795365 unique relations

The dependency files are somehow being corrupted...
It is quite surprising. If the dependency files are corrupted then msieve should say that "Dependency file is corrupt" ("Unable to read dependency files" or "Dependency file is not valid"), right? How come is it possible that msieve reads the relations properly, but algebraic side is not a perfect square at all? How is it possible that wrong set of linearly independent relations are being picked up at the end of the linear algebra only? And then what is the reason behind the occurring of error "Algebraic side is not a square!"?

2010-01-12, 20:18	#183
Raman Noodles "Mr. Tuch" Dec 2007 Chennai, India 3·419 Posts	So, shall I try out the entire post processing of the 4 numbers with msieve 1.43 binary that is being compiled under the compute cluster itself or some old, for example msieve 1.41 binary? It shouldn't fail again this time, at all. If so, another 4 days get wasted up. I had noticed that the capacity of the task executing (scratch) directory is only 63 GB, while for the files storage directory within the compute cluster, it was 3.1 TB. Is that low capacity of the file system the cause for the corrupt of the files? Last fiddled with by Raman on 2010-01-12 at 20:19

2010-01-13, 01:45	#184
jasonp Tribal Bullet Oct 2004 3·1,181 Posts	Possibly, but you don't have to guess. Code: md5sum msieve.dat md5sum msieve.dat.cyc md5sum msieve.dat.dep on both new and old machines. Don't assume, verify! If one of the files is different, just scp again and go straight to the square root, saving 4 days of machine time. Last fiddled with by jasonp on 2010-01-13 at 01:49

2010-01-13, 04:36	#185
Raman Noodles "Mr. Tuch" Dec 2007 Chennai, India 3×419 Posts	All the relations, dependency, matrix, cycles file are exactly the same now by using md5sum, but that original dependency file that was produced by the linear algebra might have got corrupted up while file transfer (or due to lack of disk space after the task failed, while writing? - In this case avoiding segmentation fault during square root phase is the best). If it was corrupted up during the file transfer, if I had known before itself, I would certainly have run up md5sum upon the dependency file immediately after the file transfer was being completed up only. Now that original dependency file that was being produced up by the Linear Algebra is itself being corrupted up. Thus, is it possible to revert back to the old revision of the dependency file by using subversion commands, etc.? How to do it up so? Finally I get up that svn status 7_320P.dat.dep svn: warning: '7_320P.dat.dep' is not a working copy Last fiddled with by Raman on 2010-01-13 at 04:41

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What are your CRUS plans?	rogue	Conjectures 'R Us	35	2013-11-09 09:03
Raman's stuff	Raman	Chess	8	2013-04-16 20:52
Further Plans	Kosmaj	Riesel Prime Search	6	2009-05-20 01:27
Further Plans	Kosmaj	Riesel Prime Search	6	2006-09-29 22:32
64 bit plans	pyrodave	Software	17	2004-06-05 12:27

2010-01-12, 20:02	#181
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 1916₁₆ Posts	I think you may have got the cause and effect wrong there; it is wildly unlikely that a crash of msieve would corrupt the log file, and entirely likely that a problem with, say, the shared filesystem of the computer cluster would corrupt lots of files to the point that the square root fails. I have often observed mysterious bugs during and after the linear-algebra stage which turn out to be because the disc is full. It is worth doing md5sum at both ends when transferring a large file between computers of whose reliability you are not extremely confident - in theory scp does the equivalent of this, but md5sum doesn't take all that long and is guaranteed (more likely that you'd win the lottery six times in consecutive weeks than that it would miss a problem) to pick up any corruption. I appreciate how frustrating this is; I have had initial intermittent faults which break a factorisation with the last three computers I've bought.

2010-01-13, 04:59	#186
jasonp Tribal Bullet Oct 2004 6727₈ Posts	Subversion in not an automatically versioned file system like VMS was; you have to add the file into subversion manually before you can get it back.