After wading through a large number of posts, I'm thinking it may be time to simply ask questions rather than attempting to deduce the answers.

1. When building MSieve, the make instructions say to use ECM= if ECM is available. I assume this is GmpEcm? When I make the libecm.a available, it does build msieve.exe but it stackdumps when a factoring is attempted. What is intended by this ECM= option? Does it require GmpEcm? Does it require a specific version of it? Are there runtime requirements? What is the advantage (disadvantage?) to using ECM?

2. Is it possible to hook MSieve into the factLat.pl GGNFS script?

3. Is it possible for MSieve to use GGNFS produced relations? If yes, would this be rels.bin files produced by procrels, or would it be the relationships produced by the siever before procrels massaging?

4. Is MSieve far enough along to where it can replace GGNFS? Or should GGNFS be used for now?

5. Documentation is thin for non-math types. How is an MSieve poly file constructed for SNFS? Can a GGNFS produced poly file be used by MSieve?

6. What are the plans (if any) for merging the abilities of MSieve with GGNFS? Is there an estimate on when this is to be done (ready for usage)?

1. You need a current version of GMP ECM, a current version of GMP (the same used to build GMP ECM; while this should be obvious, even my own system had a pre-existing GMP library that was not compatible with a locally compiled GMP), and have to add ECM=1 to the build line for msieve.

2,3,4,6. Msieve's postprocessing is much faster and more robust than what is in GGNFS. Many people use it to finish jobs started by the GGNFS tools. It uses the same text relation format as GGNFS (not the binary format in the rels.bin files). However, msieve is not ready to replace GGNFS, and probably never will. The polynomial selection and lattice sieving there are incomparably better than what's in msieve, so it makes the most sense to use both tools. Greg Childers (frmky) has said he will try to integrate msieve into the factLat perl script sometime soon. In the meantime, the msieve code has been added to the GGNFS codebase, it just isn't in any of the binary snapshots (those are all pretty old).

5. Look in the GGNFS and XYYXF yahoo groups, several people have posted detailed instructions and perl scripts for converting a GGNFS polynomial file to an msieve factor base file. Greg has also posted instructions in one of the threads here. Finally, the msieve source has several readme files with background references and detailed information on running the quadratic sieve part. The number field sieve documentation of course needs some work.

Thank you for the information.

It appears that I need to be patient awhile longer until things are a bit more integrated. While the pieces appear to be in place to use a mix of tools, it is still a bit too raw for "fire and forget". Consequently I think my best bet is to continue using the GGNFS system for now and continue to monitor the GGNFS Yahoo group for merging details.

Once the MSieve and GGNFS tools are integrated, I'll be very happy. It seems as though a lot of people are bringing some very nice abilities to the table.

Bug in msieve-1.32 batch sieving
 

I have about two million linesieved relations on 6,383+, so I tried running msieve -v -nc to check that everything was working well.

Sadly, it doesn't seem to be; I get 'error -15' messages on a large proportion of the batch-factored relations, which I think mean 'omitted factor'.

An example that I know fails is

[code]
-496019644,1:5217b,298e77,c9223a9,61583605:35f,4e05,288271,7376b7,3e186b1,1b222a99,f406e4f
[/code]

[code]
R0=-7253635851193924156735160443739
R1= 2391424041494417171
A0= 547440910672314203689898814059115360
A1= 33277562211750204806364306268284
A2= 107677876784557388243547221
A3=-2612363701552248486716
A4=-3795305047120954
A5= 11472718320

g(y,x) = R0*x+R1*y
f(y,x) = A0*x^5 + A1*x^4*y + A2*x^3*y^2 + A3*x^2*y^3 + A4*x*y^4 + A5*y^5
factor(f(-496019644,1))
factor(g(-496019644,1))
[/code]

For this point, the rational side has factors 0x17, 0x5217b, 0x298e77, 0xc9223a9, 0x61583605 as claimed.

The algebraic side, however, has factors
[code]
? factor(f(-496019644,1))
%21 =
[-1 1]
[2 5]
[3 2]
[5 4]
[41 1]
[863 1]
[19973 1]
[2654833 1]
[7567031 1]
[65111729 1]
[455223961 1]
[4550848079 1]
[/code]

or, in hex,
[code]
29, 35f, 4e05, 288271, 7376B7, 3E186B1, 1B222A99, 10F406E4F
[/code]
versus the claimed
[code]
35f,4e05,288271,7376b7,3e186b1,1b222a99,f406e4f
[/code]

IE you've found a factor of more than 32 bits, and somehow truncated it to 32 bits. I am not certain whether I did this sieve run with a 32-bit or 64-bit binary.

This isn't a serious problem in this case - it's affecting about 5% of a sample of a million relations that I tried it on, throwing those away is not a great cost - but it would be good to fix it; it makes me paranoid about the consequences of things like composite large primes.

[QUOTE=fivemack;122405]
IE you've found a factor of more than 32 bits, and somehow truncated it to 32 bits. I am not certain whether I did this sieve run with a 32-bit or 64-bit binary.
[/QUOTE]
Grrr, I think I didn't see this before because this example uses 31-bit large primes and I'd never tried large primes that large. Unless there's another bug in the sieving, relations should not have composite factors.

Fixing it is easy, thanks for noticing.

Hi Jason,

I'd have a request about msieve. I do the postprocessing of ggnfs jobs with msieve and I often have to concatenate some spairs.out files to a large msieve.dat file.

It has happened many times that, due to a wrong command line, or because msieve.dat.line is missing msieve does erase my .dat file and start with a 0 byte new one. It is frustrating to restart the process of ungzip and concatenation of Gb files...:cry:

Is it possible for msieve to check if a large msieve.dat file exists when it starts and to ask the user if it should erase it ?

Thanks and regards.

Philippe

Oops ...
 

Compression algorithms, by design, are things for turning a corrupted file into plausible nonsense.

Not, however, all [B]that[/B] plausible:
(from Andi47's sieving of 6^383+1)
[code]
-130296002188863:53401,37380e015dd630cf,623614269237f2f17:5,d,4f,dd3,fbb,4318e7,1e1bec3,1f3267f,15b66435,3e22913f32d,8276f1723ab
46705201829,197:87b9675,89c599f6d55d27c8c518f3631ddf1772:3,3,3,3,72b91fb6d,79f5f1723ab,2d024b1d7eb6d2f23,39c501d362df481014aec57120bd5f
38898392423844611:89f,4e057f3d,47ed2b80f,13c43a9d5acd1fb77e1:3,7,f23,280385bb3,11b3526b1f1723ab,58fc19cf8e45eb93,b336a5211e6ff121f9d5ffd
329600953d9843007:fb2,e004a6d972fd,c03b426d23b753275121273:5db,459ee3,db2996b1f1723ab,8daf331fe2f0d936023,43f874238a6d52fe69521813eb
24852535181744133:9,53ebddb1,2ccf81de6c,5fb589f1f1f,1f96f19949b29:3,3,1b08ebd23,723b2a79b3,5812f1723ab,348e94c523bad,433b4921451b51715a139d
-4005223869842465:1b296915,3e5d7cddf,4c47a258f71c92ca23b5:3,3,7,1af8f,0322f34f,f1723ab,16e4be0f2f58f97e53,1b032f323523f72ff1c9cd
-45169653783,37549:2df1f,7f141bb977,6efb1dd4c62354003ed940dd53:5,777,4230c34d,4fd66532f1723ab,1afbc255c4459886087b1746b72166d972d47cff6023
-987564622AD62433:f373af352763426f24f9a92ff1ce121f2472e15f1f:5,7,3fb,dd3,32dd32,ff21c1,bc15f7d,f1723ab,5095f923,c8768b988654af2280c7f
525B6F28559,111291:3,2df,49e123796d,437299492277,12caeac7431617d15:e8f59af,2a7fd5151d3f,136a8d1f76ee3,6c080ed36264b67abb,360202022652f1723ab
C0687242631345:53b43713f3,17b211958656e7d5216f2d11eabb:fb33,4ddf79f1f1f1723ab,3779599331ed19f87f1994864629bae0fff79f25
-200014848857037:3f,7cc9acb,259f5729,b0945be47e2bd13bdb5d:3,3,f2f23,331a7f6b,b8a5d10521d,2886492558f1fb1c49b73,7c3251f1f1f189f1f1f1723ab
2477627E91,0:58690c773,1cfafbd23b1381,2c5232034afecad1f:6b8b2156d1efef5d,d2437527e5121e3a31d92a7,8632ff0946a51e9c2df2d8fd1aad0123b23
-192052844,8705:de92f,50021f,519f29,1e0059588bd2812f71:b,365,dd3,50923,f2f2f23,367e524f2649b69d9,89f1f1f1f1f1723ab,b55769b0493b9131aa7
[/code]

I was briefly amazed that the decompression mostly got the colons in the right places, but in fact that's the fault of my code that reads input files and sorts the primes.

Sadly, on attempting to read the line
[code]
2477627E91,0:58690c773,1cfafbd23b1381,2c5232034afecad1f:6b8b2156d1efef5d,d2437527e5121e3a31d92a7,8632ff0946a51e9c2df2d8fd1aad0123b23
[/code]
msieve-1.32 segfaulted; poly_get_zeros was called with p=0. Given line 87 of the extract below, this surprised me slightly.

The explanation is that '2477627E91' is a valid floating-point number 2.477627*10^97, so strtod returns that rather large double; casting this to an int64 gives -2^63, and casting that to a uint32 in the section
[code]
87 if (a == 0)
88 return -2;
89
90 num_roots = poly_get_zeros(roots, &fb->rfb.poly,
91 (uint32)a, &high_coeff, 0);
[/code]

obviously leaves p=0.

It's a fairly obscure case, but maybe reading the decimal sections of relations with an explicit decimal reader might be worth it

[QUOTE=fivemack;122614]
It's a fairly obscure case, but maybe reading the decimal sections of relations with an explicit decimal reader might be worth it[/QUOTE]
I'll add a patch for this; my preference is to use strtoll(), but this is a C99 function that win32 doesn't have.

Standing by with the crash cart for the postprocessing...

[QUOTE=Phil MjX;122597]
Is it possible for msieve to check if a large msieve.dat file exists when it starts and to ask the user if it should erase it ?
[/QUOTE]
The savefile is checked to determine whether new relations should be appended to the existing list or should replace the existing list (i.e. you have a new factorization writing to an old .dat file). If you are not sieving, then the check doesn't need to delete the file. I'll update the source. Enough bug fixes have accumulated that I'll be pushing out another release this weekend.

[QUOTE=jasonp;122634]Standing by with the crash cart for the postprocessing...[/QUOTE]

You can stand down the crash cart for a few days; I need to wait for a new motherboard to arrive (probably Monday) and rebuild the computer so I can use all my memory. The 3GB I can use at the moment is not enough even to do singleton-removal on a >200-megarelation lp=2^31 problem like 2,841- or 6,383+ !

[QUOTE=fivemack;122614]Compression algorithms, by design, are things for turning a corrupted file into plausible nonsense.

Not, however, all [B]that[/B] plausible:
(from Andi47's sieving of 6^383+1)
<snip>
Sadly, on attempting to read the line
[code]
2477627E91,0:58690c773,1cfafbd23b1381,2c5232034afecad1f:6b8b2156d1efef5d,d2437527e5121e3a31d92a7,8632ff0946a51e9c2df2d8fd1aad0123b23
[/code]
msieve-1.32 segfaulted; poly_get_zeros was called with p=0. Given line 87 of the extract below, this surprised me slightly.
[/QUOTE]

I just did a check with my original file - no segfaults, thus indicating that the original file is OK and .zip compressing somehow corrupts the file.

[QUOTE=fivemack;122638]You can stand down the crash cart for a few days; I need to wait for a new motherboard to arrive (probably Monday) and rebuild the computer so I can use all my memory. The 3GB I can use at the moment is not enough even to do singleton-removal on a >200-megarelation lp=2^31 problem like 2,841- or 6,383+ ![/QUOTE]

When do you want to start start postprocessing? Early at the beginning of next week, or do you want to wait until all ranges are done and uploaded / snail-mailed?

I just uploaded a BZIP2'ed test file - if this one works, I will upload my relations in BZIP2 format, else I will send a DVD via snail mail around wednesday next week - after 135M-137M is finished.