Yafu

[bsquared]

Originally Posted by BudgieJane

Another odd factorization:

Thanks, I'll look into it.

[LaurV]

Originally Posted by bsquared

As for making yafu always run siqs, just delete (or move) the gnfs-lasieve*.exe files. Yafu (assuming you are using the latest version) will revert to siqs if it can't find the lattice sievers.

Thanks, this did the trick, I renamed "ggnfs" into "ggnfs_" on the path. Siqs is now crunching on aliquotes, as intended.

[bsquared]

Originally Posted by LaurV

Done. I used a text viewer, copy, then paste into text editor (programmers' notepad 2), it could be that the line ends were modified from LF to CRLF, but as I found before, this has no influence. The nfs64.dat files contain relations from the beginning of the criminal file, and respectively from the end of it. Please rename them accordingly, if need (delete 64xxx from the names). The nfs32.dat contains a copy/paste from a job in progress, same polinomial, but moved with all the house onto a winXP32 computer, and using yafu32 (already 150megs of relations, so the progress is about 1/8 from the size of the criminal file). They look quite differently, the last number on each line is a magnitude bigger, and as I saw by reading mseive source pointed by jason, the -11 error is somehow related to the size of the numbers. He may not like the higher-bits integers, or so?

I've verified that msieve doesn't like the relations from your 64 bit run. I also noticed that the 64 bit run was finding relations with algebraic side special-q and the 32 bit run was finding rational side special-q. By itself, this is perfectly fine, but together with the fact that the 64 bit version's relations are bad leads me to suspect that they came from a different factorization. It's difficult to know what's going on without being in your shoes.

One thing I'll point out is that there is a difference between -r and -R, and in general -r should not be used unless you know exactly what you're doing. (-r means to use rational side special-q while -R means to restart a factorization). Not sure if this is the cause of your problem, but wouldn't hurt to check that you're using -R and not -r (GNFS problems tend to work best with the default algebraic side special-q).

Oh, and the -11 error from msieve has nothing to do with the size of the numbers in the file. It means that the rational side factors of the relation do not multiply together to produce the polynomial value at the lattice coordinates specified.

[bsquared]

Originally Posted by BudgieJane

Another odd factorization:

Originally Posted by bsquared

Thanks, I'll look into it.

Rare case where the number was small enough to be passed to squfof, but squfof failed. I assumed (badly) that squfof would always succeed - should be fixed now. Not sure when I'll have things together enough to post the fix.

PS S:\buhrow\yafu\test> .\yafu-32k-Win32.exe "factor(487415210250370733)"

factoring 487415210250370733
using pretesting plan: normal
div: primes less than 10000
fmt: 1000000 iterations
rho: x^2 + 3, starting 1000 iterations on C18
rho: x^2 + 2, starting 1000 iterations on C18
rho: x^2 + 1, starting 1000 iterations on C18
b = 1
Total factoring time = 0.1050 seconds

***factors found***
PRP9 = 837532169
PRP9 = 581965957
ans = 1

[Andi47]

Originally Posted by bsquared

One thing I'll point out is that there is a difference between -r and -R, and in general -r should not be used unless you know exactly what you're doing. (-r means to use rational side special-q while -R means to restart a factorization). Not sure if this is the cause of your problem, but wouldn't hurt to check that you're using -R and not -r (GNFS problems tend to work best with the default algebraic side special-q).

-R does resume a previously interrupted factorization, not restart it.

[bsquared]

Originally Posted by Andi47

-R does resume a previously interrupted factorization, not restart it.

Resume, yes. Thanks.

[LaurV]

Originally Posted by bsquared

It's difficult to know what's going on without being in your shoes.
One thing I'll point out is that there is a difference between -r and -R

This could be a reason... I have some batch file where -r could be used.

I am fully aware of the difference between -r and -R, we discussed it before (remember the discussion about my fibo C185, I am still working on it, fortunately this is on XP32 computer and I have no reason to stop the sieving process, I will let it finish at least till it will reach the first filtering trial, to see what's going on).

I know what -r is doing, in spite of the fact that I don't fully understand the difference between the algebraic side and rational side relations. The only time I voluntarily used -r is for that C185, at your suggestion. But the fact is: that time I created some batch files to call yafu/msieve/ggnfs with different parameters, and there is quite possible that I have -r switch in one of these .bat files, which I used for trying to factor the C104 mentioned above. I will check when I return home this evening. Somehow I doubt that the cause is this (as I said, it didn't work for trials of rsa(285) neither), but I will try to play with it. I believe the real reason is related to Vista64. Note that the "tune" command detects it like win32, in spite of the fact the OS is 64 bits and I used 64 bits yafu. For now, renaming the ggnfs folders (so it won't be accessible by yafu) as you suggested, did the trick of switching to siqs always, and I am happy with this setup for that vista laptop.

Thanks a lot for looking into it. I will let you know the results.

[bsquared]

Originally Posted by LaurV

I am fully aware of the difference between -r and -R, we discussed it before ...

Sorry, I'd forgotten about that. I was just surprised to see rational side relations in the file.

There are 32 bit and 64 bit ggnfs lattice sievers - do you know which you are using on the Vista system? Try switching to the other set and see if that helps. Just run a small test once the new gnfs-lasieve* files are in place, e.g. (assuming you have a polynomial for some number)

.\yafu-32k-x64.exe "nfs(<some number>)" -ns 100000,105000

followed by

.\yafu-32k-x64.exe "nfs(<some number>)" -nc

[LaurV]

I think I found the problem, because was happening again. It was indeed related to the nfs.dat relations comming from another polinomial. In some circumstances the nfs.dat file won't be deleted, but the polinomial will. Subsequent restarts (with or without -R switch) will add relations to the old nfs.dat. For example, yafu was started from aliqueit with -y switch, started to nfs. Then I aborted (with ctrl+c, wait to gracefully shut down) aliqueit (and yafu of course) because I already had a factor for that number (from different source) and it was making no sense to continue. I added the factor to factordb, I redownloaded the elf file, started aliqueit again. Next term was also candidate for nfs, and when I looked after few minutes to nfs.dat, it was already 700 megs, impossible to produce so many relations in so short time. So I stopped all and try filtering. Of course, same error, plenty of "error -11" in logs. Then looked into the nfs.dat file and found a strange mismatch of the specialq somewhere at about 90% of the file size:

3145449,33326:52afef,7ad791,F2501,935,3,161:bdb6dd,3dda143,7F709,228485,5,71,1DF,26B,313,D,2,2,2,2,2,2,2A0D9D
43920671,36278:2b61503,4045,51131,445,11,49,15D:4bd6f3,1e6bf81,1639,59E9,81A7,4F2CD,3,3,5,B,13,2F,20B,2,2,2,2A0D9D
-37652801,33442:11c58b5,DCC7,4148B,64D,527,B5:664475,df16dd,4613,AE8B,F3127,16171F,3,5,AD,2,2,2,2,2,2,2,2,2,2,2,2A0D9D
1192636037,14256:4def37,1014427,144D,8CE63,B3:905267,47d979,727F,2C405,2C5BD,A3D,5,7,7,B,17,29,2DD,3,2,2,2,2,2A0D9D
-978305603,7464:4afa95,1C3D,1689D,15353F,2F,71:1cbe423,A81D,635C5,6AD17,164599,5,17,35,53,191,3,3,3,2,2,2,2,2,2,2,2A0D9D
-734941207,20464:3509e5b,e35735,2CBD23,3B,1CF:3df2bf3,238F,2A8B7,9811F,BFB,7CD,3,3,3,5,5,5,5,7,D,11,BF,17B,2,2,2,2,2A0D9D
-34166367,125:46487f,529ED,1D55A7,3D7,397,35,2:CF7,184D,15AE1,58B41,469,43F,3,3,5,11,11,9D,ECD2F
-36936187,3:1b355d5,1093,21997,42D,6B,71,2,2,2,2,2:8a741d,119dd1,1A231,3,3,3,3,5,7,7,11,65,13,ECD2F
-119919825,451:f61669,DAB,A0123,A9AED,1EB,29,2:4186b1,3127,194C3,47F2F,93721,4FF,3,3,5,5,ECD2F
-114208969,145:5F65,126BFB,145A6B,6D3,20B,C1,3,2,2,2,2:57b62d,9caf55,C5B,C42DD,17F,5,11,13,ECD2F
108257401,481:12f7a0b,1A2F,33CB,BCB9,79691,D,2:291ed5,18EB7,2959D,30C83,55EAF,3,3,3,5,B,8B,ECD2F
112216339,1021:8d88c9,18C7,2C23,27883,71F,5D5,2,2,2:be155b,6FBF,103BB,8B5CB,4CD,FB,3,3,7,11,2F,3B,ECD2F

As I guessed by converting the values from hex to dec, the last value is the specialq. I used the "split" command of total commander (very fast!) to split the file in 500 parts (it was too huge to be possible to edit with any editor!) and I deleted all parts with 2xxxxx at the end, I edited the file where relations were mixed, keeping only last part of the file, and also keeping all the files coming after that. Then used "combine" command of totalcmd.

As I started filtering again, it corectly found 3.5M relations, and everything worked perfectly up to the end, with no error.

nfs: found 3590694 relations, continuing job at specialq = 1209973
nfs: commencing algebraic side lattice sieving over range: 1249973 - 1289973
nfs: commencing algebraic side lattice sieving over range: 1209973 - 1249973
 Warning:  lowering FB_bound to 1209972.
 Warning:  lowering FB_bound to 1249972.
total yield: 621151, q=1249999 (0.00193 sec/rel)
total yield: 602086, q=1290013 (0.00200 sec/rel)

Now, why yafu won't delete that file in some circumstances, is over my head. Does the process of calling it from aliqueit has any influence in this? Or it is a yafu bug? I could reconstruct it on XP32 too. It has nothing to do with ggnfs or nfs algorithm itself. What a pitty I don't have the original nfs.dat file that made all the discussion above (as I said here, that number was a brilliant, factored also by yafu nfs, and splitted nicely into 2 prp53's) because I would like to have a look inside for this anomaly. Most probably, that file came from two different polys too.

[bsquared]

Originally Posted by LaurV

I think I found the problem...

Yep, that sequence of events can cause the problem. Although note that ctrl-c by itself never deletes anything. On the contrary, the purpose of ctrl-c is to *preserve* the state of the factorization so that it can be picked up later. If the user changes his/her mind before restarting, the assumption was that the user would be responsible for cleaning things up before starting new work. The only time yafu ever deletes files is on completion of the job.

The basic root cause of this issue is the design philosophy of over-cautiousness when talking about the automated deletion of large data files. That's why things like "-R" are in place, to place some level of decision making on the user who is much smarter than yafu in determining the value of a given pile of memory sectors .

So, to make sure we are on the same page, we're talking about instituting this change:

If yafu is told to work on a number, N, that is different from one or more existing files based on a different number, to go ahead and blow away all existing files before starting work on N. One alternative would be to refuse to do this and notify the user. Another alternative would be to look for a flag for permission - maybe something like "-F" for "Force new start", or something.

Any other thoughts/suggestions/ideas?

[bsquared]

For those that build yafu from source, SVN now contains a fix for the bug that BudgieJane reported.

There is also a new (minor) feature: yafu can now resume NFS in the polynomial selection phase in addition to the sieving phase. So a ctrl-c in the middle of polyselect will write information to the .p file such that on resume, polyselect will be picked up where it left off and time spent in previous runs will be accounted for.

I'm in the middle of converting a large portion of the code base to use GMP, instead of my homegrown AP library, so I don't want to do a new version release just yet. I don't think anything is broken, but it needs more testing first. Just an FYI for anyone building from source...