Filtering

[R.D. Silverman]

The CWI filtering code does an excellent job of filtering data and producing small matrices.

When the operating system lets it.

I continue to have problems with filtering under Windoze.
The filter code sometimes want to grab more memory and
does a realloc(). However, this realloc() frequently fails because
Windoze insists that that whatever memory is requested is contiguous.


I now have a 64-bit version of XP running on my laptop. I have
compiled the filter code with LARGEADDRESSAWARE turned on. My
system has 4GB of real memory. And the filter code still has
realloc() failures.

This is very frustrating when you set up a filter pass, it runs for 7
hours and then fails. You have to reset the filter parameters and
try again. It is one royal pain in the gluteus medimus.

[Andi47]

Originally Posted by R.D. Silverman

It is one royal pain in the gluteus medimus.

I thought, the "ass muscle" is named "gluteus medius" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).

[R.D. Silverman]

Originally Posted by Andi47

I thought, the "ass muscle" is named "gluteus medius" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).

I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.

[Andi47]

Originally Posted by R.D. Silverman

I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.

If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* download msieve.exe
* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1  3484803096777727
A0  68619560356658365509867543692766345
A1  1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4  12133739733
A5  9180

* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

msieve -p -v -nc -t <number of threads>

whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.

[R.D. Silverman]

Originally Posted by Andi47

If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* download msieve.exe
* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1  3484803096777727
A0  68619560356658365509867543692766345
A1  1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4  12133739733
A5  9180

* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

msieve -p -v -nc -t <number of threads>

whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.

Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.

My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??

[axn]

Originally Posted by R.D. Silverman

Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.

Not needed for the dat file. msieve will work fine without it.

[jasonp]

The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.

It's somewhat convoluted, and maybe I should change it because nobody batch factors multiple NFS jobs using just msieve. But the full library, including the QS code, does run for each number in the ini file in turn, and I've occaisionally used batch factoring that didn't involve NFS.

[axn]

Originally Posted by R.D. Silverman

My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??

a,b:rational side factors:algebraic side factors

a,b in decimal. factors in hex (case insensitive), comma separated. you may leave out small factors < 1000.

[Andi47]

Originally Posted by jasonp

The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.

When N is not specified in the .dat file, I have seen msieve (not sure which version) giving an error "error -xy in relation 0" and continuing anyway, but I wouldn't rely on it.

BTW: @JasonP: some time ago I accidently copied an old relations file into the .dat file and got gazillions of relation errors. So: What about stopping with a message like "too many relation errors - are you sure you have copied the right relations file?" if hundreds of relation errors in a row are popping out - instead (or additional to) checking if N is in the first line of the .dat file? (so there would be no need to add it to a multi-GB-file which can be pain in the gluteus maximus...)

Note: how to add N to the .dat file in windows:
1.) rename the relations file to 1.out
2.) create a file named 0.out (so that it is prior to the relns file in alphabetic sorting!!) which contains the line: "N <your number>". Note: This will also work if you want to concatenate multiple relation files, e.g. named 1.out, 2.out, xy.out, etc...
3.) make sure that 0.out and your <relations>.out files are the only files in the directory with the extension .out
4.) in the command line type "copy *.out msieve.dat"
5.) go for a coffee - depending of the size of your relation file(s), it might take some minutes, but it works even for multi-GB-relation-files.
6.) done.

[jasonp]

Not sure I would want to prematurely abort reading relations in...what if you are concatenating hundreds of files and one of them is from a previous factorization? You get a block of relations that will never work but hundreds of millions that do.

Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other. Admittedly, in an era where everyone supports huge files then being forced to build a huge file is a small inconvenience.

[axn]

Originally Posted by jasonp

Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other.

This feature would be much appreciated. Read the files from a file list and use the non-existent dat file (-s option) as a scratch pad (for example, free relations).

EDIT:- File copying with AV scanners running is not a pretty thing (esp. under Windoze)