Filtering
 

The CWI filtering code does an excellent job of filtering data and producing small matrices.

When the operating system lets it.

I continue to have problems with filtering under Windoze.
The filter code sometimes want to grab more memory and
does a realloc(). However, this realloc() frequently fails because
Windoze insists that that whatever memory is requested is contiguous.


I now have a 64-bit version of XP running on my laptop. I have
compiled the filter code with LARGEADDRESSAWARE turned on. My
system has 4GB of real memory. And the filter code still has
realloc() failures.

This is very frustrating when you set up a filter pass, it runs for 7
hours and then fails. You have to reset the filter parameters and
try again. It is one royal pain in the gluteus medimus.

[QUOTE=R.D. Silverman;223988] It is one royal pain in the gluteus medimus.[/QUOTE]

I thought, the "ass muscle" is named "[URL="http://en.wikipedia.org/wiki/Gluteus_medius_muscle"]gluteus [B]medius[/B][/URL]" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).

[QUOTE=Andi47;223991]I thought, the "ass muscle" is named "[URL="http://en.wikipedia.org/wiki/Gluteus_medius_muscle"]gluteus [B]medius[/B][/URL]" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).[/QUOTE]

I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.

[QUOTE=R.D. Silverman;224000]I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.[/QUOTE]

If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* [URL="http://sourceforge.net/projects/msieve/files/msieve/Msieve%20v1.46/msieve146.exe/download"]download[/URL] msieve.exe
* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

[CODE]N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1 3484803096777727
A0 68619560356658365509867543692766345
A1 1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4 12133739733
A5 9180[/CODE]
* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

[CODE]msieve -p -v -nc -t <number of threads>[/CODE]

whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.

[QUOTE=Andi47;224019]If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* [URL="http://sourceforge.net/projects/msieve/files/msieve/Msieve%20v1.46/msieve146.exe/download"]download[/URL] msieve.exe
* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

[CODE]N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1 3484803096777727
A0 68619560356658365509867543692766345
A1 1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4 12133739733
A5 9180[/CODE]
* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

[CODE]msieve -p -v -nc -t <number of threads>[/CODE]

whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.[/QUOTE]

Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.

My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??

[QUOTE=R.D. Silverman;224035]Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.[/QUOTE]
Not needed for the dat file. msieve will work fine without it.

The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.

It's somewhat convoluted, and maybe I should change it because nobody batch factors multiple NFS jobs using just msieve. But the full library, including the QS code, does run for each number in the ini file in turn, and I've occaisionally used batch factoring that didn't involve NFS.

[QUOTE=R.D. Silverman;224035]My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??[/QUOTE]

a,b[B]:[/B]rational side factors[B]:[/B]algebraic side factors

a,b in decimal. factors in hex (case insensitive), comma separated. you may leave out small factors < 1000.

[QUOTE=jasonp;224040]The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.
[/QUOTE]

When N is not specified in the .dat file, I have seen msieve (not sure which version) giving an error "error -xy in relation 0" and continuing anyway, but I wouldn't rely on it.

BTW: @JasonP: some time ago I accidently copied an old relations file into the .dat file and got gazillions of relation errors. So: What about stopping with a message like "too many relation errors - are you sure you have copied the right relations file?" if hundreds of relation errors in a row are popping out - instead (or additional to) checking if N is in the first line of the .dat file? (so there would be no need to add it to a multi-GB-file which can be pain in the gluteus maximus...)

Note: how to add N to the .dat file in windows:
1.) rename the relations file to 1.out
2.) create a file named 0.out (so that it is prior to the relns file in alphabetic sorting!!) which contains the line: "N <your number>". Note: This will also work if you want to concatenate multiple relation files, e.g. named 1.out, 2.out, xy.out, etc...
3.) make sure that 0.out and your <relations>.out files are the only files in the directory with the extension .out
4.) in the command line type "copy *.out msieve.dat"
5.) go for a coffee - depending of the size of your relation file(s), it might take some minutes, but it works even for multi-GB-relation-files.
6.) done.

Not sure I would want to prematurely abort reading relations in...what if you are concatenating hundreds of files and one of them is from a previous factorization? You get a block of relations that will never work but hundreds of millions that do.

Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other. Admittedly, in an era where everyone supports huge files then being forced to build a huge file is a small inconvenience.

[QUOTE=jasonp;224056]Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other.[/QUOTE]

This feature would be much appreciated. Read the files from a file list and use the non-existent dat file (-s option) as a scratch pad (for example, free relations).

EDIT:- File copying with AV scanners running is not a pretty thing (esp. under Windoze)

[QUOTE=Andi47;224051]4.) in the command line type "copy *.out msieve.dat"[/QUOTE]I've seen this go wrong. I always use "Copy /B" to make sure the dat file remains valid.

Also, instead of *.out I would use 0.out+1.out to make sure the files are in the right order.

Thus "Copy /B 0.out+1.out msieve.dat"

[QUOTE=smh;224079]I've seen this go wrong. I always use "Copy /B" to make sure the dat file remains valid.

Also, instead of *.out I would use 0.out+1.out to make sure the files are in the right order.

Thus "Copy /B 0.out+1.out msieve.dat"[/QUOTE]


What does the /B flag do? Check the .dat file for integrity?

[quote=Andi47;224101]What does the /B flag do? Check the .dat file for integrity?[/quote]
It sets copy to binary mode, which makes sure that it doesn't mess with the file at all. (The default is text mode, which may mess with line endings or something like that...I'm not really sure exactly what it does, but at any rate it's not something you want on a dat file like this.)

[quote=mdettweiler;224121]It sets copy to binary mode, which makes sure that it doesn't mess with the file at all. (The default is text mode, which may mess with line endings or something like that...I'm not really sure exactly what it does, but at any rate it's not something you want on a dat file like this.)[/quote]
It has nothing to do with line endings; the only difference between copy's "text" and "binary" modes is that in "text" mode Control-Z is interpreted as an end-of-file character; reading of source files stops at the first C-Z, and it's written at the end of destination files. This bit of insanity comes from the fact that copy emulates faithfully the MS-DOS copy command (on the assumption that someone still might be running old DOS BAT files which depend on this idiocy).