After people get a chance to test Brian's latest binary and confirm it is ok, Brian please send me the latest good version that you want me to put on the website.

[QUOTE=Brian Gladman;346790]I suspect that this issue is a result of a update 3 for the Visual C++ redistributable package that occurred in late June. This is available [URL="http://http://www.microsoft.com/en-gb/download/details.aspx?id=30679"]here[/URL] and probably contains the files you need. Please let me know if installing this doesn't solve the problem.[/QUOTE]

Unfortunately, this does not solve the problem. It looks like this is the one that frmky pointed to earlier (post 132) and that I already had downloaded and installed. I went ahead and downloaded the 32-bit redist also, ie I now have both the 32-bit and 64-bit installed, but it still says vcomp120.dll not found. I pulled up dependency walker and found out that this msieve.exe is also missing ieshims.dll. From just a quick search, it seems like that dll was only necessary on XP. Did you perhaps compile this for a specific OS, like XP? Or, since frmky's post talks about a missing vcomp110.dll, did you perhaps switch to a different compiler, like Visual Studio 2013 (instead of 2012) for this binary? Hopefully we can figure out how to get this to work. Please let me know if you'd like me to try anything.

Hello,

When I run the following with Linux Ubuntu 12.04 64-Bit:
msieve -s C238_121_93.dat -l C238_121_93.log -i C238_121_93.ini -nf C238_121_93.fb -t 4 -nc

I keep getting the following error:
[CODE]integrator failed nan inf
[euler:10931] *** Process received signal ***
[euler:10931] Signal: Segmentation fault (11)
[euler:10931] Signal code: (128)
[euler:10931] Failing at address: (nil)
[euler:10931] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fe50695ccb0]
[euler:10931] [ 1] /usr/local/lib/libgmp.so.10(+0x13465) [0x7fe506e79465]
[euler:10931] [ 2] msieve(eval_poly+0x94) [0x41dd34]
[euler:10931] [ 3] msieve(nfs_read_relation+0x29b) [0x4390fb]
[euler:10931] [ 4] msieve(nfs_purge_duplicates+0x3a5) [0x45fca5]
[euler:10931] [ 5] msieve(nfs_filter_relations+0x4f7) [0x42b637]
[euler:10931] [ 6] msieve(factor_gnfs+0x7ef) [0x41d81f]
[euler:10931] [ 7] msieve(msieve_run+0x584) [0x40e2e4]
[euler:10931] [ 8] msieve(factor_integer+0x13b) [0x40d03b]
[euler:10931] [ 9] msieve(main+0x7a0) [0x40c010]
[euler:10931] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fe5062fe76d]
[euler:10931] [11] msieve() [0x40cc7d]
[euler:10931] *** End of error message ***
Segmentation fault (core dumped)[/CODE]Here is the log it creates:
[CODE]Fri Jul 19 15:50:11 2013 Msieve v. 1.51 (SVN unknown)
Fri Jul 19 15:50:11 2013 random seeds: f1a22004 c780709f
Fri Jul 19 15:50:11 2013 MPI process 0 of 1
Fri Jul 19 15:50:11 2013 factoring 15361611413579626537050783586280481077905158262497946644429242618828114393675749532960693763487891170755488543105808804212915950808875094929083135048627004135175632408003998127029859491160527729903274486887631105056371237195122963118170454 (239 digits)
Fri Jul 19 15:50:13 2013 searching for 15-digit factors
Fri Jul 19 15:50:14 2013 commencing number field sieve (238-digit input)
Fri Jul 19 15:50:14 2013 warning: NFS input not found in factor base file
Fri Jul 19 15:50:14 2013 R0: 0
Fri Jul 19 15:50:14 2013 A0: 0
Fri Jul 19 15:50:14 2013 skew 1.00, size 0.000e+00, alpha 0.000, combined = 0.000e+00 rroots = 0
Fri Jul 19 15:50:14 2013
Fri Jul 19 15:50:14 2013 commencing relation filtering
Fri Jul 19 15:50:14 2013 warning: NFS input not found in factor base file
Fri Jul 19 15:50:14 2013 estimated available RAM is 15937.9 MB
Fri Jul 19 15:50:14 2013 commencing duplicate removal, pass 1
[/CODE]

Note: msieve works fine on factoring smaller c70.

Mathew, the C70 would have used the quadratic sieve, whose code is much simpler than the NFS used in your large test.

Your crash occurs while using GMP (or MPIR). I wonder if it's because GMP has a problem or is only being given something wrong...

[QUOTE=WraithX;346810]Unfortunately, this does not solve the problem. It looks like this is the one that frmky pointed to earlier (post 132) and that I already had downloaded and installed. I went ahead and downloaded the 32-bit redist also, ie I now have both the 32-bit and 64-bit installed, but it still says vcomp120.dll not found. I pulled up dependency walker and found out that this msieve.exe is also missing ieshims.dll. From just a quick search, it seems like that dll was only necessary on XP. Did you perhaps compile this for a specific OS, like XP? Or, since frmky's post talks about a missing vcomp110.dll, did you perhaps switch to a different compiler, like Visual Studio 2013 (instead of 2012) for this binary? Hopefully we can figure out how to get this to work. Please let me know if you'd like me to try anything.[/QUOTE]

Jeff also has problems so it seems that the dependencies are not in good shape. I am confident that I used VS 2012 for building all the dependencies over which I have direct control.

But I do have VS 2012 and 2013 preview installed side by side (which MS says is OK!) so it is possible that this has introduced some unexpected dependencies. I have just cleaned everything and rebuilt the binary and asked Jeff to test it. I will report on progress as soon as I have answers.

[QUOTE=Mathew;346811]Hello,

When I run the following with Linux Ubuntu 12.04 64-Bit:
msieve -s C238_121_93.dat -l C238_121_93.log -i C238_121_93.ini -nf C238_121_93.fb -t 4 -nc

Here is the log it creates:
[CODE]Fri Jul 19 15:50:11 2013 Msieve v. 1.51 (SVN unknown)
Fri Jul 19 15:50:11 2013 random seeds: f1a22004 c780709f
Fri Jul 19 15:50:11 2013 MPI process 0 of 1
Fri Jul 19 15:50:11 2013 factoring 15361611413579626537050783586280481077905158262497946644429242618828114393675749532960693763487891170755488543105808804212915950808875094929083135048627004135175632408003998127029859491160527729903274486887631105056371237195122963118170454 (239 digits)
Fri Jul 19 15:50:13 2013 searching for 15-digit factors
Fri Jul 19 15:50:14 2013 commencing number field sieve (238-digit input)
Fri Jul 19 15:50:14 2013 warning: NFS input not found in factor base file
Fri Jul 19 15:50:14 2013 R0: 0
Fri Jul 19 15:50:14 2013 A0: 0
Fri Jul 19 15:50:14 2013 skew 1.00, size 0.000e+00, alpha 0.000, combined = 0.000e+00 rroots = 0[/CODE]

Note: msieve works fine on factoring smaller c70.[/QUOTE]

Mathew, can you post here the contents of your .fb file? From the msieve log it looks like you may be missing the A0 and/or R0 parameters.

WraithX,
Here is the contents of C238_121_93.fb

[CODE]N 15361611413579626537050783586280481077905158262497946644429242618828114393675749532960693763487891170755488543105808804212915950808875094929083135048627004135175632408003998127029859491160527729903274486887631105056371237195122963118170454
A6 1
A0 93
R1 2342388736625917052139104541473924426001
R0 -191943424957750480504146841291811
SKEW 2.12853350832997
FRMAX 90000000
FAMAX 90000000
SRLPMAX 2147483648
SALPMAX 2147483648
[/CODE]

I could be completely wrong but doesn't msieve require m?

I think I've seen problems of this sort before; what happens if you move the SKEW line up above the A/R lines?

Msieve doesn't require a line for m, but the SKEW line needs to be right after the N line. Putting it where you put it will trigger different behavior parsing the .fb file, before the last few poly coefficients are read.

For future reference, the A and R lines can appear in any order, and missing coefficients are treated as 0. Readme.nfs contains a description of the format of the .fb file; I admit that it isn't documented anywhere else.

I moved the SKEW line to after N and am still getting the same results.

As we suspected there is an unwelcome interaction between VS 2012 and 2013 preview. If you install the x64 version of the VS 2013 redistributable from [URL="http://www.microsoft.com/en-gb/download/details.aspx?id=39315"]here[/URL], the binary should then work. This is a temporary fix until I work out how to build with VS 2012 without adding a 2013 dependency.

ah... upgrading, updating... why fix when it is not broken?

[QUOTE=Mathew;346822]I moved the SKEW line to after N and am still getting the same results.[/QUOTE]
The only time you would get an 'integrator failed' message is if the polynomials were not parsed correctly. Maybe remove the 'MAX' lines? Maybe also put a trailing newline in the file? I vaguely remember these could sometimes cause problems.

Also, and this could be silly, does your file have DOS line endings on a unix machine, or unix line endings on a windows machine? It shouldn't matter, but you could try using dos2unix or unix2dos...

jasonp,

I deleted the 4 max lines (I think this is what you meant), and added a newline at the bottom.

[CODE]N 15361611413579626537050783586280481077905158262497946644429242618828114393675749532960693763487891170755488543105808804212915950808875094929083135048627004135175632408003998127029859491160527729903274486887631105056371237195122963118170454
SKEW 2.12853350832997
A6 1
A0 93
R1 2342388736625917052139104541473924426001
R0 -191943424957750480504146841291811
[/CODE]with vim I did :set list
The .fb file has no ^M lines in it.

I also tried [B]make all[/B] opposed to [B]make all ECM=1 MPI=1[/B]
msieve -s C238_121_93.dat -l C238_121_93.log -i C238_121_93.ini -nf C238_121_93.fb -t 4 -nc
returns:

[CODE]integrator failed nan inf
Segmentation fault (core dumped)[/CODE]The log file gives the following:


[CODE]Sat Jul 20 14:34:20 2013
Sat Jul 20 14:34:20 2013
Sat Jul 20 14:34:20 2013 Msieve v. 1.51 (SVN unknown)
Sat Jul 20 14:34:20 2013 random seeds: 4d1ac874 9c22c9ee
Sat Jul 20 14:34:20 2013 factoring 15361611413579626537050783586280481077905158262497946644429242618828114393675749532960693763487891170755488543105808804212915950808875094929083135048627004135175632408003998127029859491160527729903274486887631105056371237195122963118170454 (239 digits)
Sat Jul 20 14:34:22 2013 no P-1/P+1/ECM available, skipping
Sat Jul 20 14:34:22 2013 commencing number field sieve (238-digit input)
Sat Jul 20 14:34:22 2013 warning: NFS input not found in factor base file
Sat Jul 20 14:34:22 2013 R0: 0
Sat Jul 20 14:34:22 2013 A0: 0
Sat Jul 20 14:34:22 2013 skew 1.00, size 0.000e+00, alpha 0.000, combined = 0.000e+00 rroots = 0
Sat Jul 20 14:34:22 2013
Sat Jul 20 14:34:22 2013 commencing relation filtering
Sat Jul 20 14:34:22 2013 warning: NFS input not found in factor base file
Sat Jul 20 14:34:22 2013 estimated available RAM is 15937.9 MB
Sat Jul 20 14:34:22 2013 commencing duplicate removal, pass 1[/CODE]

I can reproduce the behavior locally. Debug in progress.

Edit: if your number contains small factors, please divide them all out before running NFS using Msieve. The library did this for you, which made the NFS input different from the library input, which is a fatal error that is reported very badly. In this particular case, you must insure that R0^6 + 93*R1^6 = N, where N is what you give both the .fb file and the ini file.

Tiny issue
 

It may help some people to know that the default -march=core2 build, with gcc-4.6, produces code for get_merge_memuse that uses the SSSE3 'pshufb %xmm1, %xmm2' instruction, which even the Opteron 61xx-series processors don't support. I may be the only person with so old a computer, and the problem only appeared when I upgraded the 48-core from ubuntu-10.04 to ubuntu-12.04

Changing to -march=barcelona and re-running five hours of filtering :bangheadonwall:

I have now posted the latest SVN927 from Brian on my website:
[url]http://gilchrist.ca/jeff/factoring/index.html[/url]

This should just be linked to the VS2012 DLL so you should just need the vcomp110.dll now (VS2012 Redistributable Package) that you can find the link off my website if you need it.

[QUOTE=jasonp;346839]I can reproduce the behavior locally. Debug in progress.

Edit: if your number contains small factors, please divide them all out before running NFS using Msieve. The library did this for you, which made the NFS input different from the library input, which is a fatal error that is reported very badly. In this particular case, you must insure that R0^6 + 93*R1^6 = N, where N is what you give both the .fb file and the ini file.[/QUOTE]

Yes,
This resolved the issue.

Thanks everyone

[QUOTE=Jeff Gilchrist;346860]I have now posted the latest SVN927 from Brian on my website:
[url]http://gilchrist.ca/jeff/factoring/index.html[/url]

This should just be linked to the VS2012 DLL so you should just need the vcomp110.dll now (VS2012 Redistributable Package) that you can find the link off my website if you need it.[/QUOTE]

Good news everyone! This version worked splendidly on the L1911 C167. It was able to complete an 8.83M^2 matrix in 21.5 hours with 16 threads. I'm currently testing it with 8 threads to see how the time changes. I'll let you know when it is done.

A big Thank You! to both Brian and Jeff for compiling/updating/testing msieve to get this to work and to Jeff for hosting the binary. Thank You!

Would you mind posting the numbers on

[url]http://www.mersenneforum.org/showthread.php?t=18374[/url]

once you've got them?

[QUOTE=fivemack;347001]Would you mind posting the numbers on

[url]http://www.mersenneforum.org/showthread.php?t=18374[/url]

once you've got them?[/QUOTE]

Sure thing. If you don't mind waiting for a bit, I'll even try to get timings for all the even numbered thread counts (2,4,6,8,10,12,14,16, maybe even 1). Since I don't have an MPI library and I don't have an MPI msieve, I may not be getting the most out of this machine. But, it still seems to be performing rather well. Just as a preview, when I last checked, it looked like the total eta for 8 threads was around 22.x hours. So, not too much benefit to running 16 threads vs 8 threads, but I think every little bit helps if/when I run larger and larger jobs on this. I'll post all the timings, in that thread, when I have them.

[QUOTE=Jeff Gilchrist;346860]I have now posted the latest SVN927 from Brian on my website:
[URL]http://gilchrist.ca/jeff/factoring/index.html[/URL]

This should just be linked to the VS2012 DLL so you should just need the vcomp110.dll now (VS2012 Redistributable Package) that you can find the link off my website if you need it.[/QUOTE]
I ran a test using SVN 927 everything ran fine up until [code]Thu Aug 01 15:57:30 2013 commencing Lanczos iteration (6 threads)
Thu Aug 01 15:57:30 2013 memory use: 3433.0 MB[/code] then the application just come up with the window on my screen saying stopped working.
one thing I did note though I have never seen the line [code]commencing Lanczos iteration [/code] in my log before. Is this a new phase?
I have attached the log and my start a bat file. I have successfully managed to run the I 7 test file without incident. I have also attached the successful run log.

I'm seeing a similar issue when trying to run msieve on the 299-digit cofactor of [url]http://www.factordb.com/index.php?id=1000000000043596293[/url], 372889817...

[code]Msieve v. 1.52 (SVN 936)
Tue Aug 6 23:06:33 2013
random seeds: 2e3714b9 5c9bc30e
factoring 37288981771039578285751767666152422444740872141841696556525922548756411612078722511313217745832358082426302486574369734100797157544170974891788191605290166523358074421434377183830973607149873256496995571602300206060954537411516595930389968935728417411471833743940981915375445496229631862855442183381 (299 digits)
no P-1/P+1/ECM available, skipping
commencing number field sieve (299-digit input)
warning: NFS input not found in factor base file
integrator failed nan inf
R0: 0
A0: 0
skew 1.00, size 0.000e+00, alpha 0.000, combined = 0.000e+00 rroots = 0

commencing relation filtering
warning: NFS input not found in factor base file
estimated available RAM is 12015.3 MB
commencing duplicate removal, pass 1
Segmentation fault (core dumped)[/code]

Here's input.fb:

[code]N 37288981771039578285751767666152422444740872141841696556525922548756411612078722511313217745832358082426302486574369734100797157544170974891788191605290166523358074421434377183830973607149873256496995571602300206060954537411516595930389968935728417411471833743940981915375445496229631862855442183381
SKEW 1.53
A6 1
A0 -13
R1 1
R0 -134106816713249934153658112422086110743809315028093
FAMAX 727400000
FRMAX 727400000
SALPMAX 8589934592
SRLPMAX 8589934592[/code]

And input.ini, which of course is just n:

[code]37288981771039578285751767666152422444740872141841696556525922548756411612078722511313217745832358082426302486574369734100797157544170974891788191605290166523358074421434377183830973607149873256496995571602300206060954537411516595930389968935728417411471833743940981915375445496229631862855442183381[/code]

I don't see anything obviously wrong here; is this a different issue?

Polynom question
 

I've tried to search poly for 3,664+1, but I can't understand why msieve generates different polys for the same C5.

For instance,
msieve.exe -np1 293700,293701
msieve.exe -nps "stage2_norm=2.0e+26"
msieve.exe -npr "min_evalue=1.0e-14"

first_run (msieve.dat.ms)
293700 1397440288541986 4478065234141680360882817 -2646467642417801303869024298 -3599037771632009548120018107216510709 634456441079135352336684238962817871111 312509308878988211203 -15926882136722045449097144514629677396 -1.48 8.253952e+025
second run:
293700 1829556961989992 7964401858144116000145295 -16290668681053306780662697488 -30191334832372427186210377555496771461 18342641143578897062225246421826683577631 291377541254794571903 -15926882071091321988301200975327033344 -1.94 1.362999e+026
third run:
293700 -3057506938837243 -4067250878705725413749304 61572828282060473603942651797 34905381927171164712790251745494442467 -156920916090391208277241328651760917010842 203997302334302799943 -15926882858844157268904316424670420895 -1.48 1.355203e+026
293700 -202924021073258 6080305114498964037396469 1345235070165910556666877880 -17509455271765713730856017988411871119 -1144573362330082304893085355558960151909 346080882673041852263 -15926882481932252427695730686237341092 -1.48 1.541913e+026
293700 -92275958057363 -5838513017255197123041550 507762258296670036318237684 13479765598879587657730718111429015834 -341387246780453842740135044603911369823 269229725042974558717 -15926882451026824041231354533379949130 -1.48 1.401040e+026

msieve_gpu also generates different polys each time i ran it with the same params.
Is it normal? How can I be sure if I don't miss a good poly? :smile:

One more thing, if I find a good poly for an exact A5. Can I find a better one running poly search for the same A5 many times?

@ryanp, it sounds as if you are trying to use msieve to do the sieving. Bur it only has a line siever which is nearly useless. You need the GGNFS lattice sievers to do any useful sieving, see [url]http://gilchrist.ca/jeff/factoring/nfs_beginners_guide.html[/url] (especially the SNFS section at the bottom).

But to do a SNFS 299 job like that in a year would need NFS@HOME to do the sieving and a large cluster for linear algebra.

@sashamkrt, the search space is far too large for msieve to search it all, it just searches a random subset. So getting different results each time is normal. Either use the poly with the best score or do some trial sieving with the best few and use the one that sieves best (this is only worthwhile for a fairly large job though).

Chris

[QUOTE=ryanp;348513]
commencing number field sieve (299-digit input)
warning: NFS input not found in factor base file
integrator failed nan inf
R0: 0
A0: 0
skew 1.00, size 0.000e+00, alpha 0.000, combined = 0.000e+00 rroots = 0
[/QUOTE]

Does the input number have small factors? These are not allowed for NFS postprocessing, which is what was wrong the last time I saw this happen.

Edit: to be more specific, if the sieving occurred on a different version of N, then that N has to be what's given to the postprocessing. Unfortunately the library is very bad at reporting that problem if it happens.

Chris: ryanp really does have the resources to go after very large jobs like this...

Edit 2: never mind, I found the problem. In include/common.h, change LINE_BUF_SIZE to be larger than 300, then recompile. This is a buffer overflow reading the factor base file.

[QUOTE=chris2be8;348552]

@sashamkrt, the search space is far too large for msieve to search it all, it just searches a random subset. So getting different results each time is normal.
[/QUOTE]

I've searched poly for C192(3,664+1) A5 from 0 to 500k, 1 to 1.5m, 11m, 12m, 210m and found several good polys.
The best score I found was ~1.477e-014 with A5=293700 then I tried to found a good poly for this A5 running 20 times
msieve.exe -np1 293700,293701
msieve.exe -nps "stage2_norm=2.0e+26"
msieve.exe -npr "min_evalue=1.0e-14"

and found no good polys at all. Some of them was equal. The best was 1.020e-014.
Now I understand that msieve uses random subset, but the results for the same leading coeff differ so much. I mean there are some probability or luck of finding good poly regardless of the range of leading coefficients.

getting a segfault
 

Hi,

I just happened to stumble upon this crash by typing a random number into a worktodo.ini file and running msieve svn 936.

The number:
[CODE]238472934729347298373323987232349283479273[/CODE]

The crash happens in svn 897 but not svn 896.

It's crashing at common/lanczos/lanczos_vv.c:386
[CODE] packed_matrix_t *p = task->matrix;[/CODE]

on the dereference of task which is NULL

Now, task should of been initialized in lanczos_matmul0.c:516
[CODE] p->tasks = (la_task_t *)xmalloc(sizeof(la_task_t) * [/CODE]

but the program doesn't actually get there because of an early return at line 436.
[CODE] if (max_nrows <= MIN_NROWS_TO_PACK)
return;
[/CODE]

(It does return early as well in svn 896.)

Can someone else confirm the bug?

Thanks.

Yes, I found this a few days ago in the CUDA branch and never corrected it in the trunk. It will happen for any QS job under about 75 digits.

Edit: both problems now fixed in SVN 938.

[QUOTE=chris2be8;348552]@ryanp, it sounds as if you are trying to use msieve to do the sieving. Bur it only has a line siever which is nearly useless. You need the GGNFS lattice sievers to do any useful sieving, see [url]http://gilchrist.ca/jeff/factoring/nfs_beginners_guide.html[/url] (especially the SNFS section at the bottom).

But to do a SNFS 299 job like that in a year would need NFS@HOME to do the sieving and a large cluster for linear algebra.[/QUOTE]

It's already sieved, to 720M unique input relations. I just can't understand why msieve is having trouble parsing the input factor base and/or .ini files...

@ryanp: Did you try this fix?
[QUOTE=jasonp;348560]Edit 2: never mind, I found the problem. In include/common.h, change LINE_BUF_SIZE to be larger than 300, then recompile. This is a buffer overflow reading the factor base file.[/QUOTE]

[QUOTE=prgamma10;348639]@ryanp: Did you try this fix?[/QUOTE]

Oops, I missed that from earlier. I tried that -- and it worked. Thanks! (to you and jasonp)

A comment to anyone reporting problems with msieve, please post the command line used to invoke it as well as the output. It would save (mis)guessing what you are trying to do.

Chris

[QUOTE=sashamkrt;348575]Now I understand that msieve uses random subset, but the results for the same leading coeff differ so much. I mean there are some probability or luck of finding good poly regardless of the range of leading coefficients.[/QUOTE]

Correct! However, small A5 values produce skews higher than large values, without producing obviously better polys. This higher skew sometimes means having to search more special-q, which sometimes produce more duplicate relations. So, in general, it is not worth your time to search A5 under 1M for composites under 170 digits, and not under 10M for composites over 175 digits. It's not that you'll find bad polys, it's that you are equally likely to find as good a poly at the higher coeff, but with lower skew.

The search is definitely probabilistic, and you should not assume there is a "magic" A5.

Another thought, it would be nice if msieve wrote a log message saying what parameters it was called with. It would have saved ryanp from being accidentally dissed by me.

Chris

Jeff has just posted an updated msieve Windows x64 binary that contains the latest bug fixes: [URL="http://gilchrist.ca/jeff/factoring/msieve152_svn927_win64_i7.zip"]http://gilchrist.ca/jeff/factoring/msieve152_svn927_win64_i7.zip[/URL]

This is a non-GPU binary but I am at last in a position to build the GPU version again (at least I will be when I get a new Nvidia card to test it in a a few days time).

Given that MPIR comes in many architecture flavours and GPUs have quite a few different compute capabilities, we will now face a combinatorial explosion in binaries if we continue to deliver statically linked binaries (I have done this in order to avoid forcing people to find and install DLLs).

So I think we may now have to accept that people will have to install an MPIR DLL of their choice in order to use the GPU versions of GMP-ECM and MSIEVE on Windows. Unless, of course, people can think of other ways of coping with all the variations.

Brian

[QUOTE=Brian Gladman;348904] Unless, of course, people can think of other ways of coping with all the variations.[/QUOTE]

Require everyone to dual-boot into Linux :wink:

[QUOTE=xilman;348920]Require everyone to dual-boot into Linux :wink:[/QUOTE]
I'll consider that option just as soon as my Raspberry Pi's can dual boot into Windows :smile:

Ok, I finally have enough results to post. I ran most of these with:
msieve -v -t X -nc2 "skip_matbuild=1"
Where X is the number of threads I was testing. I would delete the .chk and .bak.chk files before each run.
These times are only the BLanczosTime value (in seconds), nothing else.
The computer is a dual Xeon E5-2687W with 64GB of DD3-1600 ram, hyper-threading is turned off.
For some strange reason, the first time I ran through with 12, 14, and 16 threads, it took a lot longer than the second time I ran them.
Nothing else was running on the computer. The OS is Windows 7 Pro 64-bit using Brian's svn927 build of msieve.
The following shows an msieve log snippet showing the characteristics of this matrix.
Below that is the table showing the BLanczosTime (in seconds) for each number of threads.
You can see how I ran 12, 14, and 16 two times.
It looks like 13 threads is the winner on this machine to solve an 8.83M^2 matrix in 18h25m.
[CODE]
commencing linear algebra
skipping matrix build
matrix starts at (0, 0)
matrix is 8830805 x 8830983 (2692.1 MB) with weight 833487913 (94.38/col)
sparse part has weight 599756073 (67.91/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 8830757 x 8830983 (2587.2 MB) with weight 664939915 (75.30/col)
sparse part has weight 589918881 (66.80/col)
using block size 8192 and superblock size 1966080 for processor cache size 20480 kB
commencing Lanczos iteration (X threads)
memory use: 2110.0 MB

T | BL time | Skip Mat Build
---|---------|------
4 | 195259 | yes
6 | 98016 | yes
8 | 80205 | no
9 | 76663 | yes
10 | 71181 | yes
11 | 68044 | yes
12 | 67405 | yes | 91616 (*1) skip mat build
13 | 66323 | yes
14 | 67283 | yes | 81532 (*1) skip mat build
15 | 69686 | yes
16 | 69429 | yes | 78535 (*2) full run start to finish

*1 = BLanczosTime for the first time I ran with this number of threads. I did use skip_matbuild for these.
*2 = Very first run where I had msieve run the whole computation from start to finish.
[/CODE]
And here are the logs in case anyone would like to see the gory details.

Nice! So once we hit asymptotic territory the differences amount to +-5%.

It would be interesting to compare these times with using a 1x2 MPI grid with 8 threads per process, though this size problem might be too small for that to be an improvement.

[QUOTE=Speedy51;347940]I ran a test using SVN 927 everything ran fine up until [code]Thu Aug 01 15:57:30 2013 commencing Lanczos iteration (6 threads)
Thu Aug 01 15:57:30 2013 memory use: 3433.0 MB[/code] then the application just come up with the window on my screen saying stopped working.
one thing I did note though I have never seen the line [/QUOTE]
I don't think anybody has commented on this.

@Brian Gladman: does 64-bit MSVC require that you compile with LARGEADDRESSAWARE set in the linker options? The tooltip for it appears to imply that you won't get more than 2GB of VM even in a 64-bit build...

[QUOTE=jasonp;349269]I don't think anybody has commented on this.

@Brian Gladman: does 64-bit MSVC require that you compile with LARGEADDRESSAWARE set in the linker options? The tooltip for it appears to imply that you won't get more than 2GB of VM even in a 64-bit build...[/QUOTE]
My understanding is that it is still necessary in 64-bit applications but, in contrast to 32-bit applications, it defaults to ON in this case.

On Windows x64, I believe that 32-bit applications with a 2GB limit will actually get 4GB rather than 2GB.

In recent MSVC builds of MSIEVE, I have been explicitly setting it to ON.

Matrix not invertible
 

While post processing [url=http://escatter11.fullerton.edu/nfs/crunching.php]C211_127_90[/url] using [url=http://gilchrist.ca/jeff/factoring/index.html]SVN 939 compiled by Brian Gladman[/url] on my i7 using 4 cores in Win 7 (64 bit), LA never really gets started. Constantly restarts, occasionally gets a bit of traction then fails with the error "matrix not invertible". Repeat.

Msieve 1.49 seems to have no problems with the same data (so far).

Can't get GC_6_270 to start whatever version I try
 

I have tried numerous versions of Msieve and I cannot get it start except within 1 to 4 seconds. I have attached the log file and my start.bat file in the hope that somebody can see what is happening.The size on disk of the.DAT file is 5,192,830,976 bytes Zipped file size on disk is 2,747,011,072 bytes Packed size is (with in zipped file) 2,747,008,871 bytes.

I have fixed the issue
 

[QUOTE=Speedy51;349622]I have tried numerous versions of Msieve and I cannot get it start except within 1 to 4 seconds. I have attached the log file and my start.bat file in the hope that somebody can see what is happening.The size on disk of the.DAT file is 5,192,830,976 bytes Zipped file size on disk is 2,747,011,072 bytes Packed size is (with in zipped file) 2,747,008,871 bytes.[/QUOTE]
Please ignore my previous post as I have discovered what the issue was. I had the wrong letters at the beginning of each file and on my batch file it should have been/now is GW. As I write it is currently running nicely under SVN 939

[QUOTE=swellman;349598]While post processing [url=http://escatter11.fullerton.edu/nfs/crunching.php]C211_127_90[/url] using [url=http://gilchrist.ca/jeff/factoring/index.html]SVN 939 compiled by Brian Gladman[/url] on my i7 using 4 cores in Win 7 (64 bit), LA never really gets started. Constantly restarts, occasionally gets a bit of traction then fails with the error "matrix not invertible". Repeat.

Msieve 1.49 seems to have no problems with the same data (so far).[/QUOTE]
GW_6_270 seems to be doing the same thing. I will start processing tomorrow with SVN 902 and if needed go backwards from there until I get the successful run. Below you will find snippet of log from the run with 939 [code] Thu Aug 15 19:55:36 2013 Msieve v. 1.52 (SVN 939)
Thu Aug 15 19:55:36 2013 random seeds: 7fb1d4bc a7a233f7
Thu Aug 15 19:55:36 2013 factoring 89051819526833660155460079452463606543882056159483477970649215674340231277546480108919148608030328217218865379171969271596155218634375703804088427073343961355157040881322584085907777739106484490967 (197 digits)
Thu Aug 15 19:55:38 2013 searching for 15-digit factors
Thu Aug 15 20:20:06 2013 commencing linear algebra
Thu Aug 15 20:20:07 2013 read 4179676 cycles
Thu Aug 15 20:20:13 2013 cycles contain 12352725 unique relations
Thu Aug 15 20:22:19 2013 read 12352725 relations
Thu Aug 15 20:22:34 2013 using 20 quadratic characters above 536870910
Thu Aug 15 20:23:29 2013 building initial matrix
Thu Aug 15 20:25:38 2013 memory use: 1567.0 MB
Thu Aug 15 20:25:41 2013 read 4179676 cycles
Thu Aug 15 20:25:42 2013 matrix is 4179498 x 4179676 (1236.4 MB) with weight 364725810 (87.26/col)
Thu Aug 15 20:25:42 2013 sparse part has weight 282307898 (67.54/col)
Thu Aug 15 20:26:17 2013 filtering completed in 2 passes
Thu Aug 15 20:26:18 2013 matrix is 4178288 x 4178466 (1236.3 MB) with weight 364693560 (87.28/col)
Thu Aug 15 20:26:18 2013 sparse part has weight 282298785 (67.56/col)
Thu Aug 15 20:26:28 2013 matrix starts at (0, 0)
Thu Aug 15 20:26:29 2013 matrix is 4178288 x 4178466 (1236.3 MB) with weight 364693560 (87.28/col)
Thu Aug 15 20:26:29 2013 sparse part has weight 282298785 (67.56/col)
Thu Aug 15 20:26:29 2013 saving the first 48 matrix rows for later
Thu Aug 15 20:26:30 2013 matrix includes 64 packed rows
Thu Aug 15 20:26:31 2013 matrix is 4178240 x 4178466 (1183.4 MB) with weight 290405376 (69.50/col)
Thu Aug 15 20:26:31 2013 sparse part has weight 268446248 (64.25/col)
Thu Aug 15 20:26:31 2013 using block size 8192 and superblock size 1179648 for processor cache size 12288 kB
Thu Aug 15 20:26:48 2013 commencing Lanczos iteration (6 threads)
Thu Aug 15 20:26:48 2013 memory use: 967.6 MB
Thu Aug 15 20:26:50 2013 lanczos error: submatrix is not invertible
Thu Aug 15 20:26:50 2013 lanczos halted after 2 iterations (dim = 64)
Thu Aug 15 20:26:50 2013 linear algebra failed; retrying...
Thu Aug 15 20:26:50 2013 commencing Lanczos iteration (6 threads)
Thu Aug 15 20:26:50 2013 memory use: 967.6 MB[/code]

Er... I started GW_6_270 this morning with msieve SVN r935, filtered at target density 80, and LA is currently underway without a hiccup so far.
The one you reserved ([url]http://www.mersenneforum.org/showpost.php?p=348655&postcount=783[/url]) was GC_6_270, and I subsequently reserved GW_6_270 for myself :smile:

That said, are you on an Internet connection plan with a monthly cap ? The computer I'm using is in a datacenter with unlimited bandwidth on gigabit Ethernet. It usually downloads from the NFS@Home server at 800 KB - 1 MB/s. If it's hard for you to download GC_6_270, then I guess I'll leave GW_6_270 to you and download GC_6_270 ?

I wouldn't go as far back as 902; I had this problem on Linux with 940 (reported elsewhere in the forum) and am running happily with 923 which has the faster linalg code.

The computer finished GW_6_270 (SNFS difficulty 213) by p54 * p114.

[QUOTE=debrouxl;349632]Er... I started GW_6_270 this morning with msieve SVN r935, filtered at target density 80, and LA is currently underway without a hiccup so far.
The one you reserved ([url]http://www.mersenneforum.org/showpost.php?p=348655&postcount=783[/url]) was GC_6_270, and I subsequently reserved GW_6_270 for myself :smile:

That said, are you on an Internet connection plan with a monthly cap ? The computer I'm using is in a datacenter with unlimited bandwidth on gigabit Ethernet. It usually downloads from the NFS@Home server at 800 KB - 1 MB/s. If it's hard for you to download GC_6_270, then I guess I'll leave GW_6_270 to you and download GC_6_270 ?[/QUOTE]
You are correct I'll hopefully grab GC_6_270 today (or when servers are back on line) it was my mistake. I miss read the names. I've stopped work on GW_6_270. I have fast unlimited data too. Sorry for getting your job.

@debrouxl Thanks I'll try with 923 (windows)

FWIW, the next batch of GCW numbers proposed for NFS@Home are presently in ECM testing. The first success/casualty (depending on whether you want to factor by SNFS or not) appeared less than an hour ago. Strangely enough, it was found by the slowest ECMnet client connecting to the server --- SWMBO's 32-bit Q6600 Windoze box.

I've been trying to update factMsieve.pl to use my GPU for polynomial selection. So far I've found I need to:
Add sortlib= to msieve parms to tell it where the sort library is.
Copy the .ptx file into the current dir I'm running polynomial selection in.
Only run 1 msieve thread using the GPU at once or the GPU runs out of memory.

After all that it does find some polynomials, but then gets a segmentation fault:
[code] Wed Jan 22 14:40:12 2014=>nice -n 19 "/home/chris/ggnfs/trunk/bin/msieve" -np 'sortlib=/home/chris/msieve-1.51/b40c/sort_engine_sm20.so 1,8000' -g 0 -v -t 4 -l rsa110.log -s rsa110.msieve.dat -nf rsa110.msieve.fb -i rsa110.ini >rsa110.msieve.np.sysout
Segmentation fault
Return value 35584. Terminating...
[/code]
I could update factMsieve.pl to ignore the return value, but segmentation faults are worrying.

Chris

In case you are not using it, the latest SVN does have a thread pool fix that v1.51 was missing.

Also, your command line is still using 4 threads for polynomial selection.

Thanks, I'll try the latest SVN.

Is there a way round having to copy the .ptx files into the current dir? Eg can I pass a parameter saying where to look for the .ptx file and the sortlib (the best case would be for msieve to look in the dir it's loaded from)?

And should the number of threads be the number of cores I want it to use or 1 less than the number of cores? Readme.gpu says -t X sets up X threads to feed the GPU and 1 more for size and root optimisation. (The test run on RSA110 didn't get much above 100% of 1 core, but it was only a toy problem.)

Chris

There currently isn't any way to customize where the PTX files come from, though it would be easy to add.

The number of threads to use will depend on how often the stage 2 thread will be running. For small problems it will run all the time and is the limiting factor in how much progress the polynomial selection makes. For larger problems it will run less and less, leaving more room for multiple GPU threads to fully utilize the GPU. But very large problems occupy the GPU fully with just one GPU thread.

I wouldn't use more than two threads for any input, though others have reported additional speedups from more threads.

The latest SVN works OK, thanks.

Now "all" I have to do is work out how to make best use of the GPU. That will take some thinking, and experimentation.

Chris

[QUOTE=jasonp;365196]There currently isn't any way to customize where the PTX files come from, though it would be easy to add.
[/QUOTE]

That would be helpful. Ideally I would like to pass it the directory the .ptx files and the GPU sortlibs are in, and let it choose the right .ptx and sortlib for the GPU architecture from there. Eg pass it "gpudir=/home/chris/msieve-svn/trunk" and let it do the rest (obviously allowing for the sortlib being in the b40c subdir).

The dream is that if it isn't passed the parm it would look in the same dir msieve was in. So if my script calls it as /home/chris/msieve-svn/trunk/msieve it would look in /home/chris/msieve-svn/trunk for support files.

Chris

Readme.nfs says:
For the largest size problems, you will find that it is much faster to run stage 1 and the size optimization together, then sort the size-optimized results and run the root optimization only on the top few (dozens, hundreds, perhaps thousands of) polynomials.

I assume that means to sort data_file_name.ms by the score in the last field. Will the number of fields in each record be constant? Does anyone have sample sort parameters to do this?

I assume that for large problems (over 155 digits) I need to run something like:
msieve -np1 -nps 1,50000 -g 0 (and other parms as necessary)
sort -k 9 msieve.dat.ms | head -100 >msieve.dat.ms.cut
mv msieve.dat.ms msieve.dat.ms.orig
mv msieve.dat.ms.cut msieve.dat.ms
msieve -npr

For medium size problems if I want to run size and root optimization in parallel I could use:
mkfifo msieve.dat.ms
msieve -np1 -nps -g 0 (etc etc) &
msieve -npr (etc)

For small problems like rsa110 it's hard to get good figures because polynomial selection does not take long enough to measure. But it seems to be limited by the single stage 2 thread. Hence the suggestion above.

Chris

[QUOTE=chris2be8;365360]That would be helpful. Ideally I would like to pass it the directory the .ptx files and the GPU sortlibs are in, and let it choose the right .ptx and sortlib for the GPU architecture from there. Eg pass it "gpudir=/home/chris/msieve-svn/trunk" and let it do the rest (obviously allowing for the sortlib being in the b40c subdir).

The dream is that if it isn't passed the parm it would look in the same dir msieve was in. So if my script calls it as /home/chris/msieve-svn/trunk/msieve it would look in /home/chris/msieve-svn/trunk for support files.

Chris[/QUOTE][aol]Me too !!!!????!!![/aol]

[QUOTE=chris2be8;365363]
I assume that means to sort data_file_name.ms by the score in the last field. Will the number of fields in each record be constant? Does anyone have sample sort parameters to do this?

I assume that for large problems (over 155 digits) I need to run something like:
msieve -np1 -nps 1,50000 -g 0 (and other parms as necessary)
sort -k 9 msieve.dat.ms | head -100 >msieve.dat.ms.cut
mv msieve.dat.ms msieve.dat.ms.orig
mv msieve.dat.ms.cut msieve.dat.ms
msieve -npr
Chris[/QUOTE]

The number of fields is 5 more than the degree of the poly. Your initial trials may be small enough to be degree 4, which would explain your "-k 9" vs 10 that I've always sorted by for degree-5 problems up to ~200 digits.

I find that setting stage2_norm= {default/20 to default/22} produces a hit rate from -nps that is about right for the root optimization; 100 or so hits per day. This saves me the step of having to do the sort, and keeps the .ms file size smaller by a factor of 100 or more.

If you use the -s flag to create your own file names, you can avoid the mv commands; when you run -npr step with "-s sortedfile" also, it looks for sortedfile.ms automatically; so whatever you write as the sorted file name can be set as the input file for -npr.
-Curtis

Following a few tests I've found I need to use:

sort -g -k 10 msieve.dat.ms | head 100

The -g tells sort the key field is a floating point number. This might be worth putting into Readme.nfs.

Just to confirm, smaller score is better so I want the head of the sorted list, not the tail. (Always check the obvious!)

Chris

Correct, the smaller size score is what you want.