Running Msieve

[jasonp]

Originally Posted by Wacky

My concern relates to Tom's report of difficulty and your reply

I don't want to spend weeks only to find out that the results are worthless.
Greg has certainly been successful using version 1.32

Okay, I follow now. v1.33 has no linear algebra changes compared to v1.32. Tom is due to report back in the next day or two on whether a followup run with an older msieve version succeeded on his C151 GNFS matrix. ISTR that he had trouble on this number before too. Just out of curiosity, what version of gcc is everyone using for linux compiles? (I use 4.1.1 and it's never caused any trouble)

I'm not sure what to make of the recent spate of reports of linear algebra trouble. People have reported both successes and failures with multithreaded runs using versions > 1.30

PS: Richard, what changes to v1.33 did you have to make?

[miklin]

[quote=(I use 4.1.1 and it's never caused any trouble)

[/quote]

Version 1.33 works well without failures on freebsd. Unequivocally. GCC 4.2.1

admin@relay.diakom.ru [19:09:48][1][~] >gcc -v
Using built-in specs.
Target: i386-undermydesk-freebsd
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 4.2.1 20070719 [FreeBSD]

[fivemack]

The C151 completed without issue (and with a noticeably smaller matrix than msieve-1.33 was offering me with default settings, about 4.5M^2 rather than 5M^2) using msieve-1.29 this morning after about 60 hours on two cores of a quad-core; I'm leaving it to Chad Davis, who did all the sieving, to post the factors.

I'm running linalg for a C154 SNFS (the last Cunningham-table composite of <512 bits) with msieve-1.29 at the moment, that should again take about 60 hours so I can post something more detailed at the weekend.

[Wacky]

Originally Posted by jasonp

Richard, what changes to v1.33 did you have to make?

After it aborted with an allocation error when the matrix started getting VERY large (I assume that the "wrong" implementation of malloc had been selected), I wanted to be able to run it from Xcode (the IDE for OSX). It was too much trouble porting your multi-architecture library of sieving "cores" to compile in that environment. So I just stripped out the options that I did not need.

In the process, I think that both __i386__ and __x86_64__ were set.
So in the cpuid code as well as the call to the architecture specific core, I inverted the order of your selection so that the 64 bit code would take precedence.

Another threading datapoint:
Machines were A64 X2 6000+ linux amd64, A64 X2 4800+ linux i386.
GCC versions from 4.1.x to 4.2.2.
msieve versions 1.31+threading patch to 1.33.
sieved with ggnfs lasieve, pp/la/sqrt with msieve -t 2.
gnfs 1 c155, 1 c154, 5+ c120-130.
snfs 1 c198, 10+ c160-c180.
40+ small snfs and gnfs.
no problems with msieve at all, had a bunch of lasieve crashes on the snfs c198.

[_dc_]

Hello!

First of all, thank you Jason for such a brilliant piece of software!
When I used Msieve for the first time, I was amazed how fast it was going. Great optimization, nice code, awesome job!

Now I'm trying my first rather "big" job (at least for single man). It's C145 with GNFS. Almost everything went smooth.

After finding good poly and sieving with GGNFS, I switched to Msieve.

When trying to do linear algebra, first problems occured.
Msieve failed few times with the following error:

commencing linear algebra
read 3432998 cycles
cycles contain 9838531 unique relations
failed to allocate 236124744 bytes

I restarted combining and matrix solving for three times. Finally it was OK and matrix was solved in 35 hours.

Fri Feb 22 15:20:05 2008  commencing Lanczos iteration (2 threads)
Fri Feb 22 15:20:05 2008  memory use: 906.2 MB
Sun Feb 24 02:20:26 2008  lanczos halted after 53238 iterations (dim = 3366183)
Sun Feb 24 02:20:39 2008  recovered 40 nontrivial dependencies
Sun Feb 24 02:20:40 2008  elapsed time 35:13:47

The worst problem occured on square root step. I tried to restart it for 10+ times total, but I always get the same error (I tried different depencies by the way).

Sun Feb 24 18:48:09 2008  Msieve v. 1.33
..skipped..
Sun Feb 24 18:48:13 2008  
Sun Feb 24 18:48:13 2008  commencing square root phase
Sun Feb 24 18:48:13 2008  reading relations for dependency 1
Sun Feb 24 18:48:15 2008  read 1682371 cycles
Sun Feb 24 18:48:21 2008  cycles contain 5796302 unique relations
Sun Feb 24 18:51:37 2008  read 5796302 relations
Sun Feb 24 18:52:33 2008  multiplying 8104966 relations

My machine has 3 GB of RAM. During first tries swap file was switched off.
I thought this could cause a problem and enabled it.
512 MB of swap for next run. No luck.
1024 MB of swap for next run. No luck.
1536 MB - same. :(
1792 MB - again

I think that 3GB RAM + 1,75GB of swap is more than enough for such a thing (matrix building/solving was finished successfully after all).
I tried two versions of Msieve - 1.32 and 1.33. If you request any files needed to find out where error is, I provide any of them.

Looking for help
Thanks in advance.

WBR,
Craig

[jasonp]

Originally Posted by _dc_

I think that 3GB RAM + 1,75GB of swap is more than enough for such a thing (matrix building/solving was finished successfully after all).
I tried two versions of Msieve - 1.32 and 1.33. If you request any files needed to find out where error is, I provide any of them.

Was this 32-bit windows? A single program is limited to 2GB of virtual memory, and probably a good deal less, no matter how much RAM and swap you have. frmky had better luck compiling his own cygwin binary, but for such a large job you may need a 64-bit OS to complete the square root.

In the past I'd started making changes to the square root that flushed intermediate results to disk, but they turned out to be more difficult than I expected and they are stalled for now.

[miklin]

Originally Posted by jasonp

In the past I'd started making changes to the square root that flushed intermediate results to disk, but they turned out to be more difficult than I expected and they are stalled for now.

It certainly your right to do or to not do a conclusion of intermediate data. But on mine it for such project vital. As to appear a certain opportunity of search of mistakes and localizations of problems. Well and the right for you. It simply my opinion.

[_dc_]

Originally Posted by jasonp

Was this 32-bit windows?

Right, sorry for not including this valuable info in previous post.

A single program is limited to 2GB of virtual memory, and probably a good deal less, no matter how much RAM and swap you have. frmky had better luck compiling his own cygwin binary, but for such a large job you may need a 64-bit OS to complete the square root.

As far as I remember it's possible to set /3GB flag in boot.ini to make 3GB of virtual memory available for applications. But it requires some special flag to be set in PE header as well, I can't recall it's name. I'll try this thing and report about results.

In the past I'd started making changes to the square root that flushed intermediate results to disk, but they turned out to be more difficult than I expected and they are stalled for now.

Well, square root is something that is computed rather fast (comparing to matrix step). So even in case of power failure or something else it's still OK just to restart a complete thing. So if it requires much effor from you, of course it's a question if it's reasonable enough to implement this feature.

Thanks for your reply, I didn't know square root computation could take more memory than matrix step.

[jasonp]

Originally Posted by _dc_

Thanks for your reply, I didn't know square root computation could take more memory than matrix step.

See this post for background about why this is an issue. It does need fixing somehow, but any solution would be a good deal of work.

[_dc_]

Originally Posted by jasonp

See this post for background about why this is an issue. It does need fixing somehow, but any solution would be a good deal of work.

I have read info behind this link. I have to say that I totally agree with you. This trade-off seems to be quite reasonable. I'm programmer and I have to deal with problems like this on daily basis (resource/speed optimization vs time/difficulty of coding).
When I have time and passion I can try harder and produce cute solution that I personally like. But usually one has loads of other problems to solve and he just can't spend time on one of them. It's always your choice and there is no need for excuse to yourself.

/3GB swith solved my issue - I attached logfile. Everything went smooth except I had to reboot once and so stop working copy of Msieve. Non-trivial factors were found on 3rd depency.
To use this switch one just has to put /3GB in boot.ini and then use application linked in special way (/LARGEADDRESSAWARE must be specified for linker).
Then you just run this application and voila - it has 3GB of virtual memory available.
May be you could put both versions on your web page (one .exe linked with /LARGEADDRESSAWARE and another without it)? Msieve is memory-hungry software, so this one could be a good solution.

By the way do you have any plans/ideas about futher development of Msieve?
I'm not sure if it's possible to call this a feature request, but still this could be useful to split sqrt problem on few PCs.
At the moment one could do this by copying all .dat and other files to other PCs and executing Msieve (in my case it's 3.5+ GB in compressed form).
May I introduce the following idea (I'm sorry if something of that is wrong from the point of underlying maths).
As far as I understand you need just a subset of relations to proceed square root step. Is it possible to make a special call to msieve, that would generate only required subset of relations for specific depency to transfer it to another machine and run sqrt procedure there?
I don't know how much effort is required for implementing this feature, because I don't know underlying theory on level. If it takes more than day to implement, then it's probably useless.