[quote=jasonp;209548]See around post 454 in the 'running GGNFS' thread for more details about the duplicate removal. Both fivemack and bdodson have written fairly straightforward C++ programs that produce a new dat file with duplicates removed, at a memory cost much larger than what msieve needs. You may not have enough memory to run those programs if you have 500M relations with 80% duplication.[/quote]
but in theory you could quite possibly run them through in chunks and then combine the chunks and run again

Well the awk one liner did do the trick. I have a lot of memory and it got me down to the 22 million relations I should have had. Instead of ~500 million.

That uses an array to search everything but does use a lot of memory.

[QUOTE=jasonp;209548]See around post 454 in the 'running GGNFS' thread for more details about the duplicate removal. Both fivemack and bdodson have written fairly straightforward C++ programs that produce a new dat file with duplicates removed, at a memory cost much larger than what msieve needs. ...[/QUOTE]

bdodson's programing "languages" include pascal, maple and Cayley/MAGMA;
no C or variants. Our duplicate removal program "remdups" was written
by Greg; and the update that works with gnfs data was Serge's, currently
"remdups3". Early versions of msieve didn't work well with datasets that
included too many duplicates, which was Greg's motivation. More recent
versions of msieve run well with moderate duplications, and, indeed notably
use far less memory than the remdups versions. -bd

Msieve doubts ...
 

Hi!

I'm at my first try at a 154 digit semiprime.

I intend to use ggnfs and MSieve.

Right now im running msieve144 with gpu support on a GTX285.

I'm using this line:

msieve144_gpu -v -np 0x ...

I have a few doubts:

1 - After almost 24 hours of alternating load on the GTX GPU and on the CPU, suddenly the load on the GTX GPU stopped.

The load on the CPU is now constant, but no screen or disk activity for more than 2 hours.

Is this normal ?

2 - When the GTX GPU is being used, the machne becomes almost unresponsive to any other task.

Is this normal ?

Is there any parameter to leave some processing time, lets say 5% to the video, so that the machine can be reasonably used for other tasks ?

3 - For a number of this size, the poly selection time is fixed at some 280 hours.

This time is independent of the GPU used ?

Should it be changed ?

Can it be changed ?

For more ? For less ?

Thanks for any help.

Regards,
scalabis

[QUOTE=scalabis;209913]Hi!

I'm at my first try at a 154 digit semiprime.

I intend to use ggnfs and MSieve.[/QUOTE]

Try to use factmsieve.py its easy to use and you are using ggnfs & Msieve

[QUOTE=scalabis;209913]
Right now im running msieve144 with gpu support on a GTX285.

I'm using this line:

msieve144_gpu -v -np 0x ...[/QUOTE]

the right command is msieve144_gpu -v -np 1[COLOR="Red"],[/COLOR]...

[QUOTE=scalabis;209913]
I have a few doubts:

1 - After almost 24 hours of alternating load on the GTX GPU and on the CPU, suddenly the load on the GTX GPU stopped.

The load on the CPU is now constant, but no screen or disk activity for more than 2 hours.

Is this normal ?

2 - When the GTX GPU is being used, the machne becomes almost unresponsive to any other task.

Is this normal ?

Is there any parameter to leave some processing time, lets say 5% to the video, so that the machine can be reasonably used for other tasks ?

3 - For a number of this size, the poly selection time is fixed at some 280 hours.

This time is independent of the GPU used ?

Should it be changed ?

Can it be changed ?

For more ? For less ?

Thanks for any help.

Regards,
scalabis[/QUOTE]

1. Maybe your GPU is ready with your district -np 1,...
2. If u use factmsieve.py you can tell the script how much you want to use your GPU in Line 69 with GPU_NUM if i see this correct.
3. For a Number with 154 digits are 280 hours on my CPU not really enough - but i don`t know how fast the polysearch on your GPU is working.

Regards Andi_HB

[quote=Andi_HB;209920]Try to use factmsieve.py its easy to use and you are using ggnfs & Msieve[/quote]

I intend to use factmsieve.py, for the 2nd and 3rd stages of processing.

I got the idea, from Jeff Gilchrist's manual, that i could do the first stage by my own, with "standard" parameters.

[quote=Andi_HB;209920]
the right command is msieve144_gpu -v -np 1[COLOR=red],[/COLOR]...
[/quote]

I didnt intend to put parameters on the -np switch.

I wrote 0x... just to specify the number in hex.

If i have to specify parameters after -np, perhaps its better to start with factsieve.py from the beginning ?

[quote=Andi_HB;209920]
1. Maybe your GPU is ready with your district -np 1,...
[/quote]

Sorry, i dont understand your reply ...

[quote=Andi_HB;209920]
2. If u use factmsieve.py you can tell the script how much you want to use your GPU in Line 69 with GPU_NUM if i see this correct.
[/quote]

I will check this ...

[quote=Andi_HB;209920]
3. For a Number with 154 digits are 280 hours on my CPU not really enough - but i don`t know how fast the polysearch on your GPU is working.

Regards Andi_HB[/quote]

How long should this stage take then ?

How can i specify this timing ?


Thank you for your reply,
scalabis

When the GPU is running CUDA programs, the screen does not refresh in windows. For reasons that are a little mysterious, interactivity of the system drops to essentially zero even though a single call to the GPU usually takes about 0.2-0.3 seconds. Windows itself has plenty of CPU time to do other things, it's just the GUI that becomes effectively unusable. The only way around that is to have the GPU run CUDA programs for very short intervals, probably at the cost of efficiency for the overall poly search. Occaisionally, when the GPU finds a polynomial that looks promising, stage 2 of the polynomial selection runs and tries to do some optimizations. That part runs completely on the CPU, so your system will become responsive again.

There are few good guidelines for how long a poly search should take when the input is large; I use two CPU weeks as a good rule of thumb, or perhaps 5% of the total expected sieving time. When you have hundreds of machines waiting for a polynomial then the search should be shorter, and when you have an enormous input and/or few machines then the search should be longer. The space that needs to be searched is gigantic, and for a 512-bit factorization you will never get through it all, so I compromise and spend the same amount of time on the search independent of how nice the GPU is. A nicer GPU will just try more polynomials, and indeed all the best polynomials found for other jobs have been found by people running high-end graphics cards.

Finally, don't expect any polynomial to be found for the first few days of the search; the bound for acceptable polynomials is pretty strict and it takes a while to find even one polynomial that exceeds the bound.

Hi Jasonp!

Thanks for replying.

Now, a few questions if you dont mind.

But first, let me quote you:
----------------------------------------------------------------------
A bigger E-value is better. For 512-bit inputs the minimum acceptable E-value is ~2.6e-12. I consider a polynomial 'good' if it has an E-value about 25% larger than that, meaning a good polynomial would be expected to speed up the sieving by at least 25%. Kleinjung has run his own code on RSA512 and found a polynomial whose E-value is 3.6e-12; that counts as spectacular.
----------------------------------------------------------------------

- So for a 512 input i should wait until i find an E value at least around ~3.2e-12, right ?

- If (on stage 2 i think) the load on the GTX285 GPU becomes nil and the load d on one of the CPU cores goes to 100%, lets say for 30 minutes, with no output on the screen, thats just normal ?

- I mean, it doesnt mean that msieve has crashed ?

- I intend to use factmsieve.py for the latter stages of this "job";

Is there any problem to start the poly stage first, with msieve144_gpu like this:

msieve144_gpu -v -np 0x...Number in Hex... ?

- Or the -np switch must have values declared for better performance ?

Thank you, for bringing me to mersenne Forum,

Best regards,
scalabis

-np with no arguments will work fine.

Whether -np should have arguments or not, i.e. whether the default search range needs to be changed for larger problems, is something I'm not sure about. jrk has anecdotal evidence that starting the search with coefficients larger than 1 leads to more hits, and possibly equally good polynomials, but lots of people have also found exceptionally good polynomials with just the defaults.

Finally, 100% CPU for a long time is expected when stage 2 runs. If something crashes, the .exe will completely stop and you'll probably get some sort of pop-up window.

Hi again!

Understood.

msieve144_gpu running for 10 hours now, of a 12 day run :smile:

Regards,
scalabis

:badspelling:

[SIZE=1]This might be relevant to your interests:[/SIZE]

I found [URL="http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx"]ProcessExplorer[/URL] for Windows!
What a relief! Now I can [I]suspend[/I] processes... (Which I always took for granted on *nix but --must confess-- thought impossible on Wind∞ze) ...finally. E.g. suspend msieve_GPU for the day - when it's my daughter's midterms or whatever.

Well, I guess it's never to late to learn.