mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Msieve

Reply
 
Thread Tools
Old 2018-03-10, 23:38   #1
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

31×109 Posts
Default Choosing Between Multiple Poly Files

I'm trying to run msieve poly select on several machines at once with a short deadline and then programatically choose the best.

I can run separate machines on separate ranges with poly_deadline=300, or such, but the .fb from each machine has no score data. I can harvest the score data from all the log files, but is there a better way?

I tried breaking the tasks into the separate selection stages, but that added way too much time.

Thanks
EdH is offline   Reply With Quote
Old 2018-03-10, 23:59   #2
Dubslow
Basketry That Evening!
 
Dubslow's Avatar
 
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

3×2,399 Posts
Default

yafu ignores the .fb file, using only the .p files generated by the threads, concatenating all the individual files into one master .p file then searching it
Dubslow is offline   Reply With Quote
Old 2018-03-11, 00:25   #3
swellman
 
swellman's Avatar
 
Jun 2012

1011001110002 Posts
Default

There’s this, hope it helps.

Quote:
Originally Posted by fivemack
Just put the polynomial in msieve format into an msieve.fb file and run msieve -nc1 with no msieve.dat file present

Code:
skew 28549120.00, size 8.396e-18, alpha -6.740, combined = 6.137e-14 rroots = 3
There’s also the log file, it has an output line which includes the highest combined e-score.

Last fiddled with by swellman on 2018-03-11 at 00:32
swellman is offline   Reply With Quote
Old 2018-03-11, 16:19   #4
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

64638 Posts
Default

This is of great interest, but I can't seem to make the function stand alone without a bit more learnin' - drat!
Quote:
Originally Posted by swellman View Post
There’s this, hope it helps.
...
There’s also the log file, it has an output line which includes the highest combined e-score.
I guess I'm too ignorant to get the msieve/.fb method to work. All I get are errors or missing file notices. Is there an example of this somewhere?

I'd already considered the log file and may just have to go that route, since it's something I can do.

Thanks guys!

Last fiddled with by EdH on 2018-03-11 at 16:19
EdH is offline   Reply With Quote
Old 2018-03-11, 16:47   #5
chris2be8
 
chris2be8's Avatar
 
Sep 2009

1,913 Posts
Default

Try this:
Code:
#!/usr/bin/perl

# Code from mkpoly.pl to find the best polynomial in NAME.*.msieve.dat.p after -npr has been split over several CPUs.

use warnings;

$bestscore = 0;

$NAME=$ARGV[0];

open(IF, "$NAME.n") or die("Cannot read $NAME.n $!");
@numberinf=;
close(IF) or warn "Problem closing $NAME.n $!";
#print "$NAME.n contains:\n",@numberinf; # DEBUG

foreach my $file (glob "$NAME.*.p") {
  open(IN,"< $file") or die "Can't read $file $!";
  print "Reading $file\n"; # Debug
  my $wantit = 0; # Do we want this line?
  while() {
    my @word = split;
    # Input is like:
    # norm 1.498232e-10 alpha -5.818790 e 9.344e-10 rroots 3
    #   1      2          3       4     5   6        7     8
    if ($word[1] eq "norm") {
      $wantit = 0; # Assume we don't want this poly
      if ($bestscore < $word[6]) {
        $bestscore = $word[6];
        print "Best score so far came from $file: $_";
        $wantit = 1;
        @poly = (); # Forget last polynomial
      }
    }
    push @poly, $_ if($wantit);
    #print       $_ if($wantit); # debugging
  }
  close(IN) or warn "Problem closing $file $!";
}
if ($bestscore == 0 ) {
  die "Could not find any polynomials";
}
# Now build $NAME.poly
open(OUT,">$NAME.poly") or die "Can't open $NAME.poly $!";
chomp $numberinf[-1]; # The last line in $NAME.n may or may not end with a newline.
# So make sure it doesn't now, then always add one to $NAME.poly
print OUT @numberinf,"\n"; # Lines read from $NAME.n
print OUT @poly;           # The best polynomial
print OUT "type: gnfs\n";  # And say it's GNFS
close(OUT) or warn "Problem closing $NAME.poly $!";
It expects 1 parameter, the prefix for the files. Put the prefix.n file and all the prefix.*.p files in 1 dir, then run it.

Chris
chris2be8 is offline   Reply With Quote
Old 2018-03-11, 22:53   #6
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

31×109 Posts
Default

Quote:
Originally Posted by chris2be8 View Post
Try this:
Code:
#!/usr/bin/perl

# Code from mkpoly.pl to find the best polynomial in NAME.*.msieve.dat.p after -npr has been split over several CPUs.

use warnings;

$bestscore = 0;

$NAME=$ARGV[0];

open(IF, "$NAME.n") or die("Cannot read $NAME.n $!");
@numberinf=;
close(IF) or warn "Problem closing $NAME.n $!";
#print "$NAME.n contains:\n",@numberinf; # DEBUG

foreach my $file (glob "$NAME.*.p") {
  open(IN,"< $file") or die "Can't read $file $!";
  print "Reading $file\n"; # Debug
  my $wantit = 0; # Do we want this line?
  while() {
    my @word = split;
    # Input is like:
    # norm 1.498232e-10 alpha -5.818790 e 9.344e-10 rroots 3
    #   1      2          3       4     5   6        7     8
    if ($word[1] eq "norm") {
      $wantit = 0; # Assume we don't want this poly
      if ($bestscore < $word[6]) {
        $bestscore = $word[6];
        print "Best score so far came from $file: $_";
        $wantit = 1;
        @poly = (); # Forget last polynomial
      }
    }
    push @poly, $_ if($wantit);
    #print       $_ if($wantit); # debugging
  }
  close(IN) or warn "Problem closing $file $!";
}
if ($bestscore == 0 ) {
  die "Could not find any polynomials";
}
# Now build $NAME.poly
open(OUT,">$NAME.poly") or die "Can't open $NAME.poly $!";
chomp $numberinf[-1]; # The last line in $NAME.n may or may not end with a newline.
# So make sure it doesn't now, then always add one to $NAME.poly
print OUT @numberinf,"\n"; # Lines read from $NAME.n
print OUT @poly;           # The best polynomial
print OUT "type: gnfs\n";  # And say it's GNFS
close(OUT) or warn "Problem closing $NAME.poly $!";
It expects 1 parameter, the prefix for the files. Put the prefix.n file and all the prefix.*.p files in 1 dir, then run it.

Chris
Perhaps I am just not meant to be able to do this:
Code:
$ perl polySel.pl rsa130
syntax error at polySel.pl line 12, near "=;"
BEGIN not safe after errors--compilation aborted at polySel.pl line 16.
EdH is offline   Reply With Quote
Old 2018-03-12, 03:01   #7
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

337910 Posts
Default

I changed:
Code:
@numberinf=;
to:
Code:
@numberinf=<IF>;
and it seems to read the rsa130.n file, but now I get:
Code:
...
Use of uninitialized value $_ in split at polySel.pl line 21.
Use of uninitialized value $word[1] in string eq at polySel.pl line 25.
Use of uninitialized value $_ in split at polySel.pl line 21.
Use of uninitialized value $word[1] in string eq at polySel.pl line 25.
Use of uninitialized value $_ in split at polySel.pl line 21.
Use of uninitialized value $word[1] in string eq at polySel.pl line 25.
...
I can't see where @word gets filled...
EdH is offline   Reply With Quote
Old 2018-03-12, 16:40   #8
chris2be8
 
chris2be8's Avatar
 
Sep 2009

1,913 Posts
Default

Hello,

Sorry, html and perl don't get on very well together, bits html thinks are an unknown tag disappear. Line 20 should read:

while(<IN>) {

or try the attached version.

Chris
Attached Files
File Type: txt refindpoly.pl.txt (1.6 KB, 73 views)
chris2be8 is offline   Reply With Quote
Old 2018-03-12, 18:02   #9
xilman
Bamboozled!
 
xilman's Avatar
 
"𒉺𒌌𒇷𒆷𒀭"
May 2003
Down not across

34×53 Posts
Default

Quote:
Originally Posted by EdH View Post
I'd already considered the log file and may just have to go that route, since it's something I can do.
It's what I do.
xilman is offline   Reply With Quote
Old 2018-03-12, 20:41   #10
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

31×109 Posts
Default

Quote:
Originally Posted by chris2be8 View Post
Hello,

Sorry, html and perl don't get on very well together, bits html thinks are an unknown tag disappear. Line 20 should read:

while(<IN>) {

or try the attached version.

Chris
Ahh, Excellent! This is working quite well. Thank you!
Quote:
Originally Posted by xilman View Post
It's what I do.
I hate to reinvent in a sub-par manner, which is what I would have done in this case. I was intending to find the best poly based on log info and then use the .fb file corresponding to the best poly, which would still leave factmsieve.py to convert the .fb to .poly. The perl script from Chris can take my combined .p file and give me the .poly. That's even better than I was planning.

Thanks for all the help, everyone...
EdH is offline   Reply With Quote
Old 2018-03-15, 03:16   #11
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

31×109 Posts
Default

Thank you to everyone that helped me through this experiment.

I was able to use my scripts to run msieve/gnfs via factmsieve.py across 32 machines of very diverse antiquity and near current hardware to factor RSA140 in just under 6 (5:42) hours. Not an exceptionally great achievement for others, but a pretty darn good one for me.

Some notables:

Of the 32 machines, 2 of them tested faster at sieving using b2's 64-bit asm sievers, while 30 of them ran slightly faster with machine specific compilations. Of note, the differences in speed were actually minuscule, on the order of ~0.006 s/rel.

Although factmsieve.py suggests only using 5 to 10 machines for distributed factoring, I had 32 running in this test and 30 in a previous test and 2/3 in other tests.

All seemed to run well in these tests, except:

--with two machines, there was a considerable waste of sieving, in that the first check had 82% of the needed relations and the second check found 162%. This can be altered, depending on the size of the composite, within factmsieve.py or in the appropriate def file in ggnfs/bin. However, this modification would need to be done in all the copies on all the machines.

--with many machines of various capability, a lot of them continue sieving when the host has already started Linear Algebra, only ceasing when they report their latest batch of (unneeded) relations.

I hope to eventually document how I set everything up for these factoring runs in my How I ... blogorrhea posts.

Thanks again everyone for the help.
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
msieve poly select: choosing Stage1norm VBCurtis Msieve 0 2016-04-11 21:33
Choosing the best polynomial wombatman Msieve 123 2013-08-27 11:27
Help choosing motherboard please. Flatlander GPU Computing 4 2011-01-26 08:15
Sieving multiple NewPGen files in a row(How?) jasong Software 18 2005-12-30 20:23
Multiple systems/multiple CPUs. Best configuration? BillW Software 1 2003-01-21 20:11

All times are UTC. The time now is 06:21.

Tue Oct 27 06:21:47 UTC 2020 up 47 days, 3:32, 0 users, load averages: 2.09, 2.77, 3.67

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.