Database Flooding

[Antonio]

Yesterday morning (all references UK local time) the smallest unfactored composites were at around C87. Last night I noticed a large number of composites were present at and below C70.

Before I went to bed I set my machine to clearing some of these, this morning it is still churning through them without making to much headway having cleared more than 6000 composites: -

 get composite (offset=0)
 Factoring 70 digits: 1741997560928006593195882978088900114959217347526265091892837922316583
P36 = 281021295403800708823233027729341771
P34 = 6198809803452520566818074644516373
*****
===========================================================================
runtime (H:M:S).................: 009:00:22
composite range.................: 65 - 77 digits
 report factors for composite #..: 6439
        New factors added.......: 1 / 7787
        Factors already known...: 1 / 3750
        Small factors...........: 0 / 3506
        Errors (does not divide): 0 / 0
        No composites received..: 0
        No results found........: 0
===========================================================================
  714.9 composites/hr
  714.0 queries/hr
  930.4 queries/hr (last 60 composites)

(Before anyone asks, I had set the lower composite bound to C65 for software testing, I know the database works on composites < C70).

Looks like some extra effort may be required to clear this latest flood of data.

[Jayder]

I noticed this too. More than 20k unfactored numbers ≤120 digits added today, not including the thousands you and others have factored, with no sign of it slowing down.

That's a cool script you have there. Mine doesn't show those stats. Is it available for download?

[Antonio]

Originally Posted by Jayder

That's a cool script you have there. Mine doesn't show those stats. Is it available for download?

The script is a development of the one provided by firejuggler at http://mersenneforum.org/showpost.ph...3&postcount=65

I have used it as a basis for learning about Perl, so you will have to forgive any silly programming, but here it is (you probably need to change the first line to point to your perl executable).

fdb.pl :-

    
#Factordb composite evaluator version 1.1 - 30 May 2014
#!D:\strawberry\perl\bin\perl -w
 
use warnings;
use strict;
use LWP::Simple;
use Term::ReadKey;
use Time::HiRes qw(sleep time);
use Getopt::Std;
 
my $mindig=70; # minimum size of composite to get
my $maxdig=97; # maximum size of composite to factor
my @numtoget=(1, 1, 5); # composites to get per call [current, normal, slow]
my $shortstop=60; # number of time samples to hold for short term query rate calculations
my $range=1000; # maximum random offset into composite data list, used to reduce collisions between multiple users
 
my $factoredn=0;
my $queries=0;
my $elapsed=0;
my @short;
my $mincomposite=100000;
my $maxcomposite=0;
my $nofcount=0;
my $knowncount=0;
my $addcount=0;
my $nocomposites=0;
my $noresults=0;
my $smallcount=0;
my $startime;
my $key;
my $yafutext=1;
 
# declare the perl command line flags/options
my %options=();
getopts("hm:M:r:fq", \%options);
 
# test for the existence of the options on the command line.
if (defined $options{h}) {
 print "\t-m\tset minimum composite size (digits) (default=70)\n\t-M\tset maximum composite size (digits) (default=97)\n";
 print "\t-r\tset maximum random offset into composite list (range is 0 to (value given - 1) (default=1000)\n";
 print "\t-f\tflag - if present YAFU files (session.log, factor.log & siqs.dat)\n\t\tare deleted when the program terminates.\n";
 print "\t-q\tflag - if present YAFU progress text is suppressed.\n\n";
 die;
}
if (defined $options{m}) {$mindig = $options{m};}
if (defined $options{M}) {$maxdig = $options{M};}
if (defined $options{r}) {$range = $options{r};}
print"\tminimum composite size= $mindig digits\n\tmaximum composite size= $maxdig digits\n";
print "\trandom offset into composite list= 0 to " . (($range>0) ? sprintf("%d",($range-1)) : 0) . "\n";
if (defined $options{f}) {print "\tDelete YAFU log files when done.\n";}
if (defined $options{q}) {print "\tSuppress YAFU progress display.\n"; $yafutext=0;}
print "\n\tCtrl-Q to exit after factoring current composite(s)\n";
sleep(5);
 
ReadMode 'raw';
 
$startime=time();
 
do{
    my $current=time();
    # Store the start time for up to '$shortstop' queries so we can calculate the short term average rate later
 if (scalar(@short) == $shortstop) {shift(@short);}
 push(@short,$current);
 
 my $rand=int(rand($range));
    print "\nget composite";
    ($numtoget[0]>1) ? (print "s (offset=$rand)\n") : (print " (offset=$rand)\n");
 
 my $contents = get("http://factorization.ath.cx/listtype.php?t=3&mindig=$mindig&perpage=$numtoget[0]&start=$rand&download=1");
    $queries += 1;
 
 if (!defined $contents or $contents =~ /[a-z]/ ){
        if (defined $contents){print "$contents\n";}
        print "Error, no composites fetched\n";
        $nocomposites +=1;
        sleep(60);
    }else {
 
    my @composites=split(/\s/, $contents);
 
    # Sort into ascending size - if more than one composite is requested
       if ($numtoget[0] > 1 ) {@composites = sort{$a<=>$b}@composites;}
 
    foreach my $composite (@composites) {
           my $compositelen=length($composite);
 
        if ($compositelen > $maxcomposite){ $maxcomposite = $compositelen;}
           if ($compositelen < $mincomposite){ $mincomposite = $compositelen;}
 
        print "\nFactoring ".$compositelen." digits: $composite\n";
           my @results;
           open(YAFU, "yafu factor($composite) -p|") or die "\n\tCouldn't start yafu!";
           while (<YAFU>) {
      if ($yafutext){ print "$_";}
                   chomp;
               if (/^[CP].*? = (\d+)/) {
                 push( @results, $1 );
                 if (!$yafutext){print "$_\n";}
              }
           }
           print "*****\n";
           close(YAFU);
 
        if ( scalar(@results) > 0 ) {
            # Sort factors into descending size order for reporting to database
             @results = sort{$b<=>$a}@results;
 
           $factoredn += 1;
              my $elapse = time()-$startime;
              my $hours= int($elapse /3600);
              my $minutes= int($elapse/60)%60;
              my $seconds= $elapse%60;
 
           my $url="http://factorization.ath.cx/report.php?report=".$composite."%3D".join('*',@results);
              $contents=get($url);
 
           my $nofactors     = ($contents =~ s/Does not divide//g);
              my $already_known = ($contents =~ s/Factor already known//g);
              my $added         = ($contents =~ s/Factor added//g);
              my $small         = ($contents =~ s/Small factor//g);
 
           $nofcount += $nofactors;
              $knowncount += $already_known;
              $addcount += $added;
              $smallcount += $small;
 
           print "===========================================================================\n";
              print "runtime (H:M:S).................: " . sprintf("%03d",$hours) . ":" . sprintf("%02d",$minutes) . ":" . sprintf("%02d",$seconds) . "\n";
              print "composite range.................: " . "$mincomposite - $maxcomposite digits\n\n";
              print "report factors for composite #..: " . $factoredn . "\n";
              print "\tNew factors added.......: " . ($added         ? $added         : 0) . " / " . $addcount . "\n";
              print "\tFactors already known...: " . ($already_known ? $already_known : 0) . " / " . $knowncount . "\n";
              print "\tSmall factors...........: " . ($small         ? $small         : 0) . " / " . $smallcount . "\n";
              print "\tErrors (does not divide): " . ($nofactors     ? $nofactors     : 0) . " / " . $nofcount . "\n";
              print "\tNo composites received..: " . ($nocomposites  ? $nocomposites  : 0) . " \n";
              print "\tNo results found........: " . ($noresults     ? $noresults     : 0) . " \n";
              print "===========================================================================\n";
          }else {
              print "Error in YAFU, no factors returned\n";
              $noresults +=1;
          }
      }
   }
 
# Now check for key press
   $key = ReadKey(-1);
 
# Display current database query rate
   $current=time();
 
$elapsed = $current-$startime; # Time to process all calls to database so far this run
 
print sprintf("%7.1f",$factoredn*3600/$elapsed) . " composites/hr\n";
   print sprintf("%7.1f",$queries*3600/$elapsed) . " queries/hr\n";
 
# Now try to ensure average rate does not exceed 1500 database queries per hour
   # Use average of last "$shortstop" factoring times
   if (scalar(@short) == $shortstop){
 
 my $hourate = 3600/(($current- $short[0])/scalar(@short));
    print sprintf("%7.1f", $hourate) . " queries/hr (last " . scalar(@short) . " composites)\n";
 
 if (($hourate > 1500) and ($numtoget[0] == $numtoget[2])){
     #Should take (2.4*number of samples) sec. at 1500/hour, actually took ($current-$short[0]) sec. so wait the remaining time.
     my $delay = 2.4*$shortstop-($current-$short[0]);
     print "Waiting ..." . sprintf("%6.2f",$delay) . " sec.\n";
     sleep($delay);
    }
 
 if ($hourate > 1200){$numtoget[0] = $numtoget[2];}
 
 if ($hourate < 1000){$numtoget[0] = $numtoget[1];}
   }
 
# Check to see if Ctrl-Q has been pressed or max digits exceeded, if not then continue otherwise quit program
} while((!$key or (ord($key) != 17)) and ($maxcomposite <= $maxdig));
 
if (defined $options{f}) {
 # delete YAFU files
 unlink("siqs.dat");
 unlink("factor.log");
 unlink("session.log");
}
 
ReadMode 'restore';
 
if ($maxcomposite > $maxdig){
 print "\n\tMaximum composite size exceeded\n";
}else{
 print "\n\tCtrl-Q detected\n";
}
 
die;

command line options are included, fdb.pl -h will display the help.

[Antonio]

Ok, 5 hours later and losing the battle
Was processing in the C70 to C75 range, now in C65 to C70 country:-

Factoring 65 digits: 94258811060627364077273933265497365205730255757960213690248625919
P36 = 721166514152406098377209697331301941
P30 = 130703255365940896808959135459
*****
===========================================================================
runtime (H:M:S).................: 014:18:35
composite range.................: 65 - 77 digits
 report factors for composite #..: 11802
        New factors added.......: 1 / 14255
        Factors already known...: 1 / 6571
        Small factors...........: 0 / 6940
        Errors (does not divide): 0 / 0
        No composites received..: 0
        No results found........: 0
===========================================================================
  824.7 composites/hr
  778.1 queries/hr
  969.3 queries/hr (last 60 composites)

[ET_]

Originally Posted by Antonio

The script is a development of the one provided by firejuggler at http://mersenneforum.org/showpost.ph...3&postcount=65

I have used it as a basis for learning about Perl, so you will have to forgive any silly programming, but here it is (you probably need to change the first line to point to your perl executable).

fdb.pl :-

command line options are included, fdb.pl -h will display the help.

I tested your script on my linux machine, exchanging "yafu" with "yafu-linux64" around line 93 (I have Yafu 1.33).

From yafu docfile.txt, I see that yafu requires double quotes for its command line parameters (i.e. yafu "factor($composite)" ).

I tried both methods (with and without quotes) but the script keep yelling at me:

without quotes: sh: 1: Syntax error: "(" unexpected
with escaped quotes (\"): Couldn't start yafu! at ./factor_db_yafu.pl line 93.

What am I missing?

Luigi

[Antonio]

Originally Posted by ET_

I tested your script on my linux machine, exchanging "yafu" with "yafu-linux64" around line 93 (I have Yafu 1.33).

From yafu docfile.txt, I see that yafu requires double quotes for its command line parameters (i.e. yafu "factor($composite)" ).

I tried both methods (with and without quotes) but the script keep yelling at me:

without quotes: sh: 1: Syntax error: "(" unexpected
with escaped quotes (\"): Couldn't start yafu! at ./factor_db_yafu.pl line 93.

What am I missing?

Luigi

I can't see anything wrong in the script (the call to YAFU is unchanged from firejuggler's original script). I assume you are running the script from your YAFU directory?
Have you used firejuggler's script pointed to in the previous post?

I'm running under windows and do not have a Linux machine available, so I can't test the script in that environment, sorry.

[EdH]

To run firejuggler's script in linux, I need to change line 21 from:

        open(YAFU, "yafu factor($composite) |") or die "Couldn't start yafu!";

to:

        open(YAFU, "./yafu \"factor($composite)\" |") or die "Couldn't start yafu!";

Red shows three additions and my yafu executable is named yafu.

Edit: Forgot to add that I can't get Antonio's script to run without digging into it. I get a large list of errors, possibly because I'm missing some modules.

[Antonio]

Originally Posted by EdH

Edit: Forgot to add that I can't get Antonio's script to run without digging into it. I get a large list of errors, possibly because I'm missing some modules.

Just to make it clear my development environment is:

Windows 7 Home Premium 64bit.
Strawberry Perl 5.18.1.1 64bit.

I haven't added any modules, everything I have used is included in the distribution.

Sorry I'm not much help, as I said earlier, this was just a Perl learning exercise for me.

[EdH]

Originally Posted by Antonio

Just to make it clear my development environment is:

Windows 7 Home Premium 64bit.
Strawberry Perl 5.18.1.1 64bit.

I haven't added any modules, everything I have used is included in the distribution.

Sorry I'm not much help, as I said earlier, this was just a Perl learning exercise for me.

Hi Antonio,

To make yours work here at all, I had to add the following line:

my $range=0; my @numtoget=0; my %options; my $maxdig;

after line 8 and add two braces ( }} ) to the very end. However, those braces are not aligned properly to enclose the appropriate segments and after a successful factor completion the code crashes:

***factors found***

P21 = 143409033092276013677
P21 = 676000456823857554787
P31 = 1500555505082519280625039989167

ans = 1

*****
===========================================================================
runtime (H:M:S).................: 000:00:03
composite range.................: 72 - 72 digits

report factors for composite #..: 1
    New factors added.......: 2 / 2
    Factors already known...: 1 / 1
    Small factors...........: 0 / 0
    Errors (does not divide): 0 / 0
    No composites received..: 0 
    No results found........: 0 
===========================================================================
1161499569.2 composites/hr
1161499569.2 queries/hr
Use of uninitialized value in subtraction (-) at antonioScript.pl line 59.
Illegal division by zero at antonioScript.pl line 59.

Do note that the above line 59 is your original line 58, due to my addition of a new line 9.

Thanks for your code to study/work with.

[ET_]

Originally Posted by EdH

To run firejuggler's script in linux, I need to change line 21 from:

        open(YAFU, "yafu factor($composite) |") or die "Couldn't start yafu!";

to:

        open(YAFU, "./yafu \"factor($composite)\" |") or die "Couldn't start yafu!";

Red shows three additions and my yafu executable is named yafu.

Edit: Forgot to add that I can't get Antonio's script to run without digging into it. I get a large list of errors, possibly because I'm missing some modules.

I resolved the escaped quotes, but didn't add the dot-slash.

Now the script is happily crunching, thanks Antonio!

Luigi

[Antonio]

Originally Posted by EdH

Hi Antonio,

To make yours work here at all, I had to add the following line:

my $range=0; my @numtoget=0; my %options; my $maxdig;

after line 8 and add two braces ( }} ) to the very end. However, those braces are not aligned properly to enclose the appropriate segments and after a successful factor completion the code crashes:

***factors found***

P21 = 143409033092276013677
P21 = 676000456823857554787
P31 = 1500555505082519280625039989167

ans = 1

*****
===========================================================================
runtime (H:M:S).................: 000:00:03
composite range.................: 72 - 72 digits

report factors for composite #..: 1
    New factors added.......: 2 / 2
    Factors already known...: 1 / 1
    Small factors...........: 0 / 0
    Errors (does not divide): 0 / 0
    No composites received..: 0 
    No results found........: 0 
===========================================================================
1161499569.2 composites/hr
1161499569.2 queries/hr
Use of uninitialized value in subtraction (-) at antonioScript.pl line 59.
Illegal division by zero at antonioScript.pl line 59.

Do note that the above line 59 is your original line 58, due to my addition of a new line 9.

Thanks for your code to study/work with.

I tried importing the script and running it myself, and it would appear that the formatting has gone haywire causing at least some of the problems!

I have tried reposting the script to see if copies correctly this time (it appears to on my machine from the 'Preview Post').

#! D:\strawberry\perl\bin\perl -w
#Factordb composite evaluator version 1.1 - 30 May 2014
 use warnings;
use strict;
use LWP::Simple;
use Term::ReadKey;
use Time::HiRes qw(sleep time);
use Getopt::Std;
 my $mindig=70; # minimum size of composite to get
my $maxdig=97; # maximum size of composite to factor
my @numtoget=(1, 1, 5); # composites to get per call [current, normal, slow]
my $shortstop=60; # number of time samples to hold for short term query rate calculations
my $range=1000; # maximum random offset into composite data list, used to reduce collisions between multiple users
 my $factoredn=0;
my $queries=0;
my $elapsed=0;
my @short;
my $mincomposite=100000;
my $maxcomposite=0;
my $nofcount=0;
my $knowncount=0;
my $addcount=0;
my $nocomposites=0;
my $noresults=0;
my $smallcount=0;
my $startime;
my $key;
my $yafutext=1;
 # declare the perl command line flags/options
my %options=();
getopts("hm:M:r:fq", \%options);
 # test for the existence of the options on the command line.
if (defined $options{h}) {
 print "\t-m\tset minimum composite size (digits) (default=70)\n\t-M\tset maximum composite size (digits) (default=97)\n";
 print "\t-r\tset maximum random offset into composite list (range is 0 to (value given - 1) (default=1000)\n";
 print "\t-f\tflag - if present YAFU files (session.log, factor.log & siqs.dat)\n\t\tare deleted when the program terminates.\n";
 print "\t-q\tflag - if present YAFU progress text is suppressed.\n\n";
 die;
}
if (defined $options{m}) {$mindig = $options{m};}
if (defined $options{M}) {$maxdig = $options{M};}
if (defined $options{r}) {$range = $options{r};}
print"\tminimum composite size= $mindig digits\n\tmaximum composite size= $maxdig digits\n";
print "\trandom offset into composite list= 0 to " . (($range>0) ? sprintf("%d",($range-1)) : 0) . "\n";
if (defined $options{f}) {print "\tDelete YAFU log files when done.\n";}
if (defined $options{q}) {print "\tSuppress YAFU progress display.\n"; $yafutext=0;}
print "\n\tCtrl-Q to exit after factoring current composite(s)\n";
sleep(5);
 ReadMode 'raw';
 $startime=time();
 do{
    my $current=time();
    # Store the start time for up to '$shortstop' queries so we can calculate the short term average rate later
 if (scalar(@short) == $shortstop) {shift(@short);}
 push(@short,$current);
     my $rand=int(rand($range));
    print "\nget composite";
    ($numtoget[0]>1) ? (print "s (offset=$rand)\n") : (print " (offset=$rand)\n");
     my $contents = get("http://factorization.ath.cx/listtype...page=$numtoget[0]&start=$rand&download=1");
    $queries += 1;
     if (!defined $contents or $contents =~ /[a-z]/ ){
        if (defined $contents){print "$contents\n";}
        print "Error, no composites fetched\n";
        $nocomposites +=1;
        sleep(60);
    }else {
        my @composites=split(/\s/, $contents);
        # Sort into ascending size - if more than one composite is requested
       if ($numtoget[0] > 1 ) {@composites = sort{$a<=>$b}@composites;}
        foreach my $composite (@composites) {
           my $compositelen=length($composite);
            if ($compositelen > $maxcomposite){ $maxcomposite = $compositelen;}
           if ($compositelen < $mincomposite){ $mincomposite = $compositelen;}
            print "\nFactoring ".$compositelen." digits: $composite\n";
           my @results;
           open(YAFU, "yafu factor($composite) -p|") or die "\n\tCouldn't start yafu!";
           while (<YAFU>) {
      if ($yafutext){ print "$_";}
                   chomp;
               if (/^[CP].*? = (\d+)/) {
                 push( @results, $1 );
                 if (!$yafutext){print "$_\n";}
              }
           }
           print "*****\n";
           close(YAFU);
            if ( scalar(@results) > 0 ) {
            # Sort factors into descending size order for reporting to database
             @results = sort{$b<=>$a}@results;
               $factoredn += 1;
              my $elapse = time()-$startime;
              my $hours= int($elapse /3600);
              my $minutes= int($elapse/60)%60;
              my $seconds= $elapse%60;
               my $url="http://factorization.ath.cx/report.php?report=".$composite."%3D".join('*',@results);
              $contents=get($url);
               my $nofactors     = ($contents =~ s/Does not divide//g);
              my $already_known = ($contents =~ s/Factor already known//g);
              my $added         = ($contents =~ s/Factor added//g);
              my $small         = ($contents =~ s/Small factor//g);
               $nofcount += $nofactors;
              $knowncount += $already_known;
              $addcount += $added;
              $smallcount += $small;
               print "===========================================================================\n";
              print "runtime (H:M:S).................: " . sprintf("%03d",$hours) . ":" . sprintf("%02d",$minutes) . ":" . sprintf("%02d",$seconds) . "\n";
              print "composite range.................: " . "$mincomposite - $maxcomposite digits\n\n";
              print "report factors for composite #..: " . $factoredn . "\n";
              print "\tNew factors added.......: " . ($added         ? $added         : 0) . " / " . $addcount . "\n";
              print "\tFactors already known...: " . ($already_known ? $already_known : 0) . " / " . $knowncount . "\n";
              print "\tSmall factors...........: " . ($small         ? $small         : 0) . " / " . $smallcount . "\n";
              print "\tErrors (does not divide): " . ($nofactors     ? $nofactors     : 0) . " / " . $nofcount . "\n";
              print "\tNo composites received..: " . ($nocomposites  ? $nocomposites  : 0) . " \n";
              print "\tNo results found........: " . ($noresults     ? $noresults     : 0) . " \n";
              print "===========================================================================\n";
          }else {
              print "Error in YAFU, no factors returned\n";
              $noresults +=1;
          }
      }
   }
    # Now check for key press
   $key = ReadKey(-1);
    # Display current database query rate
   $current=time();
    $elapsed = $current-$startime; # Time to process all calls to database so far this run
    print sprintf("%7.1f",$factoredn*3600/$elapsed) . " composites/hr\n";
   print sprintf("%7.1f",$queries*3600/$elapsed) . " queries/hr\n";
    # Now try to ensure average rate does not exceed 1500 database queries per hour
   # Use average of last "$shortstop" factoring times
   if (scalar(@short) == $shortstop){
     my $hourate = (3600*$shortstop) / ($current-$short[0]);
    print sprintf("%7.1f", $hourate) . " queries/hr (last $shortstop composites)\n";
     if (($hourate > 1500) and ($numtoget[0] == $numtoget[2])){
     #Should take (2.4*number of samples) sec. at 1500/hour, actually took ($current-$short[0]) sec. so wait the remaining time.
     my $delay = 2.4*$shortstop-($current-$short[0]);
     print "Waiting ..." . sprintf("%6.2f",$delay) . " sec.\n";
     sleep($delay);
    }
     if ($hourate > 1200){$numtoget[0] = $numtoget[2];}
     if ($hourate < 1000){$numtoget[0] = $numtoget[1];}
   }
 # Check to see if Ctrl-Q has been pressed or max digits exceeded, if not then continue otherwise quit program
} while((!$key or (ord($key) != 17)) and ($maxcomposite <= $maxdig));
 if (defined $options{f}) {
 # delete YAFU files
 unlink("siqs.dat");
 unlink("factor.log");
 unlink("session.log");
}
 ReadMode 'restore';
 if ($maxcomposite > $maxdig){
 print "\n\tMaximum composite size exceeded\n";
}else{
 print "\n\tCtrl-Q detected\n";
}
 die;

Just attached my working script as a text document to be double sure.