the scripts thread

[mdettweiler]

Originally Posted by rogue

I had problems with your newest script. I also think that it might have problems WRT removing entries from the ABC file. For example if you have k=123 and the ABC file has n with n=%123%, then won't egrep also remove those lines?

Would this be fixed by having the script remove, in the above example, lines matching the regular expression:

'123 ' instead of '123'

Of course, while this limits the script to numbers with "123" at the end of their k-value, it still doesn't completely ensure that there's nothing else prior to the 123 in the k. Possibly it could check to verify that the 123 is at the beginning of the line--combined with the trailing space, that would do the trick.

[Mini-Geek]

Sorry for not thinking about those sorts of problems. I made it with Riesel base 3 work in mind, where these problems wouldn't come up.

Originally Posted by rogue

I had problems with your newest script.

If you want any help, you'll have to tell me the problems. (unless you meant the issue you said next)

Originally Posted by rogue

What we really need is for srfile to be able to remove multiple sequences with a single run rather than having to run it once for each sequence.

Yeah, that'd be nice. If it could also print a list of the sequences in a file, then the need for something like my script here would be eliminated. (maybe a batch file to do everything you want could be used instead)

Originally Posted by rogue

I also think that it might have problems WRT removing entries from the ABC file. For example if you have k=123 and the ABC file has n with n=%123%, then won't egrep also remove those lines?

Originally Posted by mdettweiler

Would this be fixed by having the script remove, in the above example, lines matching the regular expression:

'123 ' instead of '123'

Of course, while this limits the script to numbers with "123" at the end of their k-value, it still doesn't completely ensure that there's nothing else prior to the 123 in the k. Possibly it could check to verify that the 123 is at the beginning of the line--combined with the trailing space, that would do the trick.

You're both right. But in pl_remain.txt the usual format is "123*..." so it would complicate things to try to match it to "k " (the k with a space after it).
There is another shorthand: ^ means the beginning of the line.
Replace the applicable line (line 26) with this to make it run a safer check for the k:

    $ks .= '^' . @linearray[0] . '[^0-9]|';

Instead of simply a space, I made it "[^0-9]" (translated: a character that is not a numeral) so that it can also work with the * from pl_remain.txt or any other separator (excluding where each k is alone on its line).
This makes the whole, generated, pattern look like:

^805817398[^0-9]|^807426766[^0-9]|...

(translated: (the beginning of a line followed by the first k followed by a character that is not a numeral) or (...))
Hopefully this should satisfy everybody. It certainly seems to me that it would work with nearly all common file formats and work types.
Edit: I removed the "?" after the "[^0-9]" since it made it match some k's that shouldn't (e.g. the code for 123 would match 1234). The only downside is it doesn't support k-only files (where each line is a single k). This could probably be added with another ^-like code if anybody thinks it useful...

[Mini-Geek]

Just so the current version is in one place, instead of having 3.0 and then a 'patch':
(added changelog)

#!/usr/bin/perl
# remove-ks.pl 3.1.1
# usage:
# have pl_remain.txt present
# have a file with one k or prime per line
# have a sieve file present, in a format that has the k on every line, e.g. PFGW/NewPGen's "k n", and not ABCD
# have egrep present
# run 'perl remove-ks.pl primefile sievefile'
# e.g. 'perl remove-ks.pl pfgw.log work.txt' uses srfile to delete all k's with primes (found in pfgw.log) from work.txt and pl_remain.txt
# the prime file can have, per line: a single k, (e.g. "1234") a prime, (e.g. "1234*3^2345-1") or a sequence (e.g. "1234*3^n-1");
# really anything, as long as it starts with the k, and if it continues, the k is followed by a *
# (e.g. "1234", "1234*3^2345-1", "1234*3^n-1", and "1234*JUNKTEXT" are all valid and will remove k=1234)

# changelog:
# 3.1.1: removed the "?" after the "[^0-9]" since it made it match some k's that shouldn't (e.g. the code for 123 would match 1234)
# 3.1: made the search pattern start on a new line and be followed by something besides a numeral to stop many possible false positives
# 3.0: added version number; rewrote to get all the k's first, then use egrep to remove them all from both files in one shot; results in a huge speed improvement
# 2.0: (unnumbered originally) uses grep to remove k's from pl_remain.txt along with using srfile to remove them from the sieve file
# 1.0: (unnumbered originally) first release, used srfile to delete k's from a sieve file

open(IN, $ARGV[0]);
$sievefile = $ARGV[1];
$ks = "";

system("echo getting list of k's...");

while(<IN>)
{
    # get ks
    $line = $_;
    chomp($line);
    @linearray = split(/\*/,$line);
    $ks .= '^' . @linearray[0] . '[^0-9]|';
}
# remove the final | character since we don't need or want it
chop($ks);

# write the list of k's, formatted as a regex search pattern, to a file
open(OUT, '>ks.txt');
print OUT $ks;
close(OUT);

system("echo done, removing k's from sieve file...");

# remove k's from sieve file
system("egrep -vhf ks.txt $sievefile > temp.txt");
system("del $sievefile");
system("ren temp.txt $sievefile");

system("echo done, removing k's from pl_remain.txt file...");

# remove k's from pl_remain.txt
system("egrep -vhf ks.txt pl_remain.txt > temp.txt");
system("del pl_remain.txt");
system("ren temp.txt pl_remain.txt");

# clean-up
system("del ks.txt");

system("echo done, exiting");

close(IN);

[Mini-Geek]

# 3.2: now supports command line arguments for what files to work on; also helpfully errors out if you don't give it sufficient input
How you give it input has changed. Now you do it with -p -s and -r command line arguments (for prime, sieve, and remaining, files, respectively). You must have -p and at least one of -s or -r. This should make it much more user-friendly.

#!/usr/bin/perl
# remove-ks.pl 3.2
# usage:
# have a file with one k or prime per line (k's to remove)
# have at least one of:
# a file of remaining k's present (e.g. pl_remain.txt)
# a sieve file present, in a format that has the k on every line, e.g. PFGW/NewPGen's "k n", and not ABCD
# 
# have egrep present
# run 'perl remove-ks.pl -p primefile -s sievefile -r remainingksfile' (-p mandatory, need at least one of -s and -r)
# e.g. 'perl remove-ks.pl -p pfgw.log -s work.txt -r pl_remain.txt' deletes all k's with primes (found in pfgw.log) from work.txt and pl_remain.txt
# or 'perl remove-ks.pl -p pfgw-prime.log -r pl_remain.txt' deletes all k's with primes (found in pfgw-prime.log) from pl_remain.txt
# the prime file can have, per line: a single k, (e.g. "1234") a prime, (e.g. "1234*3^2345-1") or a sequence (e.g. "1234*3^n-1");
# really anything, as long as it starts with the k, and if it continues, the k is followed by a *
# (e.g. "1234", "1234*3^2345-1", "1234*3^n-1", and "1234*JUNKTEXT" are all valid and will remove k=1234)

# changelog:
# 3.2: now supports command line arguments for what files to work on; also helpfully errors out if you don't give it sufficient input
# 3.1.1: removed the "?" after the "[^0-9]" since it made it match some k's that shouldn't (e.g. the code for 123 would match 1234)
# 3.1: made the search pattern start on a new line and be followed by something besides a numeral to stop many possible false positives
# 3.0: added version number; rewrote to get all the k's first, then use egrep to remove them all from both files in one shot; results in a huge speed improvement
# 2.0: (unnumbered originally) uses grep to remove k's from pl_remain.txt along with using srfile to remove them from the sieve file
# 1.0: (unnumbered originally) first release, used srfile to delete k's from a sieve file

use Getopt::Std;
getopt('spr');

if (length($opt_p) < 1) { # if no prime file was specified
    print "must supply a file with primes (k's to remove), usage: -p filename.txt\n";
    exit;
}
if (length($opt_s) < 1) {
    $remsieve = 0;
} else {
    $sievefile = $opt_s;
    $remsieve = 1;
}
if (length($opt_r) < 1) {
    $remremain = 0;
} else {
    $remainfile = $opt_r;
    $remremain = 1;
}
if ((!$remremain) && (!$remsieve)) { # if neither sieve nor remain file was specified
    print "must supply at least one file to remove from, usage: -r pl_remain.txt or -s sievefile.txt\n";
    exit;
}
$kstxt = "temp-ks.txt";
if ($remainfile == $kstxt) { # just in case the list of remaining k's is at temp-ks.txt, change our temp list
    $kstxt = "temp-ks-alt.txt";
}


open(IN, $opt_p);

print "getting list of k's to remove from $opt_p...\n";

while(<IN>)
{
    # get ks
    $line = $_;
    chomp($line);
    @linearray = split(/\*/,$line);
    $ks .= '^' . @linearray[0] . '[^0-9]|';
}
# remove the final | character since we don't need or want it
chop($ks);

# write the list of k's, formatted as a regex search pattern, to a file
open(OUT, ">$kstxt");
print OUT $ks;
close(OUT);

if ($remremain) {
print "done, removing k's from $remainfile...\n";

# remove k's from remaining k's file
system("egrep -vhf $kstxt $remainfile > temp.txt");
system("del $remainfile");
system("ren temp.txt $remainfile");
}

if ($remsieve) {
print "done, removing k's from $sievefile...\n";

# remove k's from sieve file
system("egrep -vhf $kstxt $sievefile > temp.txt");
system("del $sievefile");
system("ren temp.txt $sievefile");
}

# clean-up
system("del $kstxt");

print "done, exiting\n";

close(IN);

I also noticed that this statement was mistaken:

Originally Posted by Mini-Geek

Edit: I removed the "?" after the "[^0-9]" since it made it match some k's that shouldn't (e.g. the code for 123 would match 1234). The only downside is it doesn't support k-only files (where each line is a single k). This could probably be added with another ^-like code if anybody thinks it useful...

It still works when each line is just a single k. So it works properly with every format I can think of where the k comes at the start of each line.
e.g. it properly recognizes every instance of 123 in this file:

123 456
1123 456
456 123
7456 789
1234 987
123
1234
123*3^n-1
1234*3^n-1
4123

[rogue]

I have a a mods to srfile so that the -d argument can take an input file or an sequence. The input file would be a file such as pfgw.log, which has a sequence (or number) on each line. It will delete all candidates from the input file before saving. This should be a huge time saver if you have very large files from srsieve/sr2sieve. I've forwarded the changes to Geoff, but his focus has been elsewhere so it is unknown if and/or when it will make it into a release. If anyone is interested I can post the code changes here.

[gd_barnes]

Originally Posted by rogue

I had problems with your newest script. I also think that it might have problems WRT removing entries from the ABC file. For example if you have k=123 and the ABC file has n with n=%123%, then won't egrep also remove those lines?

What we really need is for srfile to be able to remove multiple sequences with a single run rather than having to run it once for each sequence.

You've used the acronym WRT numerous times. Please clarify what it means.

I googled it to no avail.

[Batalov]

'with respect to'?

[Xentar]

http://www.acronymfinder.com/WRT.html

[gd_barnes]

Thanks for the info guys. That's a cool link Xentar. I've bookmarked the page for future confusing acronyms. :-)

[Mini-Geek]

Originally Posted by Batalov

A script for algebraic elimination hints for pl_remain.txt
Attachment 4870

If it reports
* x^3 ...
then it means the whole k can be eliminated (for a hypothetical example 8*125^n+1) -- practice shows that even squares are occasionally missed.

If it says
1|3 46^3 184*529^n+1
then just eliminate n=1 (mod 3) from the ...b529_k184... file

copied here

[Mini-Geek]

Originally Posted by Batalov

A script for algebraic elimination hints for pl_remain.txt
Attachment 4870

If it reports
* x^3 ...
then it means the whole k can be eliminated (for a hypothetical example 8*125^n+1) -- practice shows that even squares are occasionally missed.

If it says
1|3 46^3 184*529^n+1
then just eliminate n=1 (mod 3) from the ...b529_k184... file

A very cool and useful script, once I figured out how to use it.
For anybody else trying to figure it out, it works off of standard input or by one or two file names added as arguments (or at least that's what seems to be the case), and takes sequences in a form like 8*125^n+1 (one per line), as you'd find in pl_remain.txt after running the PFGW script.
So you can run "hiddenPowers.pl pl_remain.txt" or "hiddenPowers.pl" followed by entering one or more sequences on standard input.
I found a minor bug: when the file has no trailing newline and the final number has more than one elimination, it does not put a newline between the lines of the last sequence's eliminations, making it run together confusingly.
e.g. with this file:

2*18^n-1
64*177^n-1

(with no trailing newline) it outputs:

1|2 6^2 2*18^n-1
0|2 8^2 64*177^n-10|3 4^3       64*177^n-1

Using a chomp on "$_" and then adding "\n"'s (newlines) where needed fixes the problem.

I thought a more human-readable output would be better, so I made it look like this (with my above bug fixed):

2*18^n-1        n=1 mod 2 factors due to 6^2
64*177^n-1      n=0 mod 2 factors due to 8^2
64*177^n-1      n=0 mod 3 factors due to 4^3
8*125^n-1       every n factors due to x^3

If you, or anybody else, is interested, here's the code:

#!/usr/bin/perl -w
# you can use this script on pl_remain.txt
# created by Batalov, modified by Mini-Geek
use Math::BigInt;

line: while(<>) {
  chomp($_);
  next unless /^(\d+)\*(\d+)\^n([+-])1/ && $1 && $2;
  $k = Math::BigInt->new($1);
  $a = Math::BigInt->new($2);
  my @powers = ($3 eq '+') ? qw(3 5 7 11) : qw(2 3 5 7 11);
  
  foreach $m (@powers) {
    $b = $k->copy()->broot($m);
    if ($b->copy()->bpow($m) == $k) {
      if ($a->copy()->broot($m)->bpow($m) == $a) {
    print "$_\tevery n factors due to x^$m\n";
    next line;
      }
      print "$_\tn=0 mod $m factors due to $b^$m\n";
    }
  }
  for ($n=1; $n<$powers[$#powers]; $n++) {
    $a->bmul($k);
    foreach $m (@powers) {
      next if $n>=$m;
      $b = $a->copy()->broot($m);
      print "$_\tn=$n mod $m factors due to $b^$m\n" if  ($b->copy()->bpow($m) == $a);
    }
  }
}