Adventures in building and using yafu

[EdH]

Just to add that I haven't been able to get my Colab sessions (trunk version) to work either. I get a Segmentation fault during SIQS as well. I was waiting until I did some further testing before mentioning, but since it was brought up, I'll chime in. I'll do some more test runs tomorrow.

As for ggnfs, can you use "patch" with the patch text I have listed in this thread?:

How I Install ggnfs onto my Ubuntu Machines

Also, did you try the wip version of YAFU? In the past I've sometimes had it work with AVX2 when the trunk version didn't. I haven't tried this with my Colab instances yet, due to the cumbersome text file editing with Colab, but I hope to soon.

[James Heinrich]

Originally Posted by EdH

As for ggnfs, can you use "patch" with the patch text I have listed in this thread?: How I Install ggnfs onto my Ubuntu Machines

Explain that in simple words with examples for someone who is largely unfamiliar with *nix...

Originally Posted by EdH

Also, did you try the wip version of YAFU? In the past I've sometimes had it work with AVX2 when the trunk version didn't.

I did not, I was looking for the least-troublesome path.
If it would be helpful to you I suppose I could give it a go, but I'm reasonably-content that what I have now works at all.

[EdH]

Originally Posted by James Heinrich

Explain that in simple words with examples for someone who is largely unfamiliar with *nix...

In the simplest sense, I believe you can save the text in a file and within a terminal enter patch <filename> and patch will make all the patches, but let me verify that tomorrow with some tests and get back with you, unless someone else beats me to it.

Originally Posted by James Heinrich

I did not, I was looking for the least-troublesome path.
If it would be helpful to you I suppose I could give it a go, but I'm reasonably-content that what I have now works at all.

The "wip" version is almost as easy as the trunk version, but again, let me test some thigs tomorrow and get back to this thread.

[EdH]

Originally Posted by James Heinrich

Explain that in simple words with examples for someone who is largely unfamiliar with *nix...
. . .

I added the patched files and the suggested testPoly file to my page on installing ggnfs. I would like to know if that procedure is understandable for your level of linux knowledge:

How I Install ggnfs onto my Ubuntu Machines

You will still have to make some simple edits to two Makefiles as described. let me know if this works or is confusing.

[EdH]

@bsquared: I have temporarily attached a copy of your sievers file to my ggnfs page. I'll remove it when your link is again valid.

@James: You can get bsquared sievers on my page I referenced above, until b²'s link is again working. That will be easier than compiling your ggnfs, but not as "fun and rewarding."

[James Heinrich]

Originally Posted by EdH

You can get bsquared sievers on my page I referenced above, until b²'s link is again working. That will be easier than compiling your ggnfs, but not as "fun and rewarding."

I already had an archived copy of bsquared's sievers, I was just mentioning it so that the link could get fixed.

But in the interest of "fun and rewarding", I tried compiling. I used your patched files, made the 2x2 "k8"->"native" changes, everything went fine until the make:

make
cc -O3 -march=native -mtune=native -funroll-loops -I. -I./asm -c -DI_bits=11 -o gnfs-lasieve4eI11.o gnfs-lasieve4e.c
gnfs-lasieve4e.c: In function ‘main’:
gnfs-lasieve4e.c:622:5: warning: implicit declaration of function ‘input_poly’ [-Wimplicit-function-declaration]
     input_poly(N, poly, poldeg, poly + 1, poldeg + 1, m, fp);
     ^~~~~~~~~~

... <snip> ...

gmp-aux.w: In function ‘string2mpz’:
gmp-aux.w:91:1: warning: implicit declaration of function ‘free’ [-Wimplicit-function-declaration]
gmp-aux.w:91:1: warning: incompatible implicit declaration of built-in function ‘free’
gmp-aux.w:91:1: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
gmp-aux.w:75:1:
gmp-aux.w:91:1:
cc -O3 -march=native -mtune=native -funroll-loops -I. -I./asm -c -o mpz-ull.o mpz-ull.c
ar rcs libgmp-aux.a gmp-aux.o mpz-ull.o
cc -O3 -march=native -mtune=native -funroll-loops -I. -I./asm -L.  -o gnfs-lasieve4I11e gnfs-lasieve4eI11.o if.o input-poly.o redu2.o recurrence6.o fbgen.o real-poly-aux.o primgen32.o lasieve-prepn.o mpqs.o libgmp-aux.a liblasieve.a liblasieveI11.a -lgmp-aux -lgmp -lm -static
/usr/bin/ld: cannot find -lm
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
make: *** [Makefile:51: gnfs-lasieve4I11e] Error 1

On the off chance it matters: /usr/bin/ld --version says GNU ld version 2.30-73.el8

[EdH]

@bsquared: I did some more testing and AVX2=1 gives me a Segault for SIQS every time for the trunk version, while the wip version works fine every time.

Segfault example:

$ ./yafu "siqs(4933005920213809643173112604600914112370058384076371328403839965347428961704813433374629)" -v -v -v

11/13/20 11:48:22 v1.34.5 @ math20, System/Build Info: 
Using GMP-ECM 7.0.5-dev, Powered by GMP 6.2.0
detected Intel(R) Core(TM) i5-4570S CPU @ 2.90GHz
detected L1 = 32768 bytes, L2 = 6291456 bytes, CL = 64 bytes
measured cpu frequency ~= 2893.271740
using 1 random witnesses for Rabin-Miller PRP checks

===============================================================
======= Welcome to YAFU (Yet Another Factoring Utility) =======
=======             bbuhrow@gmail.com                   =======
=======     Type help at any time, or quit to quit      =======
===============================================================
cached 78498 primes. pmax = 999983

>> input from file = 4778555184394245735580659768930575786491800066919387684914576038684308805214696320559247
input to yafu = 4933005920213809643173112604600914112370058384076371328403839965347428961704813433374629

starting SIQS on c88: 4933005920213809643173112604600914112370058384076371328403839965347428961704813433374629
static memory usage:
    initial cycle hashtable: 16777216 bytes
    initial cycle table: 160000 bytes
    factor base: 1357440 bytes
allocated 1784 bytes for roots
allocated 0 bytes for lower mod prime
allocated 458752 bytes for sieve lines
time to compute linear sieve roots = 0.00
starting root computation over 446 to 446
time to compute bucket sieve roots = 0.00
allocated 1784 bytes for offsets for 446 sieving primes 
finding requested range 0 to 10000000
sieving range 0 to 11010048
using 446 primes, max prime = 3162  
using 2 residue classes
lines have 229376 bytes and 1835008 flags
lines broken into = 7 blocks of size 32768
blocks contain 262144 flags and cover 1572864 primes
using 462936 bytes for sieving storage
thread 0 finding primes from byte offset 0 to 229376
fb bounds: 85%
    small: 1024
    SPV: 38
    10bit: 104
    11bit: 160
    12bit: 288
    13bit: 512
    32k div 3: 664
    14bit: 944
    15bit: 1776
    med: 2528
    large: 19056
    all: 67872
start primes
    SPV: 251
    10bit: 1049
    11bit: 2083
    12bit: 4133
    13bit: 8219
    32k div 3: 11027
    14bit: 16417
    15bit: 32843
    med: 49193
    large: 458593
memory usage during sieving:
    curr_poly structure: 131152 bytes
    relation buffer: 1310720 bytes
    factor bases: 2333696 bytes
    update data: 882336 bytes
    sieve: 32768 bytes
    bucket data: 1835944 bytes

==== sieve params ====
n = 90 digits, 296 bits
factor base: 67872 primes (max prime = 1809403)
single large prime cutoff: 199034330 (110 * pmax)
double large prime range from 42 to 50 bits
double large prime range from 3273939216409 to 867102391777026
allocating 8 large prime slices of factor base
buckets hold 2048 elements
using SSE2 enabled 32k sieve core
sieve interval: 14 blocks of size 32768
polynomial A has ~ 11 factors
using multiplier of 21
using SPV correction of 21 bits, starting at offset 38
trial factoring cutoff at 93 bits

==== sieving in progress (1 thread):   67936 relations needed ====
====           Press ctrl-c to abort and save state           ====
Segmentation fault (core dumped)

Hee is my flags list (via lscpu):

Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                 r pge mca cmov pat pse36 clflush dts acpi mmx f
                                 xsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rd
                                 tscp lm constant_tsc arch_perfmon pebs bts rep_
                                 good nopl xtopology nonstop_tsc cpuid aperfmper
                                 f pni pclmulqdq dtes64 monitor ds_cpl vmx smx e
                                 st tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_
                                 1 sse4_2 x2apic movbe popcnt tsc_deadline_timer
                                  aes xsave avx f16c rdrand lahf_lm abm cpuid_fa
                                 ult epb invpcid_single pti ssbd ibrs ibpb stibp
                                  tpr_shadow vnmi flexpriority ept vpid ept_ad f
                                 sgsbase tsc_adjust bmi1 avx2 smep bmi2 erms inv
                                 pcid xsaveopt dtherm ida arat pln pts md_clear 
                                 flush_l1d

@James: The wip version shouldn't be too much more difficult than the trunk version of YAFU to compile. I think the only extra item would be to get rid of -7.3.0 or comment out the entire "CC = gcc-7.3.0" line as described on my YAFU installation page, and then leave out "x86_64" in your make command.

[EdH]

Originally Posted by James Heinrich

I already had an archived copy of bsquared's sievers, I was just mentioning it so that the link could get fixed.

But in the interest of "fun and rewarding", I tried compiling. I used your patched files, made the 2x2 "k8"->"native" changes, everything went fine until the make:

make
cc -O3 -march=native -mtune=native -funroll-loops -I. -I./asm -c -DI_bits=11 -o gnfs-lasieve4eI11.o gnfs-lasieve4e.c
gnfs-lasieve4e.c: In function ‘main’:
gnfs-lasieve4e.c:622:5: warning: implicit declaration of function ‘input_poly’ [-Wimplicit-function-declaration]
     input_poly(N, poly, poldeg, poly + 1, poldeg + 1, m, fp);
     ^~~~~~~~~~

... <snip> ...

gmp-aux.w: In function ‘string2mpz’:
gmp-aux.w:91:1: warning: implicit declaration of function ‘free’ [-Wimplicit-function-declaration]
gmp-aux.w:91:1: warning: incompatible implicit declaration of built-in function ‘free’
gmp-aux.w:91:1: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
gmp-aux.w:75:1:
gmp-aux.w:91:1:
cc -O3 -march=native -mtune=native -funroll-loops -I. -I./asm -c -o mpz-ull.o mpz-ull.c
ar rcs libgmp-aux.a gmp-aux.o mpz-ull.o
cc -O3 -march=native -mtune=native -funroll-loops -I. -I./asm -L.  -o gnfs-lasieve4I11e gnfs-lasieve4eI11.o if.o input-poly.o redu2.o recurrence6.o fbgen.o real-poly-aux.o primgen32.o lasieve-prepn.o mpqs.o libgmp-aux.a liblasieve.a liblasieveI11.a -lgmp-aux -lgmp -lm -static
/usr/bin/ld: cannot find -lm
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
 make: *** [Makefile:51: gnfs-lasieve4I11e] Error 1

On the off chance it matters: /usr/bin/ld --version says GNU ld version 2.30-73.el8

Unfortunately, this is beyond my scope to troubleshoot at this range from your machine. I may try some things later on to see if I can duplicate your error, but since it worked fine here, I'm at a loss.

Thanks for trying. Glad you have the bsquared ones to use.

[James Heinrich]

Originally Posted by EdH

The wip version shouldn't be too much more difficult than the trunk version of YAFU to compile. I think the only extra item would be to get rid of -7.3.0 or comment out the entire "CC = gcc-7.3.0" line as described on my YAFU installation page, and then leave out "x86_64" in your make command.

Indeed, it's just as easy, your instructions just scared me off. And the wip version seems to run through tune just fine, except at the end it spews out a pile of warnings (and no linebreaks) about an invalid line in yafu.ini:

Invalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsInvalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywordsfound OS = LINUX64 and CPU = Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz in tune_info field
Adding tune_info entry for LINUX64 - Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz


***factors found***

0

I'm not sure which line it's complaining about, this is my yafu.ini:

% This file is parsed by yafu during startup to apply
% options that impact various algorithms.
% NOTE:
% Command line options that have an argument
% WILL OVERRIDE anything specfied here.
% Use this file to specify commonly used options and/or as
% documentation on the available options.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% General options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Run all multi-threaded algorithms with specified thread count
threads=6
% the number of Rabin-Miller witnesses to use during PRP checks
% nprp=1
% Turn verbosity to level 1.  Default level is 0.
% Note that more v's increase the verbosity
% v
% Turn verbosity completely off
% silent
% skip testing the clock speed for 100 ms during startup
% no_clk_test
% Run the script located in the given filename
% script="my_script_file"
% specify a session logfile.  The session logfile records
% commands run, the session random seeds, System/Build info
% and some startup options
% session="session.log"
% Run the given command on numbers inside the given filename
% batchfile="my_batch_file"
% use the following random seed for this session
% seed=42
% print verbose processor info during startup
% vproc
% use the given logfile to record factorization info
% logfile="factor.log"
% set yafu to idle priority
% p
% repeat the given expression/command N times
% repeat=1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Factor options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%pretest_ratio=0.25
%xover=93
%snfs_xover
%plan
%pretest
%work
%op
%of
%ou
%no_expr
%noecm
%one
% set the decimal digit threshold below which numbers
% are proved prime using APR-CL
aprcl_p=500
% set the decimal digit threshold above which numbers
% proved prime using APR-CL have additional verbosity enabled
aprcl_d=200
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% QS options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% forceDLP
% forceTLP
% siqsLPB
% siqsMFBD
% siqsMFBT
% siqsBDiv
% siqsBT
% siqsTFSm
% qssave
% siqsB
% siqsTF
% siqsR
% siqsT
% siqsNB
% siqsM
% noopt
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NFS options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%ggnfs_dir=..\ggnfs-bin\x64\
ggnfs_dir=/root/yafu/ggnfs/bin/
% degree
% testsieve
% filt_bump
% nc1
% gnfs
% nc2
% nc3
% siever
% ncr
% lathreads
% ns
% np
% nc
% psearch
% R
% pbatch
% ggnfsT
% a
% r
%
% input job file for NFS
% job
%
% output file name for gnfs sieving
% o
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% ECM options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Set the B1 level for ECM.
B1ecm=11000
% Set the B2 level for ECM.
% Only needed if you want something other than the default
% for the current B1.
% B2ecm=1100000
%ecm_path=..\gmp-ecm\install\mingw\bin\ecm.exe
%ecm_path=../gmp-ecm/install/mingw/bin/ecm.exe
ecm_path=/usr/local/bin/ecm
% ecmtime
% prefer_gmpecm
% saveB1
% ext_ecm
% sigma
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% P-1 options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Set the B1 level for P-1.
% B1pm1=100000
% Set the B2 level for P-1.
% Only needed if you want something other than the default
% for the current B1.
% B2pm1=10000000
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% P+1 options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Set the B1 level for P+1.
B1pm1=20000
% Set the B2 level for P+1.
% Only needed if you want something other than the default
% for the current B1.
% B2pm1=2000000
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Brent-Pollard Rho options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% rhomax=200
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Fermat options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% fmtmax=1000000
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Eratosthenes options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% pfile
% pscreen
% soe_block
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Tune options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% If you run tune(), some information about the results should
% appear below here
tune_info=Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz,LINUX64,1.59078e-05,0.196092,0.299688,0.0999245,102.36,42
tune_info=Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz,LINUX64,2.14582e-05,0.198616,0.49383,0.0988064,100.63,2194.76

I'm running my pet number through it now to see if wip+AVX2 performs any faster than before.

[EdH]

I think the "complaint" is actually because it sees an empty line and I'm pretty sure it can be ignored. B² will correct me if I'm wrong on that.

At this point, are you up and running, then?

[James Heinrich]

Originally Posted by EdH

At this point, are you up and running, then?

I was up and running yesterday, until I decided to try compiling the sievers and trying the wip version for AVX2. I am back up and running, my pet number benchmark came back at 1423s, a slight 4% difference from the 1368s I got yesterday, but in the wrong direction... wip+AVX2 is now running slower than base without AVX2.