Getting aliqueit working with GNFS

[EdH]

Originally Posted by birtwistlecaleb

My severe disability to get aliqueit to run gnfs properly makes it take a lot more effort than it should to do aliquot sequences. To prevent the tediousness of grabbing the numbers themselves, I'm going to unreserve the two sequences and find something else to do. Hopefully someone else does the same as I did with a working version of aliqueit.

Perhaps we can help in a separate thread to get your GNFS operational.

Alternatively, I have Aliqueit call CADO-NFS instead of the earlier factmsieve.py/factMsieve.pl route.

I've complicated mine further by inserting Msieve and a GPU after the CADO-NFS sieving, but I can help you write a method for just CADO-NFS, if you're interested.* It should be a separate thread, but if you mention it here, I'll branch the posts off and we can work there. I might put together another "How I..." eventually, but not in the near term and my scripts have too much going on to be of benefit by themselves. Of course, if we do a good enough job we can leave the result for other interested readers.

* If you're running Windows, it will take a little more effort, but maybe someone else can help with .bat files, if needed.

[birtwistlecaleb]

Originally Posted by EdH

Perhaps we can help in a separate thread to get your GNFS operational.

Alternatively, I have Aliqueit call CADO-NFS instead of the earlier factmsieve.py/factMsieve.pl route.

I've complicated mine further by inserting Msieve and a GPU after the CADO-NFS sieving, but I can help you write a method for just CADO-NFS, if you're interested.* It should be a separate thread, but if you mention it here, I'll branch the posts off and we can work there. I might put together another "How I..." eventually, but not in the near term and my scripts have too much going on to be of benefit by themselves. Of course, if we do a good enough job we can leave the result for other interested readers.

* If you're running Windows, it will take a little more effort, but maybe someone else can help with .bat files, if needed.

I'm running windows, does that change anything about the process above?

[Rubiksmath]

Might want to merge my reply here onto a different thread too:
I also want to run some aliquot sequences and have decent PC and GPU however have a similar problem in that no matter what I try, it seems that the compilation for literally any GPU factoring application fails somewhere which always has something to do with CUDA, and I can fix the issues up to a point but there is always a compiling snag I hit which I cannot pass with literally any GPU factoring application and any operating system running in any build environment. Obviously I am doing something wrong but don't know what. Right now as it stands if I want to kill a C100 it takes most of the day, and I know my setup is capable of way more than that.

[EdH]

I'll move these posts to a new thread in the morning and we can start addressing the issues there tomorrow. I'm tight on time tonight.

Windows will be a little challenging, because I'm really rusty with .bat files, but basically, if you can run something in a terminal (cmd), you should be able to include it in a <script>.bat file. Actually, I'm thinking CADO-NFS isn't Windows usable directly, but I may need some education on things like WSL(?). Unfortunately, my Windows VM ,( that I rarely use anyway) is right now broken due to a GPG signature issue with the repositories.

But anyway, let's tackle the issues by starting with a basic Aliqueit setup and then hit each step along the way.

Do both of you have GMP-ECM running? What about YAFU?

[VBCurtis]

Thread created; Ed's wishes are my command.

I'm also interested in getting aliqueit to call CADO; right now I have 6 sequences all running on my main machine, with aliqueit managing ECM and calling yafu for QS, or dumping to the command line for me to copy/paste the cofactor into a CADO window to break it. I don't mind this on larger cofactors because it allows me to pick composites of particular sizes for params testing with CADO, but I'd really benefit from an (existing) option in aliqueit to proceed to GNFS for cofactors below a specified cutoff; it's just that I don't work with GGNFS anymore so I'd like it to call CADO.

[Drdmitry]

I use aliqueit with ggnfs on a Windows machine. I remember it required some efforts to set it up but since then all works fine.

I suggest to first try to implement this paragraph from Getting started thread:
Download aliqueit from here. There are Windows executables and source code included. Also download GMP-ECM, Msieve, Yafu and GGNFS from the "External factoring Programs" links. Put GMP-ECM, Msieve and Yafu in your aliqueit folder. Modify the paths in aliqueit.ini so they correspond with the paths of your binaries (note: yafu path must be kept as just "yafu", it seems to crash if you use the full path). "gnfs_cutoff" defines the cutoff between running Yafu and GGNFS, and the optimum value (the point at which GGNFS becomes faster than Yafu) depends on your system. 100 is a good starting value, but it is lower for 64-bit Linux due to faster sievers.

I remember trying CADO-NFS in WSL environment. But for a number approximately 140 digits long it was slightly slower than ggnfs.

[EdH]

Thanks VBCurtis and Drdmitry. I'm glad to see some extra help.

First, a question: Is CADO-NFS an option in Windows?

Next, there are two programs which interface ggnfs with Aliqueit: factmsieve.py (Python) and factMsieve.pl (Perl). I'm more familiar with the Python one. I use a third option in linux that runs a hybrid of CADO-NFS and Msieve, to try to speed things up even more.

I know gd_barnes had some issues he worked out with factmsieve.py running in Windows. Those posts start with #605 in the Aliqueit.exe thread.

Let's first review what Aliqueit does and looks for:

aliqueit.ini has a line:

//Name (and path, if needed) of your ggnfs executable.
ggnfs_cmd =

which it uses to run GGNFS, but in order to interact with that process, it first creates a folder called "ggnfs_<aliqueit sequence>" and moves into that folder. It creates a file called test.n which has:

n: <composite>

as its only line. Then it calls whatever is given as the ggnfs_cmd.

Whatever is run as the ggnfs_cmd reads the test.n file to get the composite, factors it and returns the factors in ggnfs.log in a format readable by Aliqueit. When the called ggnfs_cmd process finishes and hands control back to Aliqueit, Aliqueit looks for that file and contents to continue. There may be some allowable variance, but my script writes the following to ggnfs.log:

Number: test
N = <composite>
factor: <factor1>
factor: <factor2>
. . .

All the rest of the GGNFS process is controlled by whatever is called by ggnfs_cmd. Whatever that process is, it needs to read test.n and write ggnfs.log to communicate with Aliqueit.

I'll wait for some more info and then post more specifics a little later.

[EdH]

Originally Posted by VBCurtis

Thread created; Ed's wishes are my command.

I'm also interested in getting aliqueit to call CADO; right now I have 6 sequences all running on my main machine, with aliqueit managing ECM and calling yafu for QS, or dumping to the command line for me to copy/paste the cofactor into a CADO window to break it. I don't mind this on larger cofactors because it allows me to pick composites of particular sizes for params testing with CADO, but I'd really benefit from an (existing) option in aliqueit to proceed to GNFS for cofactors below a specified cutoff; it's just that I don't work with GGNFS anymore so I'd like it to call CADO.

I'll go ahead and address this separately from the other post.

To call CADO-NFS in Aliqueit, I use a bash script I call "runCADO.sh." I have the following for my ggnfs_cmd for one of my machines (math99):

//Name (and path, if needed) of your ggnfs executable.
ggnfs_cmd = gnome-terminal --geometry=80x22+420+0 --wait -- bash /home/math89/Math/factorMain/runCADO.sh

The part in bold can be omitted. It is there so CADO-NFS runs in a separate terminal.

The first thing that the runCADO.sh script has to do is retrieve the composite left by Aliqueit. There are lots of ways, but I use this:

comp=$(cat test.n)
comp=${comp:3}

Next, I move into the cado-nfs directory and call CADO-NFS:

cd $HOME/Math/cado-nfs
#export OMP_DYNAMIC=true # This can be used if other programs cause CADO-NFS to stall during lingen.
./cado-nfs.py $comp <add all the things (like workdir) that are included in the cado-nfs.py call>

Again, the line in bold is not necessary, but is sometimes useful if lingen is having trouble finishing.

The final step for the script is to harvest the factors and write them to ggnfs.log. The following is the block of script I use, but it could be improved:

factors=$(cat $HOME/Math/factorMain/alicado/alicado.log | grep "Factors: ")
factors=${factors:54}
IFS=' ' read -r -a farray <<< "$factors"
numfacts=${#farray[@]}
cd $HOME/Math/factorMain/ggnfs_${comp}
echo "Number: test" >ggnfs.log
echo "N = $comp" >>ggnfs.log
fcount=1
while [ $fcount -lt $numfacts ]
  do
    echo "factor: ${farray[$fcount]}" >>ggnfs.log
    let fcount=${fcount}+1
  done

Of course, paths will need to be edited for others' systems.

[Drdmitry]

First, a question: Is CADO-NFS an option in Windows?

I've never seen a CADO-NFS executable for Windows. I once compiled it in Linux inside the WSL environment on a Windows machine. But that is not an option if you want to use CADO-NFS together with a Windows-run aliqueit.

In aliqueit.ini file one needs to provide a path to factmsieve.py in ggnfs_cmd:

ggnfs_cmd = "C:\distributed\aliqueit_new\GGNFS\factMsieve-0.86.py"

Then in factmsieve.py, one needs to update two paths: GGNFS_PATH and MSIEVE_PATH:

GGNFS_PATH = 'C:\\distributed\\aliqueit_new\\GGNFS\\'
MSIEVE_PATH = 'C:\\distributed\\aliqueit_new\\GGNFS\\'

I am pretty sure that I made some other changes but that was long ago, I do not remember.

[EdH]

Originally Posted by Drdmitry

I've never seen a CADO-NFS executable for Windows. I once compiled it in Linux inside the WSL environment on a Windows machine. But that is not an option if you want to use CADO-NFS together with a Windows-run aliqueit.
. . .
I am pretty sure that I made some other changes but that was long ago, I do not remember.

I took a brief look at wsl and it seems like a batch file could call wsl cado-nfs.py etc. Could something like:

ggnfs_cmd = runCADO.bat

allow for the Aliqueit call, and then have wsl runCADO.sh with all the extras in the .bat file?

[henryzz]

Originally Posted by Drdmitry

I've never seen a CADO-NFS executable for Windows. I once compiled it in Linux inside the WSL environment on a Windows machine. But that is not an option if you want to use CADO-NFS together with a Windows-run aliqueit.

I believe you can run WSL commands from windows by prefixing them with wsl.