[QUOTE=scalabis;210050]I'm using an i7 795 quad core CPU, (as you know8 CPU's declared by hiperthreading).

Should i try the "32bit P4 lattice siever" versions, or is this an old issue ?[/QUOTE]

Every system is slightly different depending on what speed CPU, RAM, motherboard, etc...

So you should try both the 32bit and 64bit versions on your system to see which is faster for you.

Jeff.

[quote=Jeff Gilchrist;210060]Every system is slightly different depending on what speed CPU, RAM, motherboard, etc...

So you should try both the 32bit and 64bit versions on your system to see which is faster for you.

Jeff.[/quote]

Ok.

So i had he 64 bit Windows version installed.

I got for a test LatSieveTime: 3844.16

Then i replaced only the 6 gnfs-lasieve4I1?e.exe files for the 32 bit Windows versions and i started again exactly the same test.

LatSieveTime: 3623.89

So the 32 bit versions are faster on this machine.

Question: Should i use only the files from the 32 bit version, or is there any "advantage" on using the files from the 64 bit version, replacing only the 6 executables files mentioned above ?

Regards,
scalabis

Can you try linux x64 versions of siever to test speed?
Seems it will be 1.5 - 2 timeas faster casue of optimized binaries

You can't run Linux sievers on Windows...

Hi!

This is a 64 bit Windows XP machine, and it must stay as it is.

I don't have another decent platform, right now, to install and test Linux versions.

But the question still stands:

Should i use a mix of 64 and 32 bit files ?

Or as the 32 bit siever versions are faster, should i only use Jeff Gilchrist's compiled 32 bit Windows version files ?

Thanks.

Regards,
scalabis

[quote=10metreh;210392]You can't run Linux sievers on Windows...[/quote]
You can run Linux on a virtual machine running on Windows...

Paul

[QUOTE=xilman;210403]You can run Linux on a virtual machine running on Windows...

Paul[/QUOTE]

Yes. I'm running 64 bit Linux under 32 bit Windows XP with no problems (AMD64 cpu).
Here's a question though - can a 64 bit guest OS access ram that the 32 bit host OS can't access? I'm trying to decide whether it's worth upgrading my PC to 4Gb ...

Chris

[quote=Chris Card;210406]Yes. I'm running 64 bit Linux under 32 bit Windows XP with no problems (AMD64 cpu).
Here's a question though - can a 64 bit guest OS access ram that the 32 bit host OS can't access? I'm trying to decide whether it's worth upgrading my PC to 4Gb ...

Chris[/quote]I don't know. It may depend on which VM hosting software you are running. The VirtualBox documentation is silent on the issue.

Best guess is that 64-bit memory access on a 32-bit host is unlikely to work. Unfortunately I don't have a suitable system on which to perform the experiment.

Paul

[quote=Chris Card;210406]Here's a question though - can a 64 bit guest OS access ram that the 32 bit host OS can't access? I'm trying to decide whether it's worth upgrading my PC to 4Gb ...[/quote]
I've been told that in such a situation you can't get the speed increases available with a 64-bit OS. I think that, similarly, you wouldn't be able to access RAM that the 32-bit OS can't.

Script to use msieve for polynomial selection.
 

Hello,

Here is a script to factor numbers by GNFS using msieve to select the polynomial:
[code] #!/bin/sh

# This script will factor a number in $1.n via GNFS using msieve for polynomial selection.

# Original version of this script contributed by: tmorrow
#
# Usage: ./convertpoly filename_base

#set -x

MSIEVE_FB="msieve.fb"

if [ "$1" == "" ]
then
echo "Usage: `basename $0` filename_base"
echo ""
echo "Example: `basename $0` C113_109_41"
echo " will factor the number in C113_109_41.n"
exit -1
fi

INPUT=$1.n
OUTPUT=$1.poly

# First check input file exists and output does not
if [[ ! -f $INPUT ]];then
echo "Can't find $INPUT"
exit
fi
if [[ -f $OUTPUT ]];then
echo "$OUTPUT already exists"
exit
fi

# Find the number
N=`awk '/^n: /{print $2}' < $INPUT`

if [[ -z "$N" ]];then
echo "Can't find input number"
exit
fi

# Generate a polynomial in msieve.fb
~/ggnfs/bin/msieve -np -v -t 2 -l ggnfs.log -p $N
if (($?>0));then exit;fi

# Convert it into the format expected by factMsieve.pl
echo "Converting $MSIEVE_FB to $OUTPUT ..."
echo "Converting $MSIEVE_FB to $OUTPUT ..." >>ggnfs.log

echo "type: gnfs" | sed '
s/^N/n:/
s/^SKEW/skew:/
s/^R\(.\)/Y\1:/
s/^A\(.\)/c\1:/
' $MSIEVE_FB - | awk '
{
if ($1=="skew:")
skew=$0
else if ($1=="type:")
type=$0
else
print
}
END {
print skew
print type
}
' > $OUTPUT

echo "Done."

# Now factor the number
~/ggnfs/tests/factMsieve.pl $OUTPUT

[/code]
It's the convertpoly.sh script by tmorrow with calls to msieve and factMsieve.pl added at the beginning and end. You may need to update the paths to msieve and factMsieve.pl (or change the latter to factmsieve.py if you prefer).

The polynomials generated by msieve are much better than the polynomials from polyselect (they can be more than twice as fast). I did 1 test against pol5, msieve was about 15% faster but that may vary.

Chris K

[quote=siew;209320]Run msieve second and third times on the same input file

[code]found 5260417 hash collisions in 28469558 relations
added 120920 free relations[/code][code]found 5268462 hash collisions in 28590477 relations
added 1 free relations[/code][code]found 5268460 hash collisions in 28590479 relations[/code]why the difference ? same input file, same verision of msieve (1.44)[/quote]

I think the problem with the duplicates is within the script.

I ran the python version using four clients, with two connected via VPN. When on a lan you don't really get to pay to attention to:

"appending spairs.out.[ClientId].T[ThreadNumber] to spairs.out.[ClientId]"
"appending spairs.out.[ClientId] to spairs.add.[ClientId]"
"appending spairs.add.[ClientId] to [filename].dat"

But over a vpn you notice when the file sizes get over a couple dozen MB and the appending takes a few minutes.

What I noticed is that after each iteration:

"appending spairs.out.[ClientId].T[ThreadNumber] to spairs.out.[ClientId]", then spairs.out.[ClientId].T[ThreadNumber] is deleted
"appending spairs.out.[ClientId] to spairs.add.[ClientId]" [B]BUG --> while spairs.out is deleted spairs.out.[ClientId] is not deleted[/B]
"appending spairs.add.[ClientId] to [filename].dat" and spairs.add.[ClientId] is deleted.

So what happens here on each iteration appending spairs.out.[ClientId] continues to grow larger and larger (for all clients except client 1) and still contains all of the relations from each previous iteration. Each time the entire contents of the file is appended to [filename].dat.

For example, on two clients C1 and C1 respectively the Relationships (Rels) in the .dat file build as follows:

[quote]Step 1 = S1:
spairs.out = C1.S1.Rels
spairs.out.2 = C2.S1.Rels
[filename].dat = C1.S1.Rels + C2.S1.Rels

Step 2:
spairs.out = C1.S2.Rels
spairs.out.2 = C2.S1.Rels+ C2.S2.Rels
[filename].dat = C1.S1.Rels + C2.S1.Rels + C1.S2.Rels + C2.S1.Rels+ C2.S2.Rels
[Now we have duplicates of Client 2 - step 1]

Step 3:
spairs.out = C1.S3.Rels
spairs.out.2 = C2.S1.Rels+ C2.S2.Rels + C2.S3.Rels
[filename].dat = C1.S1.Rels + C2.S1.Rels + C1.S2.Rels + C2.S1.Rels+ C2.S2.Rels + C2.S1.Rels+ C2.S2.Rels + C2.S3.Rels
[Now we have triplicates of Client 2 - step 1 and duplicates of Client 2 - step 2]

Etc...
[/quote]You will also notice, let's say on the first iteration client 2 adds 3%, then on the second iterations it adds 6% and on the third 9% etc.

My duplicates are not as high as above because my configuration was Client 1 = 8 Threads, Client 2 = 3 Threads, Client 3 = 2 Threads and Client 4 = 1 threads, so most relations came from client 1 which did not have the duplicate issue. To accomplish different threads on different clients, I simply modified factmsieve.py to take an additional argument as the number of threads for the client.

Changes in bold:
[quote]
[B]if len(sys.argv) != 2 and len(sys.argv) != 4 and len(sys.argv) !=5 :[/B]
print('USAGE: {0:s} <number file | poly file| msieve poly file> [ id num]'
.format(sys.argv[0]))
print(' where <polynomial file> is a file with the poly info to use')
print(' or <number file> is a file containing the number to factor.')

.....

[B]f len(sys.argv) >= 4:[/B]
client_id = int(sys.argv[2])
num_clients = int(sys.argv[3])
if client_id < 1 or client_id > num_clients:
die('-> Error: client id should be between 1 and the number of clients {0:d}'
.format(num_clients))
PNUM += client_id

[B]if len(sys.argv) ==5:
NUM_CPUS = int(sys.argv[4])
NUM_THREADS = NUM_CPUS
else:
num_clients = 1
client_id = 1[/B]


[/quote]As noted above the script will complete (at least it did for me). I ended up with the following final sieve step before the algebra step.

[quote]Tue Apr 06 05:24:43 2010
Tue Apr 06 05:24:43 2010 Msieve v. 1.44
Tue Apr 06 05:24:43 2010 random seeds: faaa3f18 191bca5b
Tue Apr 06 05:24:43 2010 factoring 44699272802994988747782413211419089665091145308606310985489874338112959109114610751990090443445518019723525404288825406699041 (125 digits)
Tue Apr 06 05:24:44 2010 searching for 15-digit factors
Tue Apr 06 05:24:45 2010 commencing number field sieve (125-digit input)
Tue Apr 06 05:24:45 2010 R0: -1236243423876030080474068
Tue Apr 06 05:24:45 2010 R1: 33851814290921
Tue Apr 06 05:24:45 2010 A0: -393699594954102801255948101715
Tue Apr 06 05:24:45 2010 A1: 38919105239047210009308099
Tue Apr 06 05:24:45 2010 A2: -513282526641579481355
Tue Apr 06 05:24:45 2010 A3: 392906978262463
Tue Apr 06 05:24:45 2010 A4: 12404505928
Tue Apr 06 05:24:45 2010 A5: 15480
Tue Apr 06 05:24:45 2010 skew 201381.52, size 5.209953e-012, alpha -5.874072, combined = 1.497895e-010
Tue Apr 06 05:24:45 2010
Tue Apr 06 05:24:45 2010 commencing relation filtering
Tue Apr 06 05:24:45 2010 estimated available RAM is 4085.5 MB
Tue Apr 06 05:24:45 2010 commencing duplicate removal, pass 1
Tue Apr 06 05:25:45 2010 error -9 reading relation 6464796
Tue Apr 06 05:25:53 2010 error -11 reading relation 7315604
Tue Apr 06 05:25:53 2010 error -15 reading relation 7339212
Tue Apr 06 05:26:41 2010 found 2363696 hash collisions in 12805689 relations
Tue Apr 06 05:27:10 2010 added 58 free relations
Tue Apr 06 05:27:10 2010 commencing duplicate removal, pass 2
Tue Apr 06 05:27:22 2010 found 2900342 duplicates and 9905405 unique relations
Tue Apr 06 05:27:22 2010 memory use: 53.3 MB
Tue Apr 06 05:27:22 2010 reading ideals above 100000
Tue Apr 06 05:27:22 2010 commencing singleton removal, initial pass
Tue Apr 06 05:29:17 2010 memory use: 344.5 MB
Tue Apr 06 05:29:17 2010 reading all ideals from disk
Tue Apr 06 05:29:17 2010 memory use: 362.1 MB
Tue Apr 06 05:29:18 2010 keeping 10958939 ideals with weight <= 200, target excess is 52285
Tue Apr 06 05:29:19 2010 commencing in-memory singleton removal
Tue Apr 06 05:29:20 2010 begin with 9905405 relations and 10958939 unique ideals
Tue Apr 06 05:29:28 2010 reduce to 3677841 relations and 3669083 ideals in 23 passes
Tue Apr 06 05:29:28 2010 max relations containing the same ideal: 104
Tue Apr 06 05:29:28 2010 filtering wants 1000000 more relations
Tue Apr 06 05:29:28 2010 elapsed time 00:04:45
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8750000 to 8762500 as file examplemc.job.T0
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8762500 to 8775000 as file examplemc.job.T1
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8775000 to 8787500 as file examplemc.job.T2
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8787500 to 8800000 as file examplemc.job.T3
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8800000 to 8812500 as file examplemc.job.T4
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8812500 to 8825000 as file examplemc.job.T5
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8825000 to 8837500 as file examplemc.job.T6
Tue Apr 06 05:29:38 2010 -> making sieve job for q from 8837500 to 8850000 as file examplemc.job.T7
Tue Apr 06 05:29:38 2010 -> Lattice sieving algebraic q from 8750000 to 8850000.
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n100 -a examplemc.job.T0
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T1 -v -n101 -a examplemc.job.T1
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T2 -v -n102 -a examplemc.job.T2
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T3 -v -n103 -a examplemc.job.T3
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T4 -v -n104 -a examplemc.job.T4
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T5 -v -n105 -a examplemc.job.T5
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T6 -v -n106 -a examplemc.job.T6
Tue Apr 06 05:29:38 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T7 -v -n107 -a examplemc.job.T7
Tue Apr 06 05:48:27 2010 LatSieveTime: 1129.11
Tue Apr 06 05:48:51 2010 Found 13070888 relations, 140.4% of the estimated minimum (9310000).[/quote]I also checked that the clients where not overstepping each other's Q values. It seems that Q is split into a BlockSize lets say say of 100,000 for each client, then that block is split into t parts where t is the number of threads. Then each client steps Q by c*BlockSize where C is the number of clients, so using different threads per client does not affect the algorithm. Its just the client 1 had 8 cores which would complete its block in 20 minutes where the client with 1 core and a much slower processor would take over two hours to complete a sieve block (perhaps VPN lag help slowed it down).

Having a better understanding now of how the job is partitioned, I think I will copy the POLY locally to each client next time and each time the a spairs.add.[ClientId] is simply copied to Client #1.


Why? Because for the above c125 I was getting less that .02 secs/rel locally. Running over the lan the speed drops to .05 secs/rel and via vpn something like .1+ rels/sec.

I think each client would run faster if the script was ran locally and only the finally relationships that are to be processed are copied to the working directory of client 1.

However, the issue of the duplicates still remains. Anyone have any suggestions on that? Of course we could each write our own hacks, but I would rather just update the script to work correctly but can't seem to find what needs to be changed, perhaps due to my limited python/perl skills.