The experiment was a success!

I ended up renaming the additions.txt to spairs.out because I noticed that the script appends spairs.out.t0 to spairs.out and the directory did not have a spairs.out file. It apparently worked since there were a little over 1M extra relations found.


[code]
Found 3423553 relations, 39.4% of the estimated minimum (8700000).
. . .
total yield: 585316, q=2700023 (0.03123 sec/rel)
. . .
Found 4008869 relations, 46.1% of the estimated minimum (8700000).
. . .
total yield: 578364, q=2800001 (0.03456 sec/rel)
. . . (added spairs.out to directory)
appending spairs.out.T0 to spairs.out
appending spairs.out to test.dat
compressing spairs.out to spairs.save.gz
Found 5598936 relations, 64.4% of the estimated minimum (8700000).
. . .
[/code]I suppose the real proof will be if the extra relations are valid later. . .

[quote=EdH;217201]The experiment was a success!

I ended up renaming the additions.txt to spairs.out because I noticed that the script appends spairs.out.t0 to spairs.out and the directory did not have a spairs.out file. It apparently worked since there were a little over 1M extra relations found.


[code]
Found 3423553 relations, 39.4% of the estimated minimum (8700000).
. . .
total yield: 585316, q=2700023 (0.03123 sec/rel)
. . .
Found 4008869 relations, 46.1% of the estimated minimum (8700000).
. . .
total yield: 578364, q=2800001 (0.03456 sec/rel)
. . . (added spairs.out to directory)
appending spairs.out.T0 to spairs.out
appending spairs.out to test.dat
compressing spairs.out to spairs.save.gz
Found 5598936 relations, 64.4% of the estimated minimum (8700000).
. . .
[/code]I suppose the real proof will be if the extra relations are valid later. . .[/quote]
One other thing that may be helpful: each time factmsieve.pl/py runs filtering, it also checks spairs.add for additional relations. In your case spairs.out works just as well, but spairs.add was actually put in there for exactly for that purpose. :smile:

I suppose spairs.add would mainly be useful with the older factMsieve.pl, which when using only one thread just wrote to spairs.out directly rather than to spairs.out.T0 (it only used T* if multithreading was enabled). Thus, if you wanted to add anything else, you'd want to use spairs.add to avoid clobbering spairs.out which is actively being written to by the siever.

[quote=mdettweiler;217241]One other thing that may be helpful: each time factmsieve.pl/py runs filtering, it also checks spairs.add for additional relations. In your case spairs.out works just as well, but spairs.add was actually put in there for exactly for that purpose. :smile:

I suppose spairs.add would mainly be useful with the older factMsieve.pl, which when using only one thread just wrote to spairs.out directly rather than to spairs.out.T0 (it only used T* if multithreading was enabled). Thus, if you wanted to add anything else, you'd want to use spairs.add to avoid clobbering spairs.out which is actively being written to by the siever.[/quote]
Thanks!

That makes sense. I originally had decided to append to the test.dat file, since spairs.out.T0 was being written to. But, for some reason, the MS console command "copy file1+file2 file1" was adding an extra character to the end of file1 in my tests and the test.dat file was too large to load into anything I could edit it with. I don't know if the extraneous character would cause a problem, but didn't want to chance it. That was when I noticed the spairs.out option.

I don't see an append for spairs.add in the console output for the Python script. I'm reluctant to assume it is there. I am running another 1M+ relations on the second machine today and will attempt to add it this evening. Maybe I should try this one as spairs.add and if it isn't picked up, I can just rename it to spairs.out for the next check.

Thanks again to both you and Mini-Geek.

[QUOTE=EdH;217251]...for some reason, the MS console command "copy file1+file2 file1" was adding an extra character to the end of file1 in my tests [/QUOTE]Try using "copy /B file1+file2 file1"
Or look for a windows version of cat

[quote=EdH;217251]I don't see an append for spairs.add in the console output for the Python script. I'm reluctant to assume it is there. I am running another 1M+ relations on the second machine today and will attempt to add it this evening. Maybe I should try this one as spairs.add and if it isn't picked up, I can just rename it to spairs.out for the next check.[/quote]
Yes, as I recall it wasn't shown in either the Perl or Python script unless there was actually something to import. I'd definitely give it a try, and if it doesn't work, as you said, you can just go back to the spairs.out method.

[quote=smh;217261]Try using "copy /B file1+file2 file1". . .[/quote]
Worked great! That's the second time I've had to use the binary option to make a text file operation perform correctly [B]with Windows[/B]. I should have looked for the option... (I am really slipping, of late...)

Thanks.

Well, I guess it was a partial success, but whether it was full or not may not be determinable. spairs.add was appended and the addition of over 2M relations from a second machine allowed the desired level to be reached faster, but the filtering stage knocked the 8.3M relations down to less than 1.8M and then msieve requested another 1M.

Here is the console output:
[code]
. . .
Found 7337434 relations, 84.3% of the estimated minimum (8700000).
-> making sieve job for q = 3100000 in 3100000 .. 3200000 as file test.job.T0
-> Lattice sieving algebraic q from 3100000 to 3200000.
-> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
FBsize 223444+0 (deg 5), 283145+0 (deg 1)
total yield: 589534, q=3200003 (0.02835 sec/rel)
6705 Special q, 20599 reduction iterations
reports: 1726046386->297093681->268825353->52728240->46690337->40477642
Number of relations with k rational and l algebraic primes for (k,l)=:

Total yield: 589534
0/0 mpqs failures, 381204/153859 vain mpqs
milliseconds total: Sieve 9347299 Sched 0 medsched 1001066
TD 3824854 (Init 94833, MPQS 824308) Sieve-Change 1754310
TD side 0: init/small/medium/large/search: 36375 318798 68254 164324 936422
sieve: init/small/medium/large/search: 128126 2485871 154330 1983263 447242
TD side 1: init/small/medium/large/search: 25325 277772 57353 166235 1677320
sieve: init/small/medium/large/search: 94875 1658863 155289 2045880 193560
appending spairs.out.T0 to spairs.out
[B]appending spairs.add to spairs.out [/B]
appending spairs.out to test.dat
compressing spairs.out to spairs.save.gz
Found 8923112 relations, 102.6% of the estimated minimum (8700000).
-> msieve -s ..\ggnfs_4858080981969920032970855563231812917802738702050691307204
4091953387200862671848975524614933791220018550035318044657829\test.dat -l ..\ggn
fs_48580809819699200329708555632318129178027387020506913072044091953387200862671
848975524614933791220018550035318044657829\test.log -i ..\ggnfs_4858080981969920
03297085556323181291780273870205069130720440919533872008626718489755246149337912
20018550035318044657829\test.ini -nf ..\ggnfs_4858080981969920032970855563231812
91780273870205069130720440919533872008626718489755246149337912200185500353180446
57829\test.fb -t 1 -nc1
-> making sieve job for q = 3200000 in 3200000 .. 3300000 as file test.job.T0
-> Lattice sieving algebraic q from 3200000 to 3300000.
-> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
FBsize 230149+0 (deg 5), 283145+0 (deg 1)
total yield: 223092, q=3236903 (0.02841 sec/rel)
[/code]I am kind of wondering if it will do two more sieves before it tries again, or just one, since the range 3200000 to 3300000 will only yield .5M.

In any case, I think tomorrow morning it will be working on a different composite. I'll have to check the db when I get up to see if it has a new line or two. . .

Thanks again for all the help.

Failure!:sad:

It crashed so bad, I couldn't save the screen shots. It had cycled back to the running ecm forever default and then proceeded to loop through system errors based on cmd.exe and ecm faults. I couldn't even bring up the task manager. I ended up having to reset the machine.

It's now at over 10M relations in a 1.1GB file and I've got it running the filtering manually. If I can't catch what happens this time, I'll pipe the output to a text file next.

More to follow. . .

What did it put in the log (..\ggnfs_48580809819699200329708555632318129178027387020506913072044091953387200862671848975524614933791220018550035318044657829\test.log) ?

Chris K

[quote=chris2be8;217377]What did it put in the log (..\ggnfs_48580809819699200329708555632318129178027387020506913072044091953387200862671848975524614933791220018550035318044657829\test.log) ?

Chris K[/quote]
test.log:
[code]
Tue Jun 01 10:53:02 2010 -> factmsieve.py (v0.68)
Tue Jun 01 10:53:02 2010 -> This is client 1 of 1
Tue Jun 01 10:53:02 2010 -> Using 1 threads
Tue Jun 01 10:53:02 2010 -> Working with NAME = test
Tue Jun 01 10:53:02 2010 -> Selected lattice siever: gnfs-lasieve4I13e
Tue Jun 01 10:53:02 2010 -> Creating param file to detect parameter changes...
Tue Jun 01 10:53:02 2010 -> Running setup ...
Tue Jun 01 10:53:02 2010 -> Estimated minimum relations needed: 8.7e+06
Tue Jun 01 10:53:02 2010 -> cleaning up before a restart
Tue Jun 01 10:53:02 2010 -> Running lattice siever ...
Tue Jun 01 10:53:02 2010 -> entering sieving loop
Tue Jun 01 10:53:02 2010 -> making sieve job for q = 2000000 in 2000000 .. 2100000 as file test.job.T0
Tue Jun 01 10:53:02 2010 -> Lattice sieving algebraic q from 2000000 to 2100000.
Tue Jun 01 10:53:02 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
Tue Jun 01 15:14:46 2010 Found 567879 relations, 6.5% of the estimated minimum (8700000).
. . .
Thu Jun 03 09:18:23 2010 Found 6758347 relations, 77.7% of the estimated minimum (8700000).
Thu Jun 03 09:18:23 2010 LatSieveTime: 15893.2
Thu Jun 03 09:18:23 2010 -> making sieve job for q = 3000000 in 3000000 .. 3100000 as file test.job.T0
Thu Jun 03 09:18:23 2010 -> Lattice sieving algebraic q from 3000000 to 3100000.
Thu Jun 03 09:18:23 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
Thu Jun 03 13:41:36 2010 Found 7337434 relations, 84.3% of the estimated minimum (8700000).
Thu Jun 03 13:41:36 2010 LatSieveTime: 15793.4
Thu Jun 03 13:41:36 2010 -> making sieve job for q = 3100000 in 3100000 .. 3200000 as file test.job.T0
Thu Jun 03 13:41:36 2010 -> Lattice sieving algebraic q from 3100000 to 3200000.
Thu Jun 03 13:41:36 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
Thu Jun 03 18:25:07 2010 Found 8923112 relations, 102.6% of the estimated minimum (8700000).
Thu Jun 03 18:25:08 2010
Thu Jun 03 18:25:08 2010
Thu Jun 03 18:25:08 2010 Msieve v. 1.45
Thu Jun 03 18:25:08 2010 random seeds: 793396c0 ef9fe4e6
Thu Jun 03 18:25:08 2010 factoring 48580809819699200329708555632318129178027387020506913072044091953387200862671848975524614933791220018550035318044657829 (119 digits)
Thu Jun 03 18:25:10 2010 searching for 15-digit factors
Thu Jun 03 18:25:11 2010 commencing number field sieve (119-digit input)
Thu Jun 03 18:25:11 2010 R0: -79189672477676783946937
Thu Jun 03 18:25:11 2010 R1: 6360030448193
Thu Jun 03 18:25:11 2010 A0: -4051842427045653038354536200
Thu Jun 03 18:25:11 2010 A1: -93642941254012491376278
Thu Jun 03 18:25:11 2010 A2: 6007458903024306275
Thu Jun 03 18:25:11 2010 A3: 13211933274479
Thu Jun 03 18:25:11 2010 A4: -1154858551
Thu Jun 03 18:25:11 2010 A5: 15600
Thu Jun 03 18:25:11 2010 skew 54940.12, size 2.012071e-011, alpha -4.777729, combined = 3.227413e-010
Thu Jun 03 18:25:11 2010
Thu Jun 03 18:25:11 2010 commencing relation filtering
Thu Jun 03 18:25:11 2010 estimated available RAM is 703.0 MB
Thu Jun 03 18:25:11 2010 commencing duplicate removal, pass 1
Thu Jun 03 18:27:54 2010 found 999329 hash collisions in 8923111 relations
Thu Jun 03 18:29:10 2010 added 62369 free relations
Thu Jun 03 18:29:10 2010 commencing duplicate removal, pass 2
Thu Jun 03 18:30:13 2010 found 602452 duplicates and 8383028 unique relations
Thu Jun 03 18:30:13 2010 memory use: 41.3 MB
Thu Jun 03 18:30:13 2010 reading ideals above 3866624
Thu Jun 03 18:30:18 2010 commencing singleton removal, initial pass
Thu Jun 03 18:34:02 2010 memory use: 188.3 MB
Thu Jun 03 18:34:03 2010 reading all ideals from disk
Thu Jun 03 18:34:35 2010 memory use: 152.0 MB
Thu Jun 03 18:34:38 2010 commencing in-memory singleton removal
Thu Jun 03 18:34:40 2010 begin with 8383028 relations and 9459517 unique ideals
Thu Jun 03 18:35:04 2010 reduce to 1833251 relations and 1505432 ideals in 29 passes
Thu Jun 03 18:35:04 2010 max relations containing the same ideal: 34
Thu Jun 03 18:35:05 2010 reading ideals above 30000
Thu Jun 03 18:35:06 2010 commencing singleton removal, initial pass
Thu Jun 03 18:36:30 2010 memory use: 47.1 MB
Thu Jun 03 18:36:30 2010 reading all ideals from disk
Thu Jun 03 18:36:34 2010 memory use: 72.9 MB
Thu Jun 03 18:36:36 2010 keeping 2202907 ideals with weight <= 200, target excess is 10076
Thu Jun 03 18:36:38 2010 commencing in-memory singleton removal
Thu Jun 03 18:36:39 2010 begin with 1860728 relations and 2202907 unique ideals
Thu Jun 03 18:37:14 2010 reduce to 1779526 relations and 1988166 ideals in 24 passes
Thu Jun 03 18:37:14 2010 max relations containing the same ideal: 199
Thu Jun 03 18:37:15 2010 filtering wants 1000000 more relations
Thu Jun 03 18:37:15 2010 elapsed time 00:12:07
Thu Jun 03 18:37:15 2010 LatSieveTime: 17739.3
Thu Jun 03 18:37:15 2010 -> making sieve job for q = 3200000 in 3200000 .. 3300000 as file test.job.T0
Thu Jun 03 18:37:15 2010 -> Lattice sieving algebraic q from 3200000 to 3300000.
Thu Jun 03 18:37:15 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
Thu Jun 03 23:16:31 2010 Found 9579177 relations, 110.1% of the estimated minimum (8700000).
Thu Jun 03 23:16:31 2010
Thu Jun 03 23:16:31 2010
Thu Jun 03 23:16:31 2010 Msieve v. 1.45
Thu Jun 03 23:16:31 2010 random seeds: b0e56c80 e2fcc3d6
Thu Jun 03 23:16:31 2010 factoring 48580809819699200329708555632318129178027387020506913072044091953387200862671848975524614933791220018550035318044657829 (119 digits)
Thu Jun 03 23:16:34 2010 searching for 15-digit factors
Thu Jun 03 23:16:35 2010 commencing number field sieve (119-digit input)
Thu Jun 03 23:16:35 2010 R0: -79189672477676783946937
Thu Jun 03 23:16:35 2010 R1: 6360030448193
Thu Jun 03 23:16:35 2010 A0: -4051842427045653038354536200
Thu Jun 03 23:16:35 2010 A1: -93642941254012491376278
Thu Jun 03 23:16:35 2010 A2: 6007458903024306275
Thu Jun 03 23:16:35 2010 A3: 13211933274479
Thu Jun 03 23:16:35 2010 A4: -1154858551
Thu Jun 03 23:16:35 2010 A5: 15600
Thu Jun 03 23:16:35 2010 skew 54940.12, size 2.012071e-011, alpha -4.777729, combined = 3.227413e-010
Thu Jun 03 23:16:35 2010
Thu Jun 03 23:16:35 2010 commencing relation filtering
Thu Jun 03 23:16:35 2010 estimated available RAM is 703.0 MB
Thu Jun 03 23:16:35 2010 commencing duplicate removal, pass 1
Thu Jun 03 23:19:30 2010 found 900382 hash collisions in 9579176 relations
Thu Jun 03 23:20:45 2010 added 232 free relations
Thu Jun 03 23:20:45 2010 commencing duplicate removal, pass 2
Thu Jun 03 23:21:52 2010 found 675784 duplicates and 8903624 unique relations
Thu Jun 03 23:21:52 2010 memory use: 34.6 MB
Thu Jun 03 23:21:52 2010 reading ideals above 3866624
Thu Jun 03 23:21:57 2010 commencing singleton removal, initial pass
Thu Jun 03 23:25:56 2010 memory use: 188.3 MB
Thu Jun 03 23:25:56 2010 reading all ideals from disk
Thu Jun 03 23:26:29 2010 memory use: 161.5 MB
Thu Jun 03 23:26:32 2010 commencing in-memory singleton removal
Thu Jun 03 23:26:35 2010 begin with 8903624 relations and 9721146 unique ideals
Thu Jun 03 23:27:03 2010 reduce to 2532316 relations and 2079492 ideals in 26 passes
Thu Jun 03 23:27:03 2010 max relations containing the same ideal: 46
Thu Jun 03 23:27:05 2010 reading ideals above 100000
Thu Jun 03 23:27:05 2010 commencing singleton removal, initial pass
Thu Jun 03 23:28:51 2010 memory use: 86.1 MB
Thu Jun 03 23:28:51 2010 reading all ideals from disk
Thu Jun 03 23:28:57 2010 memory use: 88.4 MB
Thu Jun 03 23:28:59 2010 keeping 2609791 ideals with weight <= 200, target excess is 19256
Thu Jun 03 23:29:01 2010 commencing in-memory singleton removal
Thu Jun 03 23:29:03 2010 begin with 2532551 relations and 2609791 unique ideals
Thu Jun 03 23:29:30 2010 reduce to 2520784 relations and 2596862 ideals in 14 passes
Thu Jun 03 23:29:30 2010 max relations containing the same ideal: 200
Thu Jun 03 23:29:32 2010 filtering wants 1000000 more relations
Thu Jun 03 23:29:32 2010 elapsed time 00:13:01
Thu Jun 03 23:29:32 2010 LatSieveTime: 17536.6
Thu Jun 03 23:29:32 2010 -> making sieve job for q = 3300000 in 3300000 .. 3400000 as file test.job.T0
Thu Jun 03 23:29:32 2010 -> Lattice sieving algebraic q from 3300000 to 3400000.
Thu Jun 03 23:29:32 2010 -> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
Fri Jun 04 03:54:33 2010 Found 10157379 relations, 116.8% of the estimated minimum (8700000).
[/code]As of a couple minutes ago, LA completed and it is now working on the sqrt phase. At least the sieving time wasn't lost and it was less than three days, anyway - not like a month or year long dataset being lost. The only real issue was that it crashed and I had to intervene to resolve it. This was running under AliWin and, really, the only thing I've done over the last few weeks (other than this experiment with relation adding) is look at it and interrupt it a couple times to update versions.

In case it is of interest to anyone, the manual steps worked fine and returned two factors which I plugged back into aliqueit via AliWin and it is now working along. Unfortunately, it has picked up a persistent 3, and now, an extra digit.

I will perhaps try this experiment again later, but right now, time is growing rare. . .

Thanks again for all the info.