mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Aliquot Sequences (https://www.mersenneforum.org/forumdisplay.php?f=90)
-   -   Too many merge attempts etc. (https://www.mersenneforum.org/showthread.php?t=11794)

hhh 2009-04-28 07:10

Too many merge attempts etc.
 
Dear friends,

the "too many merge attempts"-error showed up at some [URL="http://factorization.ath.cx/search.php?id=31962963"]c104[/URL] of mine. I tried to fiddle aroud a bit, with the sole success to replace it by some "the matrix must have more coloums than rows" (or vice versa) error. Here are my options, I guess:

- I upload thezipped folder to some volunteer who does the postprocessing for me
- You help me to get out of this mess
- I restart sieving, with perhaps slightly different parameters
- Somebody else does this number from scratch.

This is the order of my preference, also. You decide, though.

Cheers, H.

schickel 2009-04-28 07:16

[QUOTE=hhh;171260]Dear friends,

the "too many merge attempts"-error showed up at some [URL="http://factorization.ath.cx/search.php?id=31962963"]c104[/URL] of mine. I tried to fiddle aroud a bit, with the sole success to replace it by some "the matrix must have more coloums than rows" (or vice versa) error. Here are my options, I guess:

- I upload thezipped folder to some volunteer who does the postprocessing for me
- You help me to get out of this mess
- I restart sieving, with perhaps slightly different parameters
- Somebody else does this number from scratch.

This is the order of my preference, also. You decide, though.

Cheers, H.[/QUOTE]I'd be willing to take a stab at it no one else does. I've got one more day at work, so I couldn't really concentrate on it until Wednesday....

hhh 2009-04-28 07:39

[QUOTE=schickel;171261]I'd be willing to take a stab at it no one else does. I've got one more day at work, so I couldn't really concentrate on it until Wednesday....[/QUOTE]

You can give me some of your work in exchange... and I'd need some link for the upload. H.

schickel 2009-04-28 07:57

[QUOTE=hhh;171264]You can give me some of your work in exchange... and I'd need some link for the upload. H.[/QUOTE]How big is the dataset? I don't have anything set up right now, but I could put an FTP server on one of my big rigs.....

schickel 2009-04-28 09:58

[QUOTE=hhh;171264]You can give me some of your work in exchange... and I'd need some link for the upload. H.[/QUOTE]PM sent.

mdettweiler 2009-04-28 14:33

[quote=schickel;171267]How big is the dataset? I don't have anything set up right now, but I could put an FTP server on one of my big rigs.....[/quote]
If you guys want, you can use the FTP server that I set up a while back for the team sieves. (Instructions for uploading can be found in the latest team sieve thread; the only difference is that it would probably be better to plop the files in the root directory of the FTP server rather than in the /c136-relations folder to avoid confusion with the team sieve files.)

schickel 2009-04-28 18:14

[QUOTE=mdettweiler;171307]If you guys want, you can use the FTP server that I set up a while back for the team sieves. (Instructions for uploading can be found in the latest team sieve thread; the only difference is that it would probably be better to plop the files in the root directory of the FTP server rather than in the /c136-relations folder to avoid confusion with the team sieve files.)[/QUOTE]Thanks for the offer, but if we can get a direct FTP to my side working, then we only have to transfer everything once....no sense wasting more electrons than needed.

henryzz 2009-04-28 18:41

[quote=Batalov;170156]As the number of collected relations grows, there are four "zones":
1) far too few relations; then -nc1 doesn't produce cycles and asks for more relns => Sieve some more;
2) [SIZE=1](rarely visited; you are here!)[/SIZE] almost enough relations; -nc1 makes cycles, -nc2 builds a matrix, cleans it up and then the matrix is not useable => Sieve some more;
3) convergent zone; here's some freedom of choice: with more relns, the matrix gets better and better, but it is well established that you generally do not save any overall time anymore => Build the matrix and do -nc2, -nc3
4) far too many relations => trim free relations from the end of .dat file (these are short lines), then trim some more, goto zone 3). But not too much or you will end up in zones 1-2)[/quote]
does this help?

Batalov 2009-04-28 19:25

You are likely in zone 4. So you want to trim, but not too much ...as per the above spell.

hhh 2009-04-29 06:50

Everything submitted (I hope). Many thanks for the help, Frank!

Thank's for the advices; that's what I thought, only that I didn't really know what to do. As for how I got there, I have no idea, because I started filtering in zone 1, and added sieving according to the standard parameters.

Anyway, soon this problem will be solved, I hope.

H.

schickel 2009-04-29 08:01

[QUOTE=hhh;171483]Everything submitted (I hope). Many thanks for the help, Frank!

Thank's for the advices; that's what I thought, only that I didn't really know what to do. As for how I got there, I have no idea, because I started filtering in zone 1, and added sieving according to the standard parameters.

Anyway, soon this problem will be solved, I hope.

H.[/QUOTE]Yes, I've got everything. I'll take a look at it, but I can't promise intense study until tomorrow AM.....

schickel 2009-04-29 17:18

[QUOTE=hhh;171260]Dear friends,

the "too many merge attempts"-error showed up at some [URL="http://factorization.ath.cx/search.php?id=31962963"]c104[/URL] of mine. I tried to fiddle aroud a bit, with the sole success to replace it by some "the matrix must have more coloums than rows" (or vice versa) error. Here are my options, I guess:

- I upload thezipped folder to some volunteer who does the postprocessing for me
- You help me to get out of this mess
- I restart sieving, with perhaps slightly different parameters
- Somebody else does this number from scratch.

This is the order of my preference, also. You decide, though.

Cheers, H.[/QUOTE]H, what version of msieve were you running? I keep forgetting to upgrade this one, so when I ran this, it was with 1.39 and it finished on the first try, so I don't know what zone you were really in:[code]linear algebra completed 222519 of 223174 dimensions (99.7%, ETA 0h 0m)
lanczos halted after 3528 iterations (dim = 222925)
recovered 32 nontrivial dependencies

commencing square root phase
reading relations for dependency 1
read 111749 cycles
cycles contain 455893 unique relations
read 455893 relations
multiplying 366356 relations
multiply complete, coefficients have about 14.51 million bits
initial square root is modulo 215881931
prp52 factor: 7570380941541769441561745312556684037938683763298177
prp52 factor: 8145994460731835857997597172380480561830720426463929
elapsed time 00:29:25[/code]:bow wave::bow wave:

Nice split there!!!

hhh 2009-04-30 00:37

Msieve 1.41. So, the problem was not oversieving, but the brilliance. Perhaps we should tell the case to the maker of Msieve, for that he knows that some functionality is lost in version 1.41.

Anyway, cheers, H.

10metreh 2009-04-30 06:10

[quote=hhh;171582]Msieve 1.41. So, the problem was not oversieving, but the brilliance. Perhaps we should tell the case to the maker of Msieve, for that he knows that some functionality is lost in version 1.41.

Anyway, cheers, H.[/quote]

When did nice splits make factorizations fail?

schickel 2009-04-30 06:27

[quote=batalov]2) ([size="1"]rarely visited; you are here![/size]) almost enough relations; -nc1 makes cycles, -nc2 builds a matrix, cleans it up and then the matrix is not useable => Sieve some more;[/quote]Looking at the relation count, I think this is where you were, h, one more sieve block might have done it......looking back at similar sized jobs, mine needed slightly more relations than you had. [quote=10metreh]When did nice splits make factorizations fail? [/quote]Probably not fail, maybe just need a few more relations than usual. It'd be interesting to see if other folks have had any problems with 50/50 splits on their jobs.

Maybe the solution is to keep 1.39 around as a fallback if there are similar problems in the future.

hhh 2009-04-30 06:54

[QUOTE=10metreh;171617]When did nice splits make factorizations fail?[/QUOTE]
You, [URL="http://www.mersenneforum.org/showthread.php?t=11271"]here[/URL], post number 4. [SPOILER]Actually, you didn't, but I understood it like that, my bad.
[/SPOILER]
I was inclined to such a suggestion as you had some similar problem [URL="http://www.mersenneforum.org/showpost.php?p=170150&postcount=63"]here[/URL] with some [URL="http://www.mersenneforum.org/showpost.php?p=170370&postcount=69"]p47*p49[/URL]. I can be completely mistaken.
[QUOTE]Looking at the relation count, I think this is where you were, h, one more sieve block might have done it......looking back at similar sized jobs, mine needed slightly more relations than you had.[/QUOTE]
I would have done that, hadn't msieve (or the perl script, I don't know) given me the error message.

Anyway, we have learned something (at least I have), that's something,too. Cheers, H.

hhh 2009-05-07 13:02

1 Attachment(s)
OK, here we go again.
[QUOTE]Thu May 07 13:29:45 2009 distribution of cycle lengths:
Thu May 07 13:29:45 2009 1 relations: 32601
Thu May 07 13:29:45 2009 2 relations: 161182
Thu May 07 13:29:45 2009 3 relations: 0
Thu May 07 13:29:45 2009 4 relations: 0
Thu May 07 13:29:45 2009 5 relations: 0
Thu May 07 13:29:45 2009 6 relations: 0
Thu May 07 13:29:45 2009 7 relations: 0
Thu May 07 13:29:45 2009 8 relations: 0
Thu May 07 13:29:45 2009 9 relations: 0
Thu May 07 13:29:45 2009 10+ relations: 0
Thu May 07 13:29:45 2009 heaviest cycle: 2 relations
Thu May 07 13:29:45 2009 matrix not dense enough, retrying

error: too many merge attempts
Return value 65280. Terminating...
WARNING: gnfs failed to find a factor. This really shouldn't happen.
etc.
[/QUOTE]
The last two lines were visible on the screen only, but I copied them into the log. When I restarted aliqueit, it sieved again a little bit, same story, and doesn't take up sieving again, now. Can you please help me to finish this job by hand?

henryzz 2009-05-07 16:31

have you tried again?

jasonp 2009-05-07 17:23

This job is very heavily oversieved; try it with ~2/3 the total number of relations (it was not oversieved enough to trigger the new code in v1.41 for dealing with very heavy oversieving).

schickel 2009-05-08 00:40

[QUOTE=jasonp;172758]This job is very heavily oversieved; try it with ~2/3 the total number of relations (it was not oversieved enough to trigger the new code in v1.41 for dealing with very heavy oversieving).[/QUOTE]I wonder if he's in the same position as last time....just a tiny bit under the required number of realtions. My older c112 jobs, all done completely with GGNFS and the factlat.pl script, required between 7 and 8 million relations. From hhh's logfile, he's at this position:[code]found 1261318 hash collisions in 8230451 relations
commencing duplicate removal, pass 2
found 2115181 duplicates and 6115270 unique relations[/code]Which says that he probably needs at least one more block of sieving, if not a couple more. Look at that duplicate rate.....nearly 25%

hhh, I can take another stab at it if you like, or you can try one more block of sieving and see what happens......let me know and we can setup an FTP transfer again.

hhh 2009-05-08 03:14

I have got the impression that when I restart aliqueit, it keeps resieving the same spot. How can I manually sieve a chunk and manually start the postprocessing? That would be of much help. Thanks for all your concern, H.

schickel 2009-05-08 03:28

[QUOTE=hhh;172822]I have got the impression that when I restart aliqueit, it keeps resieving the same spot. How can I manually sieve a chunk and manually start the postprocessing? That would be of much help. Thanks for all your concern, H.[/QUOTE]Maybe someone else can chime in, since I only run aliqueit on sequences until they hit 100-digits; but do you only run NFS jobs under aliqueit? My solution would be to use one of the fact*.pl scripts and adjust the .job file to move the next chunks up above where it's already been sieved.

henryzz 2009-05-08 06:17

the fact*.pl scripts by default sieve further i think
if for some reason it is sieving the same range again change factmsieve.pl to sieve on the rational side and try again

jasonp 2009-05-11 04:10

[QUOTE=schickel;172802]I wonder if he's in the same position as last time....just a tiny bit under the required number of realtions. My older c112 jobs, all done completely with GGNFS and the factlat.pl script, required between 7 and 8 million relations.[/QUOTE]
I suspect you're right, I was looking at the results from the merge and they are superficially similar to what you see when there's a lot of oversieving. But if you have almost enough relations I can see a threshold phenomenon where deleting a few relations during the merge causes a cascade of deletions that destroys most of the matrix.

hhh 2009-05-11 04:56

I have got yet another c99 with this problem, and will actively try to figure it out once I find the time. Won`t happen soon though, perhaps in a week or so. Until then, please don`t use too many brain cells on this problem. I am already grateful for all your support. Cheers, H.

hhh 2009-05-14 13:03

1 Attachment(s)
OK, I managed to finish both the stuck c99 and c112, and I have a theory about what was going wrong.

First, my girlfriend shut down the computer/closed the window/whatever [U]during some filtering run[/U]. As the process was killed in action, the files spairs.out, something.add and so on failed to get deleted, and the jobfiles remained untouched. I restarted aliqueit, and sieving was carried out [U]once again on the same spot[/U], resulting in loads of duplicates. For some reason, msieve didn't like to be fed with the same stuff over and over, and threw up by dint of the merge attempts error. After this, the perlscript [U]still [/U] didn't clean up all the mess, which I had to do manually. So, I deleted

-spairs.out which was huge
-*.add (forgot the name)
- the job.T1 and job.T2 files

and adjusted the job file to some higher range. From now, everything went fine. Yet, I don't know if the new, last sieving run was indeed necessary or if a minuscule chunk would have been enough to put things in order again.

Thanks for your help again, pals, I will just post the logfiles (with annotations inside), so you can confirm of reject my theory... I wonder if something can be easily implemented into the perlscript or aliqueit in order to go around this kind of annoying behaviour.


H.

hhh 2009-05-14 13:04

1 Attachment(s)
:juggle:

mklasson 2009-05-14 15:29

[QUOTE=hhh;173514]I wonder if something can be easily implemented into the perlscript or aliqueit in order to go around this kind of annoying behaviour.[/QUOTE]

I recall someone (you?) asking me to fix resuming partial ggnfs sieving in aliqueit earlier, and I must confess I've been annoyed by it as well. I think someone even posted the exact details of what needs to be done to fix it, so maybe I can include that in an upcoming release.

Andi47 2009-05-14 19:54

[QUOTE=mklasson;173528]I recall someone (you?) asking me to fix resuming partial ggnfs sieving in aliqueit earlier, and I must confess I've been annoyed by it as well. I think someone even posted the exact details of what needs to be done to fix it, so maybe I can include that in an upcoming release.[/QUOTE]

This might have been me, who has asked - and if I remember correctly, the answer has been posted to the aliqueit.exe discussion thread.


All times are UTC. The time now is 08:22.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.