[code] This is a standalone one-pass duplicate relations remover;
only syntax (a,b:<csv-of-factors>:<csv-of-factors>) is checked and incomplete lines are discarded;
validity of relations is not tested (nor is polynomial needed).[/code]
rather suggests the answer is no.

[QUOTE=charybdis;567859][code] This is a standalone one-pass duplicate relations remover;
only syntax (a,b:<csv-of-factors>:<csv-of-factors>) is checked and incomplete lines are discarded;
validity of relations is not tested (nor is polynomial needed).[/code]
rather suggests the answer is no.[/QUOTE]

I just ran my relations file through remdups4 anyway, just to make the file smaller, and gzipped it after. I have 1.36B unique rels.

I'm trying two runs on different machines:[LIST=1][*]msieve old revision (R965), TD=140[*]latest msieve, TD=130[/LIST]
Will see how they go.

According to the CADO-NFS documentation (README.msieve), it is possible to perform CADO-NFS filtering on Msieve produced relations. Maybe, then running Msieve on the CADO-NFS results from that point might work.

After 2+ months of LA, and two days of square root phase we're all done:

[CODE]Wed Mar 31 13:08:30 2021 reading relations for dependency 3
Wed Mar 31 13:08:50 2021 read 49325186 cycles
Wed Mar 31 13:12:51 2021 cycles contain 167252566 unique relations
Wed Mar 31 13:51:41 2021 read 167252566 relations
Wed Mar 31 14:24:46 2021 multiplying 167252566 relations
Wed Mar 31 17:40:33 2021 multiply complete, coefficients have about 5103.31 million bits
Wed Mar 31 17:40:54 2021 initial square root is modulo 528433
Wed Mar 31 21:03:38 2021 found factor: 4342820369934629070129641050057768321597558556674831420859929736305770718265897304519624965158902046467403
Wed Mar 31 21:03:38 2021 reading relations for dependency 4
Wed Mar 31 21:04:01 2021 read 49326365 cycles
Wed Mar 31 21:08:42 2021 cycles contain 167231042 unique relations
Wed Mar 31 21:48:12 2021 read 167231042 relations
Wed Mar 31 22:22:26 2021 multiplying 167231042 relations
Thu Apr 1 01:43:35 2021 multiply complete, coefficients have about 5102.66 million bits
Thu Apr 1 01:43:56 2021 initial square root is modulo 527599
Thu Apr 1 05:11:07 2021 sqrtTime: 114940
Thu Apr 1 05:11:07 2021 p88 factor: 5275421214383200513922797574688119880498697251589940847936777262894189343169550133252051
Thu Apr 1 05:11:07 2021 p106 factor: 4342820369934629070129641050057768321597558556674831420859929736305770718265897304519624965158902046467403
Thu Apr 1 05:11:07 2021 p111 factor: 887310813910120829656915049394563462363864534415286197288227631719949635510944144198346994308391761241694168633
Thu Apr 1 05:11:07 2021 elapsed time 31:55:41[/CODE]

[QUOTE=ryanp;574923]After 2+ months of LA, and two days of square root phase we're all done:

[CODE]Wed Mar 31 13:08:30 2021 reading relations for dependency 3
Wed Mar 31 13:08:50 2021 read 49325186 cycles
Wed Mar 31 13:12:51 2021 cycles contain 167252566 unique relations
Wed Mar 31 13:51:41 2021 read 167252566 relations
Wed Mar 31 14:24:46 2021 multiplying 167252566 relations
Wed Mar 31 17:40:33 2021 multiply complete, coefficients have about 5103.31 million bits
Wed Mar 31 17:40:54 2021 initial square root is modulo 528433
Wed Mar 31 21:03:38 2021 found factor: 4342820369934629070129641050057768321597558556674831420859929736305770718265897304519624965158902046467403
Wed Mar 31 21:03:38 2021 reading relations for dependency 4
Wed Mar 31 21:04:01 2021 read 49326365 cycles
Wed Mar 31 21:08:42 2021 cycles contain 167231042 unique relations
Wed Mar 31 21:48:12 2021 read 167231042 relations
Wed Mar 31 22:22:26 2021 multiplying 167231042 relations
Thu Apr 1 01:43:35 2021 multiply complete, coefficients have about 5102.66 million bits
Thu Apr 1 01:43:56 2021 initial square root is modulo 527599
Thu Apr 1 05:11:07 2021 sqrtTime: 114940
Thu Apr 1 05:11:07 2021 p88 factor: 5275421214383200513922797574688119880498697251589940847936777262894189343169550133252051
Thu Apr 1 05:11:07 2021 p106 factor: 4342820369934629070129641050057768321597558556674831420859929736305770718265897304519624965158902046467403
Thu Apr 1 05:11:07 2021 p111 factor: 887310813910120829656915049394563462363864534415286197288227631719949635510944144198346994308391761241694168633
Thu Apr 1 05:11:07 2021 elapsed time 31:55:41[/CODE][/QUOTE]Nice work!

FYI I think Greg Childers finally fixed the large dataset bug, it was a buffer overflow in the clique removal and should be fixed in r1038. More generally with the average size of the largest jobs increasing, Msieve's filtering code is long overdue for a refactoring that removes many current limits on the size of filtering jobs.

Congrats on a huge effort!