Sequence terminations and merges

[Batalov]

1459788 1329336
1464640 694728
1465062 705042
1467474 1105734
1468482 559758
1468548 206244
1473834 183942
1475346 1347726
1476342 1207818
1479174 1479162
1479522 1195554
1480620 581976
1482312 1184232
1482712 237384
1487574 509736
1491318 1145718
1493142 1039434
1496928 890016
1497294 924462
1499736 1249464
1502592 191352
1502880 480792
1982796 1487076
1985034 1479162

[Batalov]

1305216 302226
1307180 872900
1319946 705744
1323738 625758
1331704 748488
1340670 473502
1344840 1327560
1347696 900576
1353436 481068
1353438 26340
1362444 207102
1363736 438984
1366950 688422
1504488 500424
1505256 1343016
1505322 78474
1507020 1291698
1507464 738936
1509240 754440
1509978 207684
1515120 179880
1525590 1196736
1527438 1315290
1531242 1493082
1531728 910912

[henryzz]

Most of these merges seem to just being found as we search the database. Are there any that need adding to the first post due to actual work going into them?
I am thinking of putting merges > 1M in the third post.

[Dubslow]

Originally Posted by henryzz

Most of these merges seem to just being found as we search the database. Are there any that need adding to the first post due to actual work going into them?
I am thinking of putting merges > 1M in the third post.

For the next few weeks, don't bother. This is part of the ongoing project to ensure the validity if u2d's original data. The original range is being searched from scratch, with "sequences" surviving up to 50 digits by aliqueit added to the standard SeqList used by the webpage; the idea was to rely on the builtin merging, but this is proving somewhat problematic for a couple of reasons. I should have a fix for one reason incoming shortly.

[Dubslow]

Okay as I see it there's one primary problem and one secondary problem leading to the seemingly-extremely slow pace of merge detection. The primary problem is the aforementioned lack of data synchronization between the two branches, while the secondary problem is that I realized a few hours ago that, in the slow-merge-quantity days before the extension, I wrote the merge code to only look for one sequence merging into another, not accounting for the fact that multiple secondary sequences might merge into one primary at the same time.

I've just pushed new code to account for that case (and I think it's cleaner with less conditions anyways).

Regarding the primary problem, I can think of two ways to improve it.

One is that a cursory perusal of the primary webpage will immediately reveal lots of 7 digit sequences with sizes >135 and "recent progress" in the several thousands. Although the former may be due to the yafu@home churn of its high-powered users, the latter case is almost certainly indicative of as-yet undetected mergers whose "primary" branch moved both before and after the merging-secondary-lines were uploaded to the FDB, so targeting those seqs for manual update will find merges; in my tests locally before posting, I've gotten a 1/4th - 1/3rd success rate at eliminating such seqs. (The script I used to locate such likely merge targets has been uploaded to my web-rkn branch, CB, for your use on the server.)

Here is a sample output of: 1) mirroring my local data to the most recent website data; 2) using the aforementioned script to find likely merge targets; 3) running the manual update of the scripts, with the improved merge finding code:

05:08:54 290 sequences complete: 1043454
05:08:55 Loop complete, new data saved
05:08:56 Found merges!
The seq(s) 1391296 seem(s) to have merged with 4800
The seq(s) 1043116 seem(s) to have merged with 806004
The seq(s) 1875080 seem(s) to have merged with 1337892
The seq(s) 1650390 seem(s) to have merged with 287574
The seq(s) 1044510 seem(s) to have merged with 784782
The seq(s) 1253102, 1257998 seem(s) to have merged with 1008526
The seq(s) 1648494 seem(s) to have merged with 102264
The seq(s) 1328766 seem(s) to have merged with 1247874
The seq(s) 1378826 seem(s) to have merged with 3876
The seq(s) 1694912 seem(s) to have merged with 894864
The seq(s) 1320828, 1343820 seem(s) to have merged with 1315724
The seq(s) 1063290 seem(s) to have merged with 1023570
The seq(s) 1303170 seem(s) to have merged with 1007106
The seq(s) 1350720 seem(s) to have merged with 1305336
The seq(s) 1299158 seem(s) to have merged with 1067616
The seq(s) 1802040 seem(s) to have merged with 706440
The seq(s) 1032668 seem(s) to have merged with 621280
The seq(s) 1072050 seem(s) to have merged with 1042698
The seq(s) 1371238, 1373858 seem(s) to have merged with 8844
The seq(s) 1062810 seem(s) to have merged with 592506
The seq(s) 1654218 seem(s) to have merged with 1371510
The seq(s) 1286068 seem(s) to have merged with 960900
The seq(s) 1766292 seem(s) to have merged with 1014780
The seq(s) 1296724 seem(s) to have merged with 1057710
The seq(s) 1327080 seem(s) to have merged with 1289640
The seq(s) 1686948 seem(s) to have merged with 814518
The seq(s) 1008426, 1548570 seem(s) to have merged with 676758
The seq(s) 1275762 seem(s) to have merged with 1087602
The seq(s) 1232898 seem(s) to have merged with 1068366
The seq(s) 1292280 seem(s) to have merged with 1264152
The seq(s) 1294422 seem(s) to have merged with 1039244
The seq(s) 1361192 seem(s) to have merged with 1233338
The seq(s) 1273878 seem(s) to have merged with 1053942
The seq(s) 1297704 seem(s) to have merged with 1038676
The seq(s) 1095292 seem(s) to have merged with 993396
The seq(s) 1370248 seem(s) to have merged with 1278088
The seq(s) 1377072 seem(s) to have merged with 1307468
The seq(s) 1025694 seem(s) to have merged with 906762
The seq(s) 1548312 seem(s) to have merged with 1378848
The seq(s) 1224510 seem(s) to have merged with 976278
The seq(s) 1008528 seem(s) to have merged with 553872
The seq(s) 1653120 seem(s) to have merged with 1228704
The seq(s) 1229776 seem(s) to have merged with 1205752
The seq(s) 1031946 seem(s) to have merged with 626934
The seq(s) 1049244, 1208880 seem(s) to have merged with 666540
The seq(s) 1376790 seem(s) to have merged with 1279014
The seq(s) 1325106 seem(s) to have merged with 1215390
The seq(s) 1250690, 1266694, 1280928 seem(s) to have merged with 1235232
The seq(s) 1696940 seem(s) to have merged with 408320
The seq(s) 1058394 seem(s) to have merged with 1044534
The seq(s) 1343922 seem(s) to have merged with 1094226
The seq(s) 1030218 seem(s) to have merged with 780066
The seq(s) 1356762 seem(s) to have merged with 1070442
The seq(s) 1205826 seem(s) to have merged with 971454
The seq(s) 1617144 seem(s) to have merged with 1087608
The seq(s) 1238788 seem(s) to have merged with 965064
The seq(s) 1301190, 1374054 seem(s) to have merged with 1249116
The seq(s) 1043454 seem(s) to have merged with 879978
The seq(s) 1387290 seem(s) to have merged with 1375782
The seq(s) 1648776 seem(s) to have merged with 1007866
The seq(s) 1388154 seem(s) to have merged with 1313274
The seq(s) 1280598, 1380944 seem(s) to have merged with 1047756
The seq(s) 1309146 seem(s) to have merged with 1296534
The seq(s) 1207130 seem(s) to have merged with 1068670
The seq(s) 1306644 seem(s) to have merged with 1033008
The seq(s) 1305216 seem(s) to have merged with 302226
The seq(s) 1064706 seem(s) to have merged with 870642
The seq(s) 1258198 seem(s) to have merged with 1007422
The seq(s) 1335138 seem(s) to have merged with 1301886
The seq(s) 1292156 seem(s) to have merged with 1024564
The seq(s) 1059954, 1618770 seem(s) to have merged with 1032846
The seq(s) 1839750 seem(s) to have merged with 994698
The seq(s) 1380244 seem(s) to have merged with 1249400
The seq(s) 1764696 seem(s) to have merged with 1273012
The seq(s) 1338870 seem(s) to have merged with 1280022
The seq(s) 1685634 seem(s) to have merged with 1001646
The seq(s) 1206320, 1231024, 1239500, 1322290 seem(s) to have merged with 1054412
The seq(s) 1286430 seem(s) to have merged with 1097442
The seq(s) 1072192 seem(s) to have merged with 939260
The seq(s) 1368492 seem(s) to have merged with 1039596
The seq(s) 1551738, 1875594 seem(s) to have merged with 1352010
The seq(s) 1031328 seem(s) to have merged with 883962
05:08:56 alimerge email failed
05:08:57 Dropped 96 seqs from seq_list: {1232898, 1266694, 1378826, 1257998, 1306644, 1273878, 1376790, 1548312, 1044510, 1208880, 1206320, 1325106, 1350720, 1072192, 1205826, 1030218, 1380944, 1294422, 1280598, 1058394, 1371238, 1059954, 1388154, 1095292, 1328766, 1305216, 1303170, 1685634, 1839750, 1875080, 1370248, 1648776, 1875594, 1049244, 1025694, 1031328, 1373858, 1043116, 1696940, 1231024, 1391296, 1694912, 1301190, 1258198, 1650390, 1299158, 1253102, 1617144, 1064706, 1238788, 1031946, 1548570, 1387290, 1286430, 1361192, 1297704, 1008426, 1377072, 1322290, 1802040, 1224510, 1343820, 1618770, 1296724, 1764696, 1207130, 1335138, 1374054, 1648494, 1275762, 1063290, 1551738, 1292156, 1320828, 1653120, 1250690, 1008528, 1766292, 1380244, 1062810, 1280928, 1686948, 1368492, 1072050, 1343922, 1286068, 1654218, 1239500, 1229776, 1356762, 1309146, 1032668, 1327080, 1338870, 1292280, 1043454}
05:08:57 Dropped 96 seqs from datadict: {1232898, 1266694, 1378826, 1257998, 1306644, 1273878, 1376790, 1548312, 1044510, 1208880, 1206320, 1325106, 1350720, 1072192, 1205826, 1030218, 1380944, 1294422, 1280598, 1058394, 1371238, 1059954, 1388154, 1095292, 1328766, 1305216, 1303170, 1685634, 1839750, 1875080, 1370248, 1648776, 1875594, 1049244, 1025694, 1031328, 1373858, 1043116, 1696940, 1231024, 1391296, 1694912, 1301190, 1258198, 1650390, 1299158, 1253102, 1617144, 1064706, 1238788, 1031946, 1548570, 1387290, 1286430, 1361192, 1297704, 1008426, 1377072, 1322290, 1802040, 1224510, 1343820, 1618770, 1296724, 1764696, 1207130, 1335138, 1374054, 1648494, 1275762, 1063290, 1551738, 1292156, 1320828, 1653120, 1250690, 1008528, 1766292, 1380244, 1062810, 1280928, 1686948, 1368492, 1072050, 1343922, 1286068, 1654218, 1239500, 1229776, 1356762, 1309146, 1032668, 1327080, 1338870, 1292280, 1043454}
05:08:58 Written all data and HTML
05:08:58 Saved state and finalized.

Note that I used the 290 sequences whose "recent progress" was north of 2000, and of those, there were over 90 mergers removed (with not-infrequent cases of multimergers that previously would have required two separate executions of the script, delaying their removal). This technique may or may not be enough to mostly (not entirely) handle the mergers currently being manually posted.

The other possible problem regarding the data synchronization is this: Since the beginning of this (insane) project, I had the master SeqList ordered in pseudo-random order for aesthetic reasons that really don't make sense even for aesthetic purposes; so, one or two weeks ago, I decided to do away with that and merely order the master list numerically.... but I'm beginning to think that might be part of the recent problem with data synchronization of not-really-unique additions to the master list. In "normal" circumstances, where every sequence has had a minimum of work on it and merges are rare, it doesn't especially matter what order they're updated in, but in the recent times where half the total seqlist are significantly more prone to mergers than the other half, updating the two halves separately (as is effectively done by going through in numerical order) might be a problem, since it basically guarantees that updates of a secondary and primary branch of a merge are necessarily days apart, whereas a pseudo-random update order would significantly increase the odds that a given merger pair has both branches updated relatively close together. Might the merge finding and data sync be somewhat improved (not solved, but improved) if we temporarily updated the master list to (say) alternate seqs from either half?

[fivemack]

1177692:355 = 2360:2 (might be a worthwhile merge though only got to 20 digits before coming down)
1178022:0 = 996666:1
1371630:0 = 613068:2
1329888:1 = 355968:3

(what I'm doing here is looking for recognizable primes in slightly earlier lines of the sequence ... on the other hand it seems reasonably clear that it would be worth computing the terms between 1e6 and 2e6 at the start of the sequences starting 0..1e6 and filtering those out really early, there are quite a lot of sequences starting with sigma(B)-B for B<1e6)

v=Set([1326232,1178958,1300328,1263192,1549800,1823688,1648770,1374864,1321884,1310754,1032668])
for(b=1,1000000,S=Set();i=0;a=b;while(setsearch(S,a)==0&&a!=1&&a<2*10^6,i=1+i;a=sigma(a)-a;S=setunion(S,Set(a));if(setsearch(v,a)!=0,print([b,a,i]))))

[604968, 1263192, 1]
[701680, 1206680, 1]
[701700, 1329420, 1]
[764526, 1009554, 1]
[784350, 1715490, 1]
[902538, 1718262, 1]
[916536, 1374864, 1]
[935320, 1206680, 1]
[954852, 1321884, 1]
[959430, 1343274, 1]

[Batalov]

Originally Posted by henryzz

Most of these merges seem to just being found as we search the database. Are there any that need adding to the first post due to actual work going into them?
I am thinking of putting merges > 1M in the third post.

You can put in the first post terminations that are > 1M and were higher than 100 digits and were not merges.

[Batalov]

1182984 273888
1183998 351930
1185498 144390
1190756 481068
1199862 607320
1313778 901302
1314690 781302
1372828 51060
1395508 869008
1396014 316080
1415412 162216
1421094 659382
1421262 710610
1427490 883674
1432488 391656
1433568 211170
1452282 877014
1538952 828408
1541382 846006
1547574 1125606
1549800 504594
1550112 736032
1554880 1507116
1561362 1347726
1561536 836358
1562288 1187536
1566870 1342974
1568046 1062678
1568588 1316700
1568840 498540
1989108 1553820

[Batalov]

1585662 1331202
1587534 1054494
1593306 1412010
1604574 1589250
1606650 753372
1611630 781254
1615040 147192
1615410 1319754
1616860 1091300
1619220 507348
1619400 627000
1620264 264954
1622130 1345422
1628994 1017990
1632270 1237236
1632918 1038318
1637508 1237000
1637892 541482
1638680 1026520
1639316 1199120

[Drdmitry]

Originally Posted by Batalov

You can put in the first post terminations that are > 1M and were higher than 100 digits and were not merges.

I suggest to put terminations and merges of >1M sequences to the third post or to the second but not to the first one. I prefer to separate >1M sequences from <1M ones.

[Batalov]

1653396 1056108
1989246 1682754