Factordb wipeout?
 

As some of you know, I have been interested in fully factored numbers of the form (M61^k-1)/(M61-1). People on this forum plus yoyo invested quite some effort and helped to factor quite a few of these. Imagine my panic when I saw a complete wipeout this morning, THERE IS NOTHING on the page:
http://factordb.com/index.php?query=(M61^k-1)%2F(M61-1)

WHAT HAPPENED???

:ouch1:

Yes, noted here as well: [url]https://www.mersenneforum.org/showthread.php?p=531659#post531659[/url]
Something has gone wrong with FactorDB. Maybe a database table got corrupted?

Some data were definitely gone, see status page on [URL]http://factordb.com/status.php[/URL] : Database 781,547MB (78.15% used) and yesterday it was 84% or so on.

this must be a cleanup in response to "spam" and indiscriminate "search" that were discussed on this forum. Is there anywhere a backup of factordb? I myself have a saved page with the numbers that interest me dating from 2013 :-(

Has anyone emailed SYD (see [url]http://factordb.com/imp.html[/url] for his address)? That's more likely to get his attention than posting here.

It would also be worth asking if he would like someone to help him administer factordb.

Chris

I informed him. He will take a look into it tomorrow.

I hope he restores the numbers that he deleted, as I had a lot of them bookmarked, and it would take forever for me to find them again. Also, sequence 4198862272 is giving me a blank page.

Why would he do that in the first place? It was rude, as a LOT of my bookmarks are now showing blank pages.

EDIT: I looked through a bunch of bookmarked numbers, and it looks like it didn't erase any of the primes.

[QUOTE=Stargate38;531717]I hope he restores the numbers that he deleted, as I had a lot of them bookmarked, and it would take forever for me to find them again. Also, sequence 4198862272 is giving me a blank page.

Why would he do that in the first place? It was rude, as a LOT of my bookmarks are now showing blank pages.

EDIT: I looked through a bunch of bookmarked numbers, and it looks like it didn't erase any of the primes.[/QUOTE]

I do not think he did it on purpose. Maybe the database got broken by some reason.
But this situation demonstrates the importance of having several backup copies of this database.

There is clearly a need for a separate, parallel backup factordb.

Maybe a monthly .torrent file could be made. That way there could both be a persistent backup of the database, and it could lend itself to mass-downloading of elements without overwhelming the servers.

Now it says "Offline for ~3 hours (maintenance)", so I think Syd might be fixing it.

It's been 18 hours, and it still has the same message.

[QUOTE=Happy5214;531789]It's been 18 hours, and it still has the same message.[/QUOTE]

It now says "Full database check (will take some time)"

I briefly got access back to factordb, but then i got a "2002: SQLSTATE[HY000] [2002] Connection refused" error and now is back to "Full database check (will take some time)".

The database was 80.xx %, so Syd managed to recover part of it.

I had brief access also. When I inquired into AS 4788 it only returned the first few hundred indices. Each subsequent inquiry added a few more terms each time. I guess the internal workers were factoring the small numbers. I wanted to see what happen when it got to a C70 but it went offline too soon.

As I previously mentioned it looks like all the numbers are there but the links or association with its larger composite is missing. Which makes the data base useless in its current form. I am guessing the links make up the missing 4% which [B]unconnected[/B] stated earlier. (Status page showed 80% when I had brief access.)

It seems the site is available again, while it keeps debugging the db in the background (it's actually showing the IDs it's working on in the status page).
The database is 817,667MB (81.77% used)

I tried uploading a primality cert. Looks like it started processing. Hopefully this won't bother any recovery processes if they are still happening

[QUOTE=RichD;531815]I wanted to see what happen when it got to a C70 but it went offline too soon.
[/QUOTE]

I've restarted my script to factor numbers between 70 and 78 digits, so they are being cleared out. But it runs on an old and rather slow PC so can take some time if there are a lot to do.

It wasn't factoring numbers above 54 digits before the outage but is now clearing them up to 70 digits. Which saves me from having to work in that area. And I still have to skip over a few errors (at 71 and 75 digits I think).

Also factordb's load average is about 20 now. Is someone hammering it?

Chris

There are some scripts running which fetch the last entry of a e.g. Aliquot Sequence. The Aliquot Sequence tables are tmp tables which were truncated. Now they are recreated on first access by using the known factors. This leads to millions of DB requests.

Now nearly all the 79 digit composites have been factored (kudos to whoever did them) I've update my script to clear 70-79 digit numbers.

Chris

[QUOTE=chris2be8;532013]Now nearly all the 79 digit composites have been factored (kudos to whoever did them) I've update my script to clear 70-79 digit numbers.

Chris[/QUOTE]
I did turn some cores loose yesterday (over 100), but not sure how responsible I was, since when I stopped them all last night, almost all were crashed for some reason.

I've also created a Colab instance that factors via GMP-ECM with the GPU branch. I ran that for a while against the 80 dd composites, but it's much slower than the YAFU version for most composites.

Can your cores cope with factordb handing them a prime number when they ask for a composite? My scripts have had to skip over 1111111111111111111111111111111111111111111111111111111111111111111111111111091 an annoying number of times.

Also there's at least one broken 79 digit entry in factordb. wget returns "500 Internal Server Error" when it tries to download it.

If you want to set them loose again aim at 80 digit numbers, there's plenty of them now.

Chris

When I really turn my machines loose on the composites for any length of time, I have an RPi, tracking the composites received, to cut down on duplication. The troublesome 79 digit gets run once and then all the machines ignore it. In reality, if I don't have the RPi running, my script processes the 79 digit number so fast it doesn't take too much time before it's on to the next, but it does show up often.

Something caused most of my machines to fail yesterday when they were running the composites. I didn't investigate, but every one of the failed machines was stuck in a failure loop trying to retrieve the next composite. A restart of the script succeeded without issue, but as to what was going on, I don't know. This is what was showing:
[code]
unrecognized token: td
Error, no result found
Factoring 9 digits: width=14%
[/code]All my machines are running a CADO-NFS job now, but I have a Colab session running the 80 digit composites, also. The Python3 script I'm playing with right now on Colab is using the GPU branch of GMP-ECM and only hitting everything lightly (11e2/11e3). It's having a success rate of a little over 50%. (The last run factored 21 of 30 in, I think, about five minutes.) Unfortunately, it also means only smaller factors (<20 digits, mostly) are found and the composites that survive will have to be run again. To counter that duplication, I hope to build a YAFU version, similar to the one I already have, that uses the GPU, but I'll have to modify the ECM calls within the YAFU code and limit the curves sent with higher B1s, or the ECM part will take a huge amount of time trying to process >3000 stage 2 residues across only two CPU cores for higher b1s.

I have one core of an i3 working on 81 digits.

Seems like some low exponent Mersenne factors were missing from FDB too?

[QUOTE=mathwiz;532162]Seems like some low exponent Mersenne factors were missing from FDB too?[/QUOTE]

Yes, there were a lot of cases with exponents below 15000 where FactorDB forgot the connection between a composite number and its factors. For example, 2^293-1 was listed as an 89-digit composite number even though its 26-digit factor 40122362455616221971122353 was already in the database. And several hundred more examples. Edit: and exponent 34499, 49081, 57329, 57383 and maybe some others...

There were also about a dozen factors for exponents in the 30k and 40k ranges that were discovered in November and December but not in FactorDB. Either not reported yet or forgotten during the outage.

torrent?
 

So, what's with the idea of a factordb torrent? If it is only 1GB then it should not be too hard right? Seems like it would potentially open up all kind of useful and not so useful derivative projects.

[QUOTE=richs;532151]I have one core of an i3 working on 81 digits.[/QUOTE]
Are you working randomly or at the low end?

I'm playing with several things that may drift into the 81 digit area as 80 digits fall below 5000. Should I modify my scripts to work further into 81 digits or move to 82 digits?

[QUOTE=EdH;532295]Are you working randomly or at the low end?

I'm playing with several things that may drift into the 81 digit area as 80 digits fall below 5000. Should I modify my scripts to work further into 81 digits or move to 82 digits?[/QUOTE]

My python script is choosing randomly from the first 100 C81's. I've had 33 worker collisions in the first 1293 composites that I've done.

Slowly checking aliquot sequences <1M I've found that 933564 is absent in the FDB.

In addition to previous post:

[CODE]$ ./aliqueit -t 933564
Reading config file...
Verifying elf 933564
Verifying index 129...
ERROR: @index 129: value != sigma - n[/CODE]

[QUOTE=richs;532300]My python script is choosing randomly from the first 100 C81's. I've had 33 worker collisions in the first 1293 composites that I've done.[/QUOTE]Some of those might have been my scripts. I completed 1108 in a window of 5000. So a few of those may have been in the low end of 81. I shouldn't hit 81 at all now, but I hope to work on 82 digits some tomorrow. I still expect to work on the 80s testing my latest Colab experiment, using the GPU branch of ECM and msieve, but not so aggressively as today.

Sorry about that. . .

[QUOTE=EdH;532340]Some of those might have been my scripts. I completed 1108 in a window of 5000. So a few of those may have been in the low end of 81. I shouldn't hit 81 at all now, but I hope to work on 82 digits some tomorrow. I still expect to work on the 80s testing my latest Colab experiment, using the GPU branch of ECM and msieve, but not so aggressively as today.

Sorry about that. . .[/QUOTE]

No problem at all.

[QUOTE=richs;532362]No problem at all.[/QUOTE]
Well, now I'm really sorry! A machine I didn't even remember was running composites apparently finished off the 80s and moved into the 81s. I "think" I now have "all" of my machines working in the 82 area. I plan on leaving them there until I put them back to their normal work.

Moved my one core of an i3 to 83 digits since the 81's are cleared out.

[QUOTE=richs;532517]Moved my one core of an i3 to 83 digits since the 81's are cleared out.[/QUOTE]
I've got everything here that's running composites aimed at 82 for now. I'll try to keep better attention as 82 dwindles. . .

[QUOTE=richs;532517]Moved my one core of an i3 to 83 digits since the 81's are cleared out.[/QUOTE]
This morning I started some sessions at 79, since you're now at 83 and 79-81 are trying to build back up. . .

It seems the DB is down for maintenance atm. I'm guessing that Syd's adding more disk space (i.e. adding HDDs), or possibly fixing the broken Aliquot sequences.

[QUOTE=richs;532517]Moved my one core of an i3 to 83 digits since the 81's are cleared out.[/QUOTE]
Well, I'm impressed! I set loose 8 i7s (all 8 cores), plus 2 RPis (all 4 cores - ECM only) on 84 digits overnight and you kept 83 digit factoring just about even with my 84 digit work.:smile:

I noticed a little less than 200 in the 79-82 area this morning, so I set all four cores of an i5 and a Colab session to work there and now there are [B]over[/B] 200!:confused:

[QUOTE=EdH;532914]Well, I'm impressed! I set loose 8 i7s (all 8 cores), plus 2 RPis (all 4 cores - ECM only) on 84 digits overnight and you kept 83 digit factoring just about even with my 84 digit work.:smile:

I noticed a little less than 200 in the 79-82 area this morning, so I set all four cores of an i5 and a Colab session to work there and now there are [B]over[/B] 200!:confused:[/QUOTE]

Well someone else must be doing the heavy lifting on 83 digits since I'm only averaging 15 composites per hour.

I'm working on 83 digits in the gaps between clearing 70-79 digits. But that's taking most of my time now so I'm only doing about 20/hour. Mostly after skipping 40000-5000 composites to keep clear of your work.

And just look how many have built up under 70 digits. Factordb will take a while to clear that lot.

Chris