|
There was a massive fire yesterday in a new apt. complex that was under construction less than a half mile from me while I was out of town. They shut off the electricity in the surrounding neighborhoods. Several houses in the neighborhood north of me burned and the wind was blowing strongly out of the north towards the south and my neighborhood. The houses that burned were only about a quarter mile to my north. Fortunately I was already scheduled to come back today and my house was not affected. Electricity was restored late yesterday and I got home an hour ago and just now got the servers running again.
Google "Overland Park KS fire" if you want to read about it. Below is one link. It was an 8-alarm fire and was the biggest in Overland Park history. (Overland Park is a suburb of Kansas City, pop. 100,000-150,000.) Even fire fighters from Kansas City over 10 miles away responded. I don't think they've determined what started it yet but several people said they heard a big boom so maybe an explosion. I feel very fortunate. [URL]http://www.kmbc.com/article/fire-rips-through-overland-park-apartment-complex/9158681[/URL] |
[QUOTE] I feel very fortunate. [/QUOTE]
Glad everything was okay. Fires are scary, unpredictable. |
[QUOTE=gd_barnes;455262]There was a massive fire yesterday in a new apt. complex that was under construction less than a half mile from me while I was out of town. They shut off the electricity in the surrounding neighborhoods. Several houses in the neighborhood north of me burned and the wind was blowing strongly out of the north towards the south and my neighborhood. The houses that burned were only about a quarter mile to my north. Fortunately I was already scheduled to come back today and my house was not affected. Electricity was restored late yesterday and I got home an hour ago and just now got the servers running again.
Google "Overland Park KS fire" if you want to read about it. Below is one link. It was an 8-alarm fire and was the biggest in Overland Park history. (Overland Park is a suburb of Kansas City, pop. 100,000-150,000.) Even fire fighters from Kansas City over 10 miles away responded. I don't think they've determined what started it yet but several people said they heard a big boom so maybe an explosion. I feel very fortunate. [URL]http://www.kmbc.com/article/fire-rips-through-overland-park-apartment-complex/9158681[/URL][/QUOTE] I saw an video about it in the german media. That was a massive fire, I hope that no one got hurt. |
A couple of months ago the NPLB DR (disaster recovery) server started "throwing its toys out of the cradle".
During the DR server outages backups were still being captured at the DR location but not restored successfully to the DR server. The NPLB DR server now sits inside a comfortable "Redback" 12U cabinet with master cooling and a new PSU has has been installed. (personal opinion - I think the server room is now several degrees cooler than it used to be - old PSUs should be herded out to the back paddock and dealt with appropriately :edit: ) No further problems have occurred over the last 2 months. The DR recovery process has been re-tested several times successfully and the NPLB DR server is now updated to the last restore as at "Last update: 2017-05-22 02:07:14" including all databases, log files and candidates. |
[QUOTE=AMDave;459596]A couple of months ago the NPLB DR (disaster recovery) server started "throwing its toys out of the cradle".
During the DR server outages backups were still being captured at the DR location but not restored successfully to the DR server. The NPLB DR server now sits inside a comfortable "Redback" 12U cabinet with master cooling and a new PSU has has been installed. (personal opinion - I think the server room is now several degrees cooler than it used to be - old PSUs should be herded out to the back paddock and dealt with appropriately :edit: ) No further problems have occurred over the last 2 months. The DR recovery process has been re-tested several times successfully and the NPLB DR server is now updated to the last restore as at "Last update: 2017-05-22 02:07:14" including all databases, log files and candidates.[/QUOTE] Cant reach the server anymore. |
Me neither.
The last known IP address is unreponsive, but I can tell you that the last log message I got from the server was timestamped at "23/05/2017 11:05:50 " |
I had a power blip. It is back up now.
|
down again :/
|
Back up. Connectivity issue.
|
It seems that the DR server is in a more reliable location than the main server.
|
The noprimeleftbehind.net server appears to have suffered a hard reboot overnight. (The website was up and running fine, but the PRPnet servers were all down. The servers' lock files had not been cleaned up indicating the shutdown was hard.)
I've restarted all the PRPnet servers as of a few minutes ago. |
Thanks Max!
I just finished migrating my own servers to IPv6 DDNS The old address for the NPLB DR machine is no more (nplb.no-ip.org is now defunct) The NPLB DR host is now at [URL="http://noprimeleftbehind.dynv6.net"]noprimeleftbehind.dynv6.net[/URL] |
I just now got home from out of town. Thanks Max for taking care of that! :smile:
|
Some things are down?
[url]www.noprimeleftbehind.net[/url] is not working for me (Finland/Europe).
Recent results in [url]http://noprimeleftbehind.dynv6.net/stats/index.php?content=recent[/url] are in the state of: Last update: 2017-11-17 02:11:22 Server Time: 2017-11-26 19:14:18 My result was the last one in... |
I may have a hard drive issue. The machine shut down and I cannot get it to reboot. I will look more into it on Sunday afternoon.
All data is backed up relatively frequently so any loss of data would be minimal. |
I cannot get the machine to boot. Dave, what is the status of the backup?
|
Well, I determined that the problem is not the hard drive. In the various attempts at rebooting the machine I had messed up the boot area. I was able to run the fsck -v command and the computer booted up fine.
I then started the servers and it ran for a while. It accepted all work units that it was waiting on, sent out new ones, and then proceeded to reboot itself again for no particular reason. I'm now thinking I have a bad motherboard or possibly a bad power supply. I will try replacing the power supply. I have a few spares laying around. |
After replacing the power supply the machine still wants to intermittently reboot itself. Sometimes it goes a few hours, sometimes only a few minutes.
I'm guessing now that it is the motherboard. So...I'll keep the servers running as best as I can until I can get the motherboard replaced. |
I have moved the server machine hard drive to another machine and it no longer reboots itself. I confirmed that the new machine has internet access. I started the servers but nothing happens. No work units are being received or sent. Also CRUS's and NPLB's web pages are not accessible from the normal links on our projects.
The machine is very similar to the previous server machine. Slightly older motherboard and CPU both very similar. I feel like there is something wrong in the internet setup even though the machine connects fine to the internet. Does anyone have any ideas? |
Apologies for the delay in responding. I did get sent the notification of your post but it was languishing in my spam filter.
The server is still sending me the log files so outbound comms is working and therefore I also have the IP address. We just need to fix the inbound routing, so that may be either on the router or DynDNS. I will be back home tonight so I'll get a good look at it then. Last restore on the DR machine was "Last update: 2017-11-17 02:11:22" [url]http://noprimeleftbehind.dynv6.net/stats/[/url] although I am still having an issue with the DynDNS automatic update but otherwise all good there. |
It appears that the NPLB server is updating the dynamic DNS route correctly.
I have confirmed that the server is reporting the correct address and is updating the dynamic DNS index with the correct address. I have to suggest that there is a setting on the router connected to the server that is the issue. Perhaps the port-forward rule is bound to the old MAC address of NIC on the old motherboard. I cannot resolve it from here. If Max has some time he may be able to help check the router config, |
I sent a PM to Max asking for his help.
I looked at the router config on the new machine and everything looks fine to me. Obviously something needs to be changed somewhere. |
when the connection is restored I need to do some DB maintenance to restore the stats.
The main table is busted due to HDD write corruption when the mobo hicupped Here is the excerpt from the logfile that I get. [CODE]Completed local_port_status at Fri Dec 1 06:00:09 2017 DBD::mysql::st execute failed: Table './nplbstats/results' is marked as crashed and last (automatic?) repair failed at /home/nplb/scripts/refresh_database.pl line 476. DBD::mysql::st execute failed: Table './nplbstats/results' is marked as crashed and last (automatic?) repair failed at /home/nplb/scripts/refresh_database.pl line 675. DBD::mysql::st execute failed: Table './nplbstats/results' is marked as crashed and last (automatic?) repair failed at /home/nplb/scripts/refresh_database.pl line 805. Starting refresh_database at Fri Dec 1 06:00:09 2017[/CODE] |
Dave's guess was exactly right, it's a silly router issue: the port-forwards for the Internet-facing servers are bound to the MAC address of the Ethernet port on Jeepford's old motherboard. (We've been using a DHCP reservation instead of directly configuring Jeepford's static IP ever since we set up Gary's new router a year or two ago.)
I can fix this easily, but to do so I need access to the router's configuration page, which would be easy if not for the fact that (due to this issue) I don't have remote access to Gary's network. :smile: So, I've PM'd him directions to download and run a handy-dandy little utility called "[url=https://ngrok.com[/url]ngrok[/url]" which will bypass the router and create a direct tunnel to Jeepford, which I can use to get in and fix this. |
OK I just got home. I will begin doing what Max requested. I will update this post if everything goes successfully.
Once Max has access to my "new" Jeepford machine I have no doubt that he will be able to fix the problem quickly. Edit: Max, everything that you asked for in the PM has been done. Let me know if I need to do anything else. |
All fixed! :smile: The router configuration has been updated and everything is once again reachable from the Internet. I see that clients have already happily connected to the PRPnet servers and gotten some long-awaited tests.
Gary, I'll send you some directions (with screenshots!) for how to do this yourself in case it ever happens again. I'll also send them to Dave. The fewer maintenance tasks that "only one of us can do", the better. :smile: |
[QUOTE=mdettweiler;473016]All fixed! :smile: The router configuration has been updated and everything is once again reachable from the Internet. I see that clients have already happily connected to the PRPnet servers and gotten some long-awaited tests.
Gary, I'll send you some directions (with screenshots!) for how to do this yourself in case it ever happens again. I'll also send them to Dave. The fewer maintenance tasks that "only one of us can do", the better. :smile:[/QUOTE] Oustanding! Thank you Max! |
@Dave: It seems that the NPLB port report for all 3 ports stuck at 2017-09-29.
Regards Odi |
Hi Odi
/me waves That's correct. I have confirmed control of the stats server is restored, but it is a non-trivial task to restore the broken results table from the last backup before it broke, reprocess the result files up to the point where it broke & then process the data that has accumulated. I work in a different city to where I live so I opted to defer it to this weekend when I can get a good run at it with my mind on the tasks. However, your contributions are being recorded correctly and will be included in the stats when I restore the processes in the next day or two. |
Apologies fro the stats blip.
[url]http://stats.free-dc.org/stats.php?page=proj&proj=nplb[/url] The hardware issue has been resolved and the corrupted database table has been rectified. Move along. Nothing to see here. [url]https://www.youtube.com/watch?v=pdFl__NlOpA[/url] |
NPLB stats database refresh rescheduled from every hour to 3-hourly to improve resource utilisation.
|
Some CRUS tables are also not updating and are empty.
|
[QUOTE=rebirther;473876]Some CRUS tables are also not updating and are empty.[/QUOTE]
Dave, I noticed this too. The first time I noticed it was a couple of days ago. I don't know if it is related to your change but the timing does seem coincidental. Can you check into this? There are CRUS tables that have previously been updating every hour at 30 minutes after the hour. Here are the bad tables: [URL]http://www.noprimeleftbehind.net/crus/vstats_new/crus-stats.htm[/URL] [URL]http://www.noprimeleftbehind.net/crus/vstats_new/crus-top20.htm[/URL] [URL]http://www.noprimeleftbehind.net/crus/vstats_new/crus-unproven.htm[/URL] [URL]http://www.noprimeleftbehind.net/crus/vstats_new/crus-proven.htm[/URL] Strangely this table is OK: [url]http://www.noprimeleftbehind.net/crus/tab/CRUS_tab.htm[/url] The links to the above five pages are on this page: [URL]http://www.noprimeleftbehind.net/crus/[/URL] |
Thanks for the report. I will have a look and see what I can do.
|
[QUOTE=AMDave;473954]Thanks for the report. I will have a look and see what I can do.[/QUOTE]
I found the problem on my end. For some reason the machine had lost its internet connection although the servers continued working just fine. I stopped the servers, rebooted it, restarted the servers, and the internet connection worked just fine. Subsequently at the next run time at 30 minutes after the hour, the stats pages reran and now look all good. |
Thanks.
I am still chasing down some NPLB db performance issues. |
Today I noticed, the overall stats [URL]http://www.noprimeleftbehind.net/stats/index.php?content=participant_stats[/URL] starts in 10/17 and so I jumped in third position ;)
Regards Odi |
Thanks again Odi. I hadn't spotted that yet.
I have more to do tonight when I get home. |
The NPLB results summary has been repaired.
Sorry Odi.:reality-check: |
NPLB stats are running optimally again.
The regular hourly stats update schedule has been reinstated. |
The server appears to be down:
[url]https://downforeveryoneorjustme.com/www.noprimeleftbehind.net[/url] |
It was down for 2-3 hours due to a power blip and a stubborn ethernet splitter. It's been back up for several hours now.
|
Down last night and this morning:
[URL="https://downforeveryoneorjustme.com/noprimeleftbehind.net"]https://downforeveryoneorjustme.com/noprimeleftbehind.net[/URL] |
Server and ports are back up. Sorry about the outage.
|
There is something strange going on.
[url]http://noprimeleftbehind.net/tps/[/url] is up and running, but [url]http://noprimeleftbehind.net:12000/all.html[/url] is down, and I can't get any work from port 12000. |
The server machine continues to reboot itself. It may be a day before I can get it fixed.
|
Bad power supply in server machine. Replaced. Everything looks good now. :smile:
|
The NPLB and CRUS PRPnet servers are down and have been for over 12 hours.
It is not a computer issue. The NPLB and CRUS web pages are OK and I am able to update them. I am getting a MySQL error that I am unable to fix when I try to start a PRPnet server. It is: Connect to database failed: [MySQL][ODBC 3.51 Driver]Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2), native code=2002 It is an old Linux machine. I have tried stopping and restarting MySQL with: sudo service mysql stop sudo service mysql start The stop works fine. When it tries to start it fails with: * Starting MySQL database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. gary@jeepford:~$ ERROR 2013 (HY000) at line 2: Lost connection to MySQL server during query I have shut down the machine and restarted several times. A few times the PRPnet servers would run for awhile and then shut down. Other times they would not run at all. I would appreciate any help. |
Gary, check mysqld log for errors, usually it is located at /var/log/mysqld.log. Also it make sense to check *.err files in the mysql data directory (/var/lib/mysql usually). Check that you have enough disk space and free ram.
If there are errors about db/tables corruption in logs, run the next command to check: 'mysqlcheck --all-databases --check --verbose'. This command only checks db structure/integrity, for fix use with --repair option (it is good idea to backup data directory first). Hope this helps. |
Dmitry,
Here is a screen shot of my recent attempt to do what you suggested: [code] gary@jeepford:~$ sudo service mysql status [sudo] password for gary: * MySQL is stopped. gary@jeepford:~$ sudo service mysql start * Starting MySQL database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. gary@jeepford:~$ ERROR 2013 (HY000) at line 2: Lost connection to MySQL server during query gary@jeepford:~$ mysqlcheck --all-databases mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect gary@jeepford:~$ mysqlcheck --all-databases --repair mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect gary@jeepford:~$ [/code] As you can see I cannot execute the commands because I cannot connect. Any suggestions? I'm ignorant about MySQL. |
Ah, mysqlcheck command is useless if mysqld service is not even run. Gary, what you see in the log files?
|
Here are the last few pages. I hope this is enough.
[code] Jan 10 02:34:22 jeepford mysqld[8351]: This could be because you hit a bug. It is also possible that this binary Jan 10 02:34:22 jeepford mysqld[8351]: or one of the libraries it was linked against is corrupt, improperly built, Jan 10 02:34:22 jeepford mysqld[8351]: or misconfigured. This error can also be caused by malfunctioning hardware. Jan 10 02:34:22 jeepford mysqld[8351]: We will try our best to scrape up some info that will hopefully help diagnose Jan 10 02:34:22 jeepford mysqld[8351]: the problem, but since we have already crashed, something is definitely wrong Jan 10 02:34:22 jeepford mysqld[8351]: and this may fail. Jan 10 02:34:22 jeepford mysqld[8351]: Jan 10 02:34:22 jeepford mysqld[8351]: key_buffer_size=134217728 Jan 10 02:34:22 jeepford mysqld[8351]: read_buffer_size=6291456 Jan 10 02:34:22 jeepford mysqld[8351]: max_used_connections=0 Jan 10 02:34:22 jeepford mysqld[8351]: max_connections=200 Jan 10 02:34:22 jeepford mysqld[8351]: threads_connected=0 Jan 10 02:34:22 jeepford mysqld[8351]: It is possible that mysqld could use up to Jan 10 02:34:22 jeepford mysqld[8351]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2588672 K Jan 10 02:34:22 jeepford mysqld[8351]: bytes of memory Jan 10 02:34:22 jeepford mysqld[8351]: Hope that's ok; if not, decrease some variables in the equation. Jan 10 02:34:22 jeepford mysqld[8351]: Jan 10 02:34:22 jeepford mysqld[8351]: thd=(nil) Jan 10 02:34:22 jeepford mysqld[8351]: Attempting backtrace. You can use the following information to find out Jan 10 02:34:22 jeepford mysqld[8351]: where mysqld died. If you see no messages after this, something went Jan 10 02:34:22 jeepford mysqld[8351]: terribly wrong... Jan 10 02:34:22 jeepford mysqld[8351]: frame pointer is NULL, did you compile with Jan 10 02:34:22 jeepford mysqld[8351]: -fomit-frame-pointer? Aborting backtrace! Jan 10 02:34:22 jeepford mysqld[8351]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Jan 10 02:34:22 jeepford mysqld[8351]: information that should help you find out what is causing the crash. Jan 10 02:34:22 jeepford mysqld_safe[8368]: Number of processes running now: 0 Jan 10 02:34:22 jeepford mysqld_safe[8370]: restarted Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: Log scan progressed past the checkpoint lsn 65 3180302167 Jan 10 02:34:22 jeepford mysqld[8373]: 200110 2:34:22 InnoDB: Database was not shut down normally! Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: Starting crash recovery. Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: Reading tablespace information from the .ibd files... Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: Restoring possible half-written data pages from the doublewrite Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: buffer... Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: Doing recovery: scanned up to log sequence number 65 3185544704 Jan 10 02:34:22 jeepford mysqld[8373]: InnoDB: Doing recovery: scanned up to log sequence number 65 3186966404 Jan 10 02:34:22 jeepford mysqld[8373]: 200110 2:34:22 InnoDB: Starting an apply batch of log records to the database... Jan 10 02:34:46 jeepford mysqld[8373]: InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Jan 10 02:34:46 jeepford mysqld[8373]: InnoDB: Apply batch completed Jan 10 02:34:46 jeepford mysqld[8373]: 200110 2:34:46 InnoDB: Started; log sequence number 65 3186966404 Jan 10 02:34:46 jeepford mysqld[8373]: 200110 2:34:46 [Note] /usr/sbin/mysqld: ready for connections. Jan 10 02:34:46 jeepford mysqld[8373]: Version: '5.0.75-0ubuntu10.5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: Error: trying to access page number 1768430057 in space 0, Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: space name ./ibdata1, Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: which is outside the tablespace bounds. Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: Byte offset 0, len 16384, i/o type 10. Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: If you get this error at mysqld startup, please check that Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: your my.cnf matches the ibdata files that you have in the Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: MySQL server. Jan 10 02:34:47 jeepford mysqld[8373]: 200110 2:34:47InnoDB: Assertion failure in thread 140683719387472 in file fil0fil.c line 3959 Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: We intentionally generate a memory trap. Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: If you get repeated assertion failures or crashes, even Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: immediately after the mysqld startup, there may be Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: corruption in the InnoDB tablespace. Please refer to Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html Jan 10 02:34:47 jeepford mysqld[8373]: InnoDB: about forcing recovery. Jan 10 02:34:47 jeepford mysqld[8373]: 200110 2:34:47 - mysqld got signal 11 ; Jan 10 02:34:47 jeepford mysqld[8373]: This could be because you hit a bug. It is also possible that this binary Jan 10 02:34:47 jeepford mysqld[8373]: or one of the libraries it was linked against is corrupt, improperly built, Jan 10 02:34:47 jeepford mysqld[8373]: or misconfigured. This error can also be caused by malfunctioning hardware. Jan 10 02:34:47 jeepford mysqld[8373]: We will try our best to scrape up some info that will hopefully help diagnose Jan 10 02:34:47 jeepford mysqld[8373]: the problem, but since we have already crashed, something is definitely wrong Jan 10 02:34:47 jeepford mysqld[8373]: and this may fail. Jan 10 02:34:47 jeepford mysqld[8373]: Jan 10 02:34:47 jeepford mysqld[8373]: key_buffer_size=134217728 Jan 10 02:34:47 jeepford mysqld[8373]: read_buffer_size=6291456 Jan 10 02:34:47 jeepford mysqld[8373]: max_used_connections=0 Jan 10 02:34:47 jeepford mysqld[8373]: max_connections=200 Jan 10 02:34:47 jeepford mysqld[8373]: threads_connected=0 Jan 10 02:34:47 jeepford mysqld[8373]: It is possible that mysqld could use up to Jan 10 02:34:47 jeepford mysqld[8373]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2588672 K Jan 10 02:34:47 jeepford mysqld[8373]: bytes of memory Jan 10 02:34:47 jeepford mysqld[8373]: Hope that's ok; if not, decrease some variables in the equation. Jan 10 02:34:47 jeepford mysqld[8373]: Jan 10 02:34:47 jeepford mysqld[8373]: thd=(nil) Jan 10 02:34:47 jeepford mysqld[8373]: Attempting backtrace. You can use the following information to find out Jan 10 02:34:47 jeepford mysqld[8373]: where mysqld died. If you see no messages after this, something went Jan 10 02:34:47 jeepford mysqld[8373]: terribly wrong... Jan 10 02:34:47 jeepford mysqld[8373]: frame pointer is NULL, did you compile with Jan 10 02:34:47 jeepford mysqld[8373]: -fomit-frame-pointer? Aborting backtrace! Jan 10 02:34:47 jeepford mysqld[8373]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Jan 10 02:34:47 jeepford mysqld[8373]: information that should help you find out what is causing the crash. Jan 10 02:34:48 jeepford mysqld_safe[8395]: Number of processes running now: 0 Jan 10 02:34:48 jeepford mysqld_safe[8397]: restarted Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: Log scan progressed past the checkpoint lsn 65 3180302167 Jan 10 02:34:48 jeepford mysqld[8400]: 200110 2:34:48 InnoDB: Database was not shut down normally! Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: Starting crash recovery. Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: Reading tablespace information from the .ibd files... Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: Restoring possible half-written data pages from the doublewrite Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: buffer... Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: Doing recovery: scanned up to log sequence number 65 3185544704 Jan 10 02:34:48 jeepford mysqld[8400]: InnoDB: Doing recovery: scanned up to log sequence number 65 3186966404 Jan 10 02:34:48 jeepford mysqld[8400]: 200110 2:34:48 InnoDB: Starting an apply batch of log records to the database... Jan 10 02:35:13 jeepford mysqld[8400]: InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Jan 10 02:35:13 jeepford mysqld[8400]: InnoDB: Apply batch completed Jan 10 02:35:13 jeepford mysqld[8400]: 200110 2:35:13 InnoDB: Started; log sequence number 65 3186966404 Jan 10 02:35:13 jeepford mysqld[8400]: 200110 2:35:13 [Note] /usr/sbin/mysqld: ready for connections. Jan 10 02:35:13 jeepford mysqld[8400]: Version: '5.0.75-0ubuntu10.5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: Error: trying to access page number 1768430057 in space 0, Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: space name ./ibdata1, Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: which is outside the tablespace bounds. Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: Byte offset 0, len 16384, i/o type 10. Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: If you get this error at mysqld startup, please check that Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: your my.cnf matches the ibdata files that you have in the Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: MySQL server. Jan 10 02:35:14 jeepford mysqld[8400]: 200110 2:35:14InnoDB: Assertion failure in thread 140060547017040 in file fil0fil.c line 3959 Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: We intentionally generate a memory trap. Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: If you get repeated assertion failures or crashes, even Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: immediately after the mysqld startup, there may be Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: corruption in the InnoDB tablespace. Please refer to Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html Jan 10 02:35:14 jeepford mysqld[8400]: InnoDB: about forcing recovery. Jan 10 02:35:14 jeepford mysqld[8400]: 200110 2:35:14 - mysqld got signal 11 ; Jan 10 02:35:14 jeepford mysqld[8400]: This could be because you hit a bug. It is also possible that this binary Jan 10 02:35:14 jeepford mysqld[8400]: or one of the libraries it was linked against is corrupt, improperly built, Jan 10 02:35:14 jeepford mysqld[8400]: or misconfigured. This error can also be caused by malfunctioning hardware. Jan 10 02:35:14 jeepford mysqld[8400]: We will try our best to scrape up some info that will hopefully help diagnose Jan 10 02:35:14 jeepford mysqld[8400]: the problem, but since we have already crashed, something is definitely wrong Jan 10 02:35:14 jeepford mysqld[8400]: and this may fail. Jan 10 02:35:14 jeepford mysqld[8400]: Jan 10 02:35:14 jeepford mysqld[8400]: key_buffer_size=134217728 Jan 10 02:35:14 jeepford mysqld[8400]: read_buffer_size=6291456 Jan 10 02:35:14 jeepford mysqld[8400]: max_used_connections=0 Jan 10 02:35:14 jeepford mysqld[8400]: max_connections=200 Jan 10 02:35:14 jeepford mysqld[8400]: threads_connected=0 Jan 10 02:35:14 jeepford mysqld[8400]: It is possible that mysqld could use up to Jan 10 02:35:14 jeepford mysqld[8400]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2588672 K Jan 10 02:35:14 jeepford mysqld[8400]: bytes of memory Jan 10 02:35:14 jeepford mysqld[8400]: Hope that's ok; if not, decrease some variables in the equation. Jan 10 02:35:14 jeepford mysqld[8400]: Jan 10 02:35:14 jeepford mysqld[8400]: thd=(nil) Jan 10 02:35:14 jeepford mysqld[8400]: Attempting backtrace. You can use the following information to find out Jan 10 02:35:14 jeepford mysqld[8400]: where mysqld died. If you see no messages after this, something went Jan 10 02:35:14 jeepford mysqld[8400]: terribly wrong... Jan 10 02:35:14 jeepford mysqld[8400]: frame pointer is NULL, did you compile with Jan 10 02:35:14 jeepford mysqld[8400]: -fomit-frame-pointer? Aborting backtrace! Jan 10 02:35:14 jeepford mysqld[8400]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Jan 10 02:35:14 jeepford mysqld[8400]: information that should help you find out what is causing the crash. Jan 10 02:35:14 jeepford mysqld_safe[8436]: Number of processes running now: 0 Jan 10 02:35:14 jeepford mysqld_safe[8438]: restarted Jan 10 02:35:14 jeepford mysqld[8441]: InnoDB: Log scan progressed past the checkpoint lsn 65 3180302167 Jan 10 02:35:14 jeepford mysqld[8441]: 200110 2:35:14 InnoDB: Database was not shut down normally! Jan 10 02:35:14 jeepford mysqld[8441]: InnoDB: Starting crash recovery. Jan 10 02:35:14 jeepford mysqld[8441]: InnoDB: Reading tablespace information from the .ibd files... Jan 10 02:35:14 jeepford mysqld[8441]: InnoDB: Restoring possible half-written data pages from the doublewrite Jan 10 02:35:14 jeepford mysqld[8441]: InnoDB: buffer... Jan 10 02:35:14 jeepford mysqld[8441]: InnoDB: Doing recovery: scanned up to log sequence number 65 3185544704 Jan 10 02:35:15 jeepford mysqld[8441]: InnoDB: Doing recovery: scanned up to log sequence number 65 3186966404 Jan 10 02:35:15 jeepford mysqld[8441]: 200110 2:35:15 InnoDB: Starting an apply batch of log records to the database... [/code] |
Here are the first few pages after shutting machine down and turning it back on:
[code] Jan 10 02:06:07 jeepford syslogd 1.5.0#5ubuntu3: restart. Jan 10 02:06:07 jeepford anacron[3197]: Job `cron.daily' terminated (exit status: 1) (mailing output) Jan 10 02:06:07 jeepford sendmail[4500]: 00A867GF004500: from=root, size=294, class=0, nrcpts=1, msgid=<202001100806.00A867GF004500@jeepford.no-ip.org>, relay=root@localhost Jan 10 02:06:07 jeepford sm-mta[4501]: 00A867BR004501: from=<root@jeepford.no-ip.org>, size=570, class=0, nrcpts=1, msgid=<202001100806.00A867GF004500@jeepford.no-ip.org>, proto=ESMTP, daemon=MTA-v4, relay=jeepford.no-ip.org [127.0.0.1] Jan 10 02:06:07 jeepford sendmail[4500]: 00A867GF004500: to=root, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30294, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (00A867BR004501 Message accepted for delivery) Jan 10 02:06:07 jeepford anacron[3197]: Normal exit (1 job run) Jan 10 02:06:07 jeepford sm-mta[4502]: 00A867BR004501: to=<root@jeepford.no-ip.org>, ctladdr=<root@jeepford.no-ip.org> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30804, dsn=2.0.0, stat=Sent Jan 10 02:06:08 jeepford avahi-daemon[3078]: Invalid query packet. Jan 10 02:06:17 jeepford avahi-daemon[3078]: Invalid query packet. Jan 10 02:06:19 jeepford mysqld[4344]: InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Jan 10 02:06:19 jeepford mysqld[4344]: InnoDB: Apply batch completed Jan 10 02:06:19 jeepford mysqld[4344]: 200110 2:06:19 InnoDB: Started; log sequence number 65 3186966404 Jan 10 02:06:19 jeepford mysqld[4344]: 200110 2:06:19 [Note] /usr/sbin/mysqld: ready for connections. Jan 10 02:06:19 jeepford mysqld[4344]: Version: '5.0.75-0ubuntu10.5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: Error: trying to access page number 1768430057 in space 0, Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: space name ./ibdata1, Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: which is outside the tablespace bounds. Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: Byte offset 0, len 16384, i/o type 10. Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: If you get this error at mysqld startup, please check that Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: your my.cnf matches the ibdata files that you have in the Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: MySQL server. Jan 10 02:06:20 jeepford mysqld[4344]: 200110 2:06:20InnoDB: Assertion failure in thread 140681679481168 in file fil0fil.c line 3959 Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: We intentionally generate a memory trap. Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: If you get repeated assertion failures or crashes, even Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: immediately after the mysqld startup, there may be Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: corruption in the InnoDB tablespace. Please refer to Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html Jan 10 02:06:20 jeepford mysqld[4344]: InnoDB: about forcing recovery. Jan 10 02:06:20 jeepford mysqld[4344]: 200110 2:06:20 - mysqld got signal 11 ; Jan 10 02:06:20 jeepford mysqld[4344]: This could be because you hit a bug. It is also possible that this binary Jan 10 02:06:20 jeepford mysqld[4344]: or one of the libraries it was linked against is corrupt, improperly built, Jan 10 02:06:20 jeepford mysqld[4344]: or misconfigured. This error can also be caused by malfunctioning hardware. Jan 10 02:06:20 jeepford mysqld[4344]: We will try our best to scrape up some info that will hopefully help diagnose Jan 10 02:06:20 jeepford mysqld[4344]: the problem, but since we have already crashed, something is definitely wrong Jan 10 02:06:20 jeepford mysqld[4344]: and this may fail. Jan 10 02:06:20 jeepford mysqld[4344]: Jan 10 02:06:20 jeepford mysqld[4344]: key_buffer_size=134217728 Jan 10 02:06:20 jeepford mysqld[4344]: read_buffer_size=6291456 Jan 10 02:06:20 jeepford mysqld[4344]: max_used_connections=0 Jan 10 02:06:20 jeepford mysqld[4344]: max_connections=200 Jan 10 02:06:20 jeepford mysqld[4344]: threads_connected=0 Jan 10 02:06:20 jeepford mysqld[4344]: It is possible that mysqld could use up to Jan 10 02:06:20 jeepford mysqld[4344]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2588672 K Jan 10 02:06:20 jeepford mysqld[4344]: bytes of memory Jan 10 02:06:20 jeepford mysqld[4344]: Hope that's ok; if not, decrease some variables in the equation. Jan 10 02:06:20 jeepford mysqld[4344]: Jan 10 02:06:20 jeepford mysqld[4344]: thd=(nil) Jan 10 02:06:20 jeepford mysqld[4344]: Attempting backtrace. You can use the following information to find out Jan 10 02:06:20 jeepford mysqld[4344]: where mysqld died. If you see no messages after this, something went Jan 10 02:06:20 jeepford mysqld[4344]: terribly wrong... Jan 10 02:06:20 jeepford mysqld[4344]: frame pointer is NULL, did you compile with Jan 10 02:06:20 jeepford mysqld[4344]: -fomit-frame-pointer? Aborting backtrace! Jan 10 02:06:20 jeepford mysqld[4344]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Jan 10 02:06:20 jeepford mysqld[4344]: information that should help you find out what is causing the crash. Jan 10 02:06:20 jeepford mysqld_safe[4515]: Number of processes running now: 0 Jan 10 02:06:20 jeepford mysqld_safe[4517]: restarted Jan 10 02:06:20 jeepford mysqld[4520]: InnoDB: Log scan progressed past the checkpoint lsn 65 3180302167 Jan 10 02:06:20 jeepford mysqld[4520]: 200110 2:06:20 InnoDB: Database was not shut down normally! Jan 10 02:06:20 jeepford mysqld[4520]: InnoDB: Starting crash recovery. Jan 10 02:06:20 jeepford mysqld[4520]: InnoDB: Reading tablespace information from the .ibd files... Jan 10 02:06:20 jeepford mysqld[4520]: InnoDB: Restoring possible half-written data pages from the doublewrite Jan 10 02:06:20 jeepford mysqld[4520]: InnoDB: buffer... Jan 10 02:06:21 jeepford mysqld[4520]: InnoDB: Doing recovery: scanned up to log sequence number 65 3185544704 Jan 10 02:06:21 jeepford mysqld[4520]: InnoDB: Doing recovery: scanned up to log sequence number 65 3186966404 Jan 10 02:06:21 jeepford mysqld[4520]: 200110 2:06:21 InnoDB: Starting an apply batch of log records to the database... Jan 10 02:06:45 jeepford mysqld[4520]: InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Jan 10 02:06:45 jeepford mysqld[4520]: InnoDB: Apply batch completed Jan 10 02:06:45 jeepford mysqld[4520]: 200110 2:06:45 InnoDB: Started; log sequence number 65 3186966404 Jan 10 02:06:45 jeepford mysqld[4520]: 200110 2:06:45 [Note] /usr/sbin/mysqld: ready for connections. Jan 10 02:06:45 jeepford mysqld[4520]: Version: '5.0.75-0ubuntu10.5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: Error: trying to access page number 1768430057 in space 0, Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: space name ./ibdata1, Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: which is outside the tablespace bounds. Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: Byte offset 0, len 16384, i/o type 10. Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: If you get this error at mysqld startup, please check that Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: your my.cnf matches the ibdata files that you have in the Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: MySQL server. Jan 10 02:06:46 jeepford mysqld[4520]: 200110 2:06:46InnoDB: Assertion failure in thread 139678118078800 in file fil0fil.c line 3959 Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: We intentionally generate a memory trap. Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: If you get repeated assertion failures or crashes, even Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: immediately after the mysqld startup, there may be Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: corruption in the InnoDB tablespace. Please refer to Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html Jan 10 02:06:46 jeepford mysqld[4520]: InnoDB: about forcing recovery. Jan 10 02:06:46 jeepford mysqld[4520]: 200110 2:06:46 - mysqld got signal 11 ; Jan 10 02:06:46 jeepford mysqld[4520]: This could be because you hit a bug. It is also possible that this binary Jan 10 02:06:46 jeepford mysqld[4520]: or one of the libraries it was linked against is corrupt, improperly built, Jan 10 02:06:46 jeepford mysqld[4520]: or misconfigured. This error can also be caused by malfunctioning hardware. Jan 10 02:06:46 jeepford mysqld[4520]: We will try our best to scrape up some info that will hopefully help diagnose Jan 10 02:06:46 jeepford mysqld[4520]: the problem, but since we have already crashed, something is definitely wrong Jan 10 02:06:46 jeepford mysqld[4520]: and this may fail. Jan 10 02:06:46 jeepford mysqld[4520]: Jan 10 02:06:46 jeepford mysqld[4520]: key_buffer_size=134217728 Jan 10 02:06:46 jeepford mysqld[4520]: read_buffer_size=6291456 Jan 10 02:06:46 jeepford mysqld[4520]: max_used_connections=0 [/code] |
try this here:
[url]https://chepri.com/mysql-innodb-corruption-and-recovery/[/url] |
[URL]https://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html[/URL]
Or follow rebirther's link but start with innodb_force_recovery = 1 Recovering InnoDB databases isn't easy, possibly the simpliest way is to completely drop database and restore from latest backup if you have any. |
I am trying Rebirther's suggested link. I was able to stop the SQL server in step 1. But I am unable to do steps 2 or 3 because I do not have permissions.
I get the following message: You do not have the permissions necessary to open the file. How do I get permissions to read, copy, write, or open these files? I know it has something to do with having to be the root user but I cannot figure out how to do that. One more thing: The my.cnf file is in /etc/mysql/my.cnf (not /etc/my.cnf). |
[QUOTE=gd_barnes;534805]I am trying Rebirther's suggested link. I was able to stop the SQL server in step 1. But I am unable to do steps 2 or 3 because I do not have permissions.
I get the following message: You do not have the permissions necessary to open the file. How do I get permissions to read, copy, write, or open these files? I know it has something to do with having to be the root user but I cannot figure out how to do that. One more thing: The my.cnf file is in /etc/mysql/my.cnf (not /etc/my.cnf).[/QUOTE] Use "chown" or "chmod" to get access. |
Yes, rebirther's link are the steps todo. Start with recovery=1 and check if the db starts.
You must be root, e.g. by "sudo bash" or "su -" to backup the IB files. |
Using the link in Reb's post as a reference.
I was able to log in as root and backup the files in step #2. Per step #3 I edited /etc/mysql/my.cnf to have the command: innodb_force_recovery = 1 I'm now to step 4. I stopped MySQL with: sudo service mysql stop Message said it stopped successfully. I attempted to restart MySQL with: sudo service mysql start I got the following message: * Starting MySQL database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. gary@jeepford:~$ ERROR 2013 (HY000) at line 2: Lost connection to MySQL server during query This is the same message that I got before. I then did an SQL status with this command: sudo service mysql status Output was this: [code] * /usr/bin/mysqladmin Ver 8.41 Distrib 5.0.75, for debian-linux-gnu on x86_64 Copyright (C) 2000-2006 MySQL AB This software comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to modify and redistribute it under the GPL license Server version 5.0.75-0ubuntu10.5 Protocol version 10 Connection Localhost via UNIX socket UNIX socket /var/run/mysqld/mysqld.sock Uptime: 1 sec Threads: 1 Questions: 2 Slow queries: 0 Opens: 12 Flush tables: 1 Open tables: 6 Queries per second avg: 2.000 [/code]My question is: Do I continue with step 5 by dumping all the tables? Or is the above error a problem that keeps me from continuing until it is fixed? I was hoping that the recovery line in my.cnf would correct this. |
So...after doing steps 2 and 3 and running into that previous error on step 4...I thought I would try step 5 with the following command:
root@jeepford:/etc/mysql# mysqldump -u gary -p prpnet1300 > dump1300.sql It then made me enter my root password, which I did. It then gave the following message: mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect I cannot seem to get away from that message. Note that I only tried to dump one of the servers. It would not allow me to use the -A command as specified in the instructions for #5. Somehow we need to get it to correctly connect to the DB so that it can be dumped. Help! |
I ran into this issue with my home server's previous OS install (Ubuntu 17.10 IIRC). I ended up reinstalling the OS (since I couldn't reinstall MySQL without killing my PIM setup), but this script may help get the server running so you can dump the contents (bypassing the permissions; make sure it's not publicly accessible):
[CODE] #!/bin/sh sudo mkdir /var/run/mysqld sudo chown -R mysql /var/run/mysqld/ sudo mysqld --skip-grant-tables [/CODE] |
[QUOTE=Happy5214;534839]I ran into this issue with my home server's previous OS install (Ubuntu 17.10 IIRC). I ended up reinstalling the OS (since I couldn't reinstall MySQL without killing my PIM setup), but this script may help get the server running so you can dump the contents (bypassing the permissions; make sure it's not publicly accessible):
[CODE] #!/bin/sh sudo mkdir /var/run/mysqld sudo chown -R mysql /var/run/mysqld/ sudo mysqld --skip-grant-tables [/CODE][/QUOTE] This did not work. I got the following gibberish: [code] gary@jeepford:~$ sudo mysqld --skip-grant-tables InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 200111 0:04:20 InnoDB: Retrying to lock the first data file InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 200111 0:04:33 InnoDB: ERROR: We were only able to scan the log up to InnoDB: 65 3180301824, but a checkpoint was at 65 3180302287. InnoDB: It is possible that the database is now corrupt! 200111 0:04:33 InnoDB: Error: page 1 log sequence number 65 3182452991 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 4 log sequence number 65 3181673642 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 6 log sequence number 65 3182821206 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 49172 log sequence number 65 3186685478 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 49153 log sequence number 65 3186345244 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 202272 log sequence number 65 3186682578 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 196609 log sequence number 65 3181146735 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 330293 log sequence number 65 3186631187 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 327681 log sequence number 65 3184275303 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 344075 log sequence number 65 3181039382 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 344065 log sequence number 65 3182245162 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 212993 log sequence number 65 3182730042 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 147457 log sequence number 65 3185010846 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 5 row operations to undo InnoDB: Trx id counter is 0 2825215232 200111 0:04:33 InnoDB: Error: page 0 log sequence number 65 3182821206 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. InnoDB: Starting in background the rollback of uncommitted transactions InnoDB: Cleaning up trx with id 0 2825225248 200111 0:04:33 InnoDB: Rollback of non-prepared transactions completed 200111 0:04:33 InnoDB: Error: page 311297 log sequence number 65 3183278764 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Error: page 114689 log sequence number 65 3181146898 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:33 InnoDB: Started; log sequence number 65 3180302287 InnoDB: !!! innodb_force_recovery is set to 1 !!! 200111 0:04:33 [Note] mysqld: ready for connections. Version: '5.0.75-0ubuntu10.5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) 200111 0:04:34 InnoDB: Error: page 278529 log sequence number 65 3182822051 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 294913 log sequence number 65 3182821788 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 131073 log sequence number 65 3183083236 InnoDB: is in the future! Current system log sequence number 65 3180302287. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 65537 log sequence number 65 3181343415 InnoDB: is in the future! Current system log sequence number 65 3180302358. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 163841 log sequence number 65 3181146680 InnoDB: is in the future! Current system log sequence number 65 3180302358. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 229377 log sequence number 65 3181075900 InnoDB: is in the future! Current system log sequence number 65 3180302358. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 32769 log sequence number 65 3186349072 InnoDB: is in the future! Current system log sequence number 65 3180302358. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 192040 log sequence number 65 3180871058 InnoDB: is in the future! Current system log sequence number 65 3180302379. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 180225 log sequence number 65 3181147455 InnoDB: is in the future! Current system log sequence number 65 3180302379. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. 200111 0:04:34 InnoDB: Error: page 43376 log sequence number 65 3181145698 InnoDB: is in the future! Current system log sequence number 65 3180302379. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: for more information. InnoDB: Error: trying to access page number 1768430057 in space 0, InnoDB: space name ./ibdata1, InnoDB: which is outside the tablespace bounds. InnoDB: Byte offset 0, len 16384, i/o type 10. InnoDB: If you get this error at mysqld startup, please check that InnoDB: your my.cnf matches the ibdata files that you have in the InnoDB: MySQL server. 200111 0:04:34InnoDB: Assertion failure in thread 139684868266320 in file fil0fil.c line 3959 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. 200111 0:04:34 - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=134217728 read_buffer_size=6291456 max_used_connections=0 max_connections=200 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2588672 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... frame pointer is NULL, did you compile with -fomit-frame-pointer? Aborting backtrace! The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. [/code] I tried it both with MySQL stopped and then again with it started. No luck. |
First enable mysql logging. I have in [mysql]
log_error = /var/log/mysql/error.log After you restarted mysql, check with e.g. "ps -ef | grep mysql" if mysql is running and with some sql command if you can access a table. You can also try to run an mysqldump. Check also the mysql error log. If mysql crashes you would see some segmentation fault of memory dumps. If mysql is not running, set recovery=2 and try again. And so on. At least with recovery=5 mysql should start and report some error in error log. But now you can run mysqldump. |
With all of your help I've made quite a bit of progress.
I changed the recovery command to: innodb_force_recovery = 5 That successfully started MySQL and allowed me to dump the databases. When doing the dump with the -A command of course it dumps them all in one file. But I got a new database error message after a long time of dumping. I had a strong suspicion which database was corrupt on my machine. It is my personal server prpnet1470. Suspecting this I dumped only prpnet1470 and there it was with the same error message! All of the other databases dumped just fine. Here is the message that I am getting for database prpnet1470 when I try to dump it: mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `CandidateTestResult` at row: 251210 So if I follow Reb's link's instructions I'm going to be dumping all the databases, dropping them all, removing some files, restarting MySQL, and restoring all of the databases from the dump. At this point, that seems like overkill. My question is: Should I just dump and drop prpnet1470? If so do I want to do step 8 which is: Remove /var/lib/mysql/ib* I'm afraid to remove all of the ib* files from that library if I am dropping only one database. I guess what I need is more specific instructions on fixing the one database that I know is corrupt, which is causing problems for MySQL. I could even just permanently delete prpnet1470 and not restore it. I was able to do enough select statements to know where I'm at in my testing so there would be no data loss. It would be nice to keep it intact but if it is too much of a problem to do so there are other personal servers that I could use. |
If you know the lib name of the affected database remove only the name of the lib file not all.
|
Gary, if you've successfully started mysql server, run mysql client and execute 'drop database prpnet1470' query in it.
|
I'm going to rename this thread Gary's Great Adventure - All I ever wanted to know about MYSQL but were to afraid to ask
|
Haha. Good one Ian. I will move the applicable posts over to the "Server outrages" thread after the issue is completely resolved.
...trying the recent suggestions shortly. |
I seem to be in a vicious loop here.
As previously specified I added the command: innodb_force_recovery = 5 to the var/mysql/my.cnf file I rebooted the machine and checked the MySQL status using: sudo service mysql status It looked good. After logging into MySQL I tried to drop prpnet1470 with: drop database prpnet1470; I got the following error: ERROR 2013 (HY000): Lost connection to MySQL server during query So...MySQL isn't starting properly due to this error and I am unable to drop the offending DB due to this error. Any suggestions? I will try increasing the force_recovery even higher to see if that does anything. |
So here is my latest attempt to start MySQL after changing my.cnf to:
innodb_force_recovery = 6 gary@jeepford:~$ sudo service mysql stop * Stopping MySQL database server mysqld [ OK ] gary@jeepford:~$ sudo service mysql start * Starting MySQL database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. So...MySQL seems to start OK but is still giving this error message. I'm stuck. |
modify /etc/my.cnf with the following settings:
net_read_timeout=600 net_write_timeout=180 wait_timeout=86400 interactive_timeout=86400 max_allowed_packet=128M Maybe the timeout is too short |
I picked another random database to drop that is no longer used: prpnet1466
The database dropped just fine. Trying Reb's suggestion next... |
Adding the parameters in Reb's recent suggestion made no difference when attempting to drop database prpnet1470.
Maybe I could just manually give myself access to all of the root and mysql files where prpnet1470 is located and delete them all. Just a crazy thought. |
[QUOTE=gd_barnes;534902]Adding the parameters in Reb's recent suggestion made no difference when attempting to drop database prpnet1470.
Maybe I could just manually give myself access to all of the root and mysql files where prpnet1470 is located and delete them all. Just a crazy thought.[/QUOTE] How big is this database? If its bigger than max_allowed_packet=128M increase the value |
[QUOTE=rebirther;534903]How big is this database? If its bigger than max_allowed_packet=128M increase the value[/QUOTE]
That's not the problem. I dumped one huge database that was 4.2 GB and it dumped fine. Problem database prpnet1470 is about 200 MB so I increased the packet size to 256M, rebooted the machine, got MySQL up and running and tried to dump it. No difference. Error: Error 2013: Lost connection to MySQL server during query when dumping table `CandidateTestResult` at row: 251210 This error occurs after dumping 131.6 MB of the file. |
A little more info:
I am able to do selects on all of the databases with the exception of the CandidateTestResult table on database prpnet1470. I can do selects on other tables in that one. So no data will be lost except a small amount on prpnet1470. But I cannot do any deletes on any table. My suspicion is that no table can be updated in any way...insert, update, or delete. That is why all of the prpnet servers start up just fine but when they try to receive a completed test an SQL error happens. This begs the question: Why does one corrupt database affect MySQL in total such that other databases cannot be updated? |
Wild assed guess. I'm going to say that the MYSQL software is corrupt. Try installing
the MYSQL software again . I'm pretty sure that won't touch your data. Be sure the MYSQL service is stopped. |
Servers are back up!
Thank you everyone for your help! I was able to determine that several of the databases were corrupt so I dropped them all, recreated them, and reloaded them...all except my personal prpnet1470. All of the PRPnet servers work great now! The problem was probably related to too many rows in prpnet1470 being hit by too many clients because that is where the problem started. Bring on those machines! :smile: I'm very sorry about the problem. |
Hey Guys, now Iam affected after windows 10 has decided to give me a bluescreen, I cant take further actions, stucking at:
mysqldump: Got error: 1146: Table 'sr5.app' doesn't exist when using LOCK TABLES Looks like a lot of tables are bad, enable skip lock tables doesnt help. Any hints? Update log: [ERROR] Cannot find or open table sr5/app from the internal data dictionary of InnoDB though the .frm file for the table exists. Maybe you have deleted and recreated InnoDB data files but have forgotten to delete the corresponding .frm files of InnoDB tables, or you have moved .frm files to another database? or, the table contains indexes that this version of the engine doesn't support. I think the files are lost. |
Again?
Is the server down again?
I can't upload my results to the server and my browser[I]s[/I] can't access the homepage. The issue started about an hour ago. |
[QUOTE=dannyridel;538575]Is the server down again?
I can't upload my results to the server and my browser[I]s[/I] can't access the homepage. The issue started about an hour ago.[/QUOTE] see news. |
A little suggestion, could you make the news send to the BOINC manager news section like PG does?
I don't look at it all day, and when I tried to return work it failed and got me alarmed looking at a "connection was reset" page. |
I assume these recent posts are not related to NPLB's PRPnet servers, which appear to be working fine.
|
[QUOTE=gd_barnes;538941]I assume these recent posts are not related to NPLB's PRPnet servers, which appear to be working fine.[/QUOTE]
yes, you can delete these posts. My DB had not so much luck then yours :/ |
[url]http://www.noprimeleftbehind.net[/url] appears to be down at the time of this post.
|
Power outage this morning. Everything is back up.
|
It looks like the servers are down. I won't be back in town until late Saturday. I will work on them then. Sorry about the problem.
|
The servers are back up. Sorry about that.
|
It seems the daily port report on [URL="http://www.noprimeleftbehind.net/stats/index.php?content=port"]http://www.noprimeleftbehind.net[/URL] is broken since some days.
Regards Odi |
[QUOTE=odicin;582122]It seems the daily port report on [URL="http://www.noprimeleftbehind.net/stats/index.php?content=port"]http://www.noprimeleftbehind.net[/URL] is broken since some days.
Regards Odi[/QUOTE] I do not know what has caused this. I have sent an Email to AMDave. Thanks for pointing it out. |
| All times are UTC. The time now is 16:50. |
Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.