GPU to 72 status...

[chalsall]

Originally Posted by James Heinrich

If I had RAID1 with two Seagate drives and one failed, I would expect the second one to also fail before the array could be rebuilt.

Yeah. That's why I always insist on different manufacturers for the drives in RAID arrays provisioned for my clients. MTBF can become a bit of a problem when all of them fail right around the same time....

[chalsall]

Originally Posted by chalsall

It is scheduled to be replaced in the next hour or so. It's "hot-swappable", and one of the drives in a RAID1 set, so there should be no downtime.

OK. The One and One techs have just now removed the /dev/sda drive.

[chalsall]

Originally Posted by chalsall

OK. The One and One techs have just now removed the /dev/sda drive.

And then all hell let loose. "Could I please speak to your supervisor? "Sure, please let me put you on hold...

[chalsall]

Originally Posted by chalsall

And then all hell let loose. "Could I please speak to your supervisor? "Sure, please let me put you on hold...

And then I finally got to do "real-time" with a level 3 tech, who understood what I was saying and laughed at my jokes.

Everything should be back to nominal. Please let me know if anyone sees anything odd.

[chalsall]

Originally Posted by chalsall

Everything should be back to nominal. Please let me know if anyone sees anything odd.

Just so everyone knows, the RAID array is rebuilding; should complete in about three days... (!) Things are a little sluggish because of this.

But the good news is the new drive is a different model (unfortunately still a Seagate), and the surviving drive in the array has zero non-recoverable read errors. Plus I have a script taking hourly snapshots of the DB mirrored to another server.

Edit: Just because it amuses my sorry ass...

[kladner]

Is gpu72.com down?

It's not just you! gpu72.com looks down from here.

[ET_]

Originally Posted by kladner

Is gpu72.com down?

It's not just you! gpu72.com looks down from here.

Back up.

[James Heinrich]

Looks up here and now.

[chalsall]

Originally Posted by James Heinrich

Looks up here and now.

Sorry guys.

As I said before, the /dev/sda drive was failing hard, so I had the techs replace it. It was hot-swappable, but that didn't go so well.

Six days later (this Sunday morning) I woke up to the server not responding to anything. Even the serial console was non-responsive. A shame, since the RAID1 rebuild was up to 39.1% complete.

A hard reset later, and the machine was back. But not responding well.

TL;DR: Don't run mprime on a Linux server during a RAID1 rebuild. The rebuild estimation has dropped from two weeks to two days. Plus the server is a whole lot more responsive to HTTP and MySQL requests.

Oh, the irony!

[retina]

Originally Posted by chalsall

TL;DR: Don't run mprime on a Linux server during a RAID1 rebuild. The rebuild estimation has dropped from two weeks to two days.

So the rebuild process has an equal or lower priority than mprime?

[chalsall]

Originally Posted by retina

So the rebuild process has an equal or lower priority than mprime?

The rebuild process (in this case, md3_resync) is at priority 20, nice of 0, while mprime is priority 30, nice of 10.

But it seems that because mprime is so incredibly efficient at saturating the CPUs, the OS wasn't able to prevent it from degrading the md3_resync process significantly.