How To Repair RAID Brokenness

From : http://www.penguinpunk.net/blog/category/storage/qnap/

I did a post a little while ago (you can see it here) that covered using mdadm to repair a munted RAID config on a QNAP NAS. So I popped another disk recently, and took the opportunity to get some proper output. Ideally you’ll want to use the web interface on the QNAP to do this type of thing but sometimes it no worky. So here you go.

Stop everything on the box.

[~] # /etc/init.d/services.sh stop

Stop service: recycled.sh mysqld.sh atalk.sh ftp.sh bt_scheduler.sh btd.sh ImRd.sh init_iTune.sh twonkymedia.sh Qthttpd.sh crond.sh nfs smb.sh lunportman.sh iscsitrgt.sh nvrd.sh snmp rsyslog.sh qsyncman.sh iso_mount.sh antivirus.sh .

Stop qpkg service: Disable Optware/ipkg

Shutting down SlimServer...

Stopping SqueezeboxServer 7.5.1-30836 (please wait) .... OK.

Stopping thttpd-ssods .. OK.

/etc/rcK.d/QK107Symform: line 48: /share/MD0_DATA/.qpkg/Symform/Symform.sh: No such file or directory

(By the way it really annoys me when I’ve asked software to remove itself and it doesn’t cleanly uninstall – I’m looking at you Symform plugin)

Unmount the volume

[~] # umount /dev/md0

Stop the array

[~] # mdadm -S /dev/md0

mdadm: stopped /dev/md0

Reassemble the volume

[~] # mdadm --assemble /dev/md0 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3 /dev/sdf3

mdadm: /dev/md0 has been started with 5 drives (out of 6).

Wait, wha? What about that other disk that I think is okay?

[~] # mdadm --detail /dev/md0

/dev/md0:

Version : 00.90.03

Creation Time : Fri May 22 21:05:28 2009

Raid Level : raid5

Array Size : 9759728000 (9307.60 GiB 9993.96 GB)

Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)

Raid Devices : 6

Total Devices : 5

Preferred Minor : 0

Persistence : Superblock is persistent

Update Time : Wed Dec 14 19:09:25 2011

State : clean, degraded

Active Devices : 5

Working Devices : 5

Failed Devices : 0

Spare Devices : 0

Layout : left-symmetric

Chunk Size : 64K

UUID : 7c440c84:4b9110fe:dd7a3127:178f0e97

Events : 0.4311172

Number Major Minor RaidDevice State

0 8 3 0 active sync /dev/sda3

1 0 0 1 removed

2 8 35 2 active sync /dev/sdc3

3 8 51 3 active sync /dev/sdd3

4 8 67 4 active sync /dev/sde3

5 8 83 5 active sync /dev/sdf3

Or in other words

[~] # cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]

md0 : active raid5 sda3[0] sdf3[5] sde3[4] sdd3[3] sdc3[2]

9759728000 blocks level 5, 64k chunk, algorithm 2 [6/5] [U_UUUU]

md6 : active raid1 sdf2[2](S) sde2[3](S) sdd2[4](S) sdc2[1] sda2[0]

530048 blocks [2/2] [UU]

md13 : active raid1 sdb4[2] sdc4[0] sdf4[5] sde4[4] sdd4[3] sda4[1]

458880 blocks [6/6] [UUUUUU]

bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sdf1[1] sda1[0] sdc1[4] sdd1[3] sde1[2]

530048 blocks [6/5] [UUUUU_]

bitmap: 34/65 pages [136KB], 4KB chunk

unused devices: 

So, when you see [U_UUUU] you’ve got a disk missing, but you knew that already. You can add it back in to the array thusly.

[~] # mdadm --add /dev/md0 /dev/sdb3

mdadm: re-added /dev/sdb3

So let’s check on the progress.

[~] # cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]

md0 : active raid5 sdb3[6] sda3[0] sdf3[5] sde3[4] sdd3[3] sdc3[2]

9759728000 blocks level 5, 64k chunk, algorithm 2 [6/5] [U_UUUU]

[>....................] recovery = 0.0% (355744/1951945600) finish=731.4min speed=44468K/sec

md6 : active raid1 sdf2[2](S) sde2[3](S) sdd2[4](S) sdc2[1] sda2[0]

530048 blocks [2/2] [UU]

md13 : active raid1 sdb4[2] sdc4[0] sdf4[5] sde4[4] sdd4[3] sda4[1]

458880 blocks [6/6] [UUUUUU]

bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sdf1[1] sda1[0] sdc1[4] sdd1[3] sde1[2]

530048 blocks [6/5] [UUUUU_]

bitmap: 34/65 pages [136KB], 4KB chunk

unused devices: 

[~] #

And it will rebuild. Hopefully. Unless the disk is really truly dead. You should probably order yourself a spare in any case

 

How To Repair RAID BrokennessHow To Repair RAID Brokenness, 5.5 out of 10 based on 2 ratings