Misc Links
Forum Archive
News Archive
File DB
 

Ads
 

Advertisement
 

Latest Forum Topics
wow 56 k modems are
Posted by Red Squirrel
on Oct 14 2013, 11:52:23 pm

I Need A Program
Posted by rovingcowboy
on Sep 23 2013, 5:37:59 pm

having trouble witn lan
Posted by rovingcowboy
on Sep 23 2013, 5:40:56 pm

new problem for me
Posted by rovingcowboy
on Sep 23 2013, 5:54:09 pm

RBC Royal Bank
Posted by Red Squirrel
on Aug 13 2013, 6:48:08 pm

 

How to Use MDADM Linux Raid
A highly resilient raid solution!
By Red Squirrel


Dealing With Drive Failures
While today's drives are much faster and bigger than before, this comes with a cost: reliability. While drives may be more reliable on a per MB bassis, they are less reliable on a per drive bassis. While ANY drive can fail, newer ones are more likely to fail, especially when you have lot of them, as you are simply increasing your chances of a single failure. Thankfully, with any raid level above 0 you can survive at least one failure. You just need to be ready to deal with it.

Unless you have a fancy disk controller and case where a LED will go red when a drive fails, it is very important to document your drives upon installation. Using the smartctl command you can find out the serial number of a particular drive. What you want to do is insert the drives one by one, type dmesg -c to find out the name of the drive as detected by the system, then use smartctl -a /dev/[name] to get the serial number. Make a spreadsheet of the drive bays on your case, and enter the serial number as well as other info such as the size. You can even put which array they are for if there are multiple arrays, but the important piece of information is the serial number.

With alerting setup, and your drives properly documented you can have piece of mind that when a drive fails you will know about it, and you will be ready to deal with it. A drive failure email will look something like this:

This is an automatically generated mail message from mdadm
running on raidtest.loc

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdd.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde[4] sdd[2](F) sdc[1] sdb[0]
      3144192 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UU_U]

unused devices: 

From that you know that /dev/sdd has failed. Login to the server to confirm the failure.




[root@raidtest ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid5
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 16:14:21 2011
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 75

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       0        0        2      removed
       4       8       64        3      active sync   /dev/sde

       2       8       48        -      faulty spare   /dev/sdd
[root@raidtest ~]#



Remove it from the array before pulling the drive out:


[root@raidtest ~]# mdadm --manage --remove /dev/md0 /dev/sdd
mdadm: hot removed /dev/sdd from /dev/md0
[root@raidtest ~]#


Get the serial number, so you can refer to your documentation to know which bay it's in:


[root@raidtest ~]# smartctl -a /dev/sdd | grep -i serial
Serial Number:    VB455d882e-8013d7c9
[root@raidtest ~]#


Physically go remove the failed drive, replacing it with a new one and then add it to the array:



[root@raidtest ~]# mdadm --add /dev/md0 /dev/sdd
mdadm: added /dev/sdd
[root@raidtest ~]#
[root@raidtest ~]#
[root@raidtest ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid5
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 17:03:44 2011
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 36% complete

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 85

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       5       8       48        2      spare rebuilding   /dev/sdd
       4       8       64        3      active sync   /dev/sde
[root@raidtest ~]#

Let it rebuild, and you will now have a fully redundant array again. If another drive fails before the point a new drive is added and the rebuild is complete, you could lose all data. There are several ways around this. One way to at least minimize this chance is to add a hot spare, which will cause the rebuild process to start right away if a drive fails, minimizing the "danger time". All you need to do is simply add another drive (as we just did) while the array already has enough. It will automatically get picked up and a rebuild will occur if a drive fails. Another solution is to have another raid level such as raid 6 which can survive 2 failures. Raid 10 can also work but it's not always 2 drives that can fail as it depends which drives go. If you are very paranoid, you can combine raid levels such as do a raid 61. Basically, this is two raid 6 arrays that are then added as a raid 1. So think of the two raid 6 arrays as two hard drives, then take those hard drives and make a raid 1. This is kind of wasteful though, you lose over half of all total drives worth of space. You are better off sticking with just raid 6, and just make sure you have good backups. If you want to play around though, you could try it.

Let's just stick to raid 6. First we need to add another drive, then transform the array. We will not gain any disk space out of this, but we will gain redundancy.


[root@raidtest ~]# mdadm --add /dev/md0 /dev/sdf
mdadm: added /dev/sdf
[root@raidtest ~]#
[root@raidtest ~]# mdadm --grow /dev/md0 --level=6 --raid-devices=5 --backup-file=/root/backup
mdadm level of /dev/md0 changed to raid6
[root@raidtest ~]#
[root@raidtest ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdf[6] sdd[5] sde[4] sdc[1] sdb[0]
      3144192 blocks super 1.2 level 6, 512k chunk, algorithm 18 [5/4] [UUUU_]
      [==>..................]  reshape = 13.2% (139264/1048064) finish=4.4min speed=3373K/sec

unused devices: 
[root@raidtest ~]#


Note the backup file argument. You will want to specify a location that is NOT in the array. I am honestly not sure how much space is really required for this backup file, to be safe I would just use the largest disk you have. It will be deleted after the rebuild is complete. You also need mdadm 3.1 or higher for this feature to work. It is rather new, and you should really backup your stuff before doing this (or ANY) changes to a raid array. Expect this reshape to take a while compared to just a normal rebuild. I found it took about 2x as long. As far as I know, if a drive was to fail here, you would probably be safe as it is still a raid 5. But I must repeat: backup your data! Raid is not a backup! Here's how it will look like when done:




[root@raidtest ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid6
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 18:54:26 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 2058

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       5       8       48        2      active sync   /dev/sdd
       4       8       64        3      active sync   /dev/sde
       6       8       80        4      active sync   /dev/sdf
[root@raidtest ~]#



If for whatever reason you want to go from 6 to 5, you can also do that. You'll then end up with a hot spare.

Growing an array

Another nice thing about mdadm is the ease of growing an array. Say you start off with 5 drives in a raid 6 and you are running out of space, no problem. Simply add another drive, and grow the array. The drive needs to be the same size as all the others of course. You can also add multiple drives to grow it much bigger in one shot. Let's add 2GB to this array by adding 2 more drives.


[root@raidtest ~]# mdadm --add /dev/md0 /dev/sdg /dev/sdh
mdadm: added /dev/sdg
mdadm: added /dev/sdh
[root@raidtest ~]#



Now you simply need to run a grow command:


[root@raidtest ~]# mdadm --grow /dev/md0 --raid-devices=7
mdadm: Need to backup 7680K of critical section..
[root@raidtest ~]#



That bit of info almost looks like an error but it's just some info you don't really have to worry about. As you can see, it worked:


[root@raidtest ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid6
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 7
  Total Devices : 7
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 18:58:53 2011
          State : clean, recovering
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

 Reshape Status : 7% complete
  Delta Devices : 2, (5->7)

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 2073

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       5       8       48        2      active sync   /dev/sdd
       4       8       64        3      active sync   /dev/sde
       6       8       80        4      active sync   /dev/sdf
       8       8      112        5      active sync   /dev/sdh
       7       8       96        6      active sync   /dev/sdg
[root@raidtest ~]#


(later)


[root@raidtest ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid6
     Array Size : 5240320 (5.00 GiB 5.37 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 7
  Total Devices : 7
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 19:01:15 2011
          State : clean
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 2089

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       5       8       48        2      active sync   /dev/sdd
       4       8       64        3      active sync   /dev/sde
       6       8       80        4      active sync   /dev/sdf
       8       8      112        5      active sync   /dev/sdh
       7       8       96        6      active sync   /dev/sdg
[root@raidtest ~]#



Now remember, a raid device is like a hard drive. Just because your hard drive is bigger, does not mean your file system sees that extra space! We now need to expand the file system volume. This procedure has nothing to do with mdadm and is based on the file system. We will use the resize2fs command which is fairly simple:


[root@raidtest ~]# resize2fs /dev/md0
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/md0 is mounted on /mnt/md0; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/md0 to 1310080 (4k) blocks.
The filesystem on /dev/md0 is now 1310080 blocks long.

[root@raidtest ~]# df -hl
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             7.9G  2.9G  4.7G  39% /
tmpfs                 499M     0  499M   0% /dev/shm
/dev/sda1             194M   25M  159M  14% /boot
/dev/md0              5.0G   70M  4.7G   2% /mnt/md0
[root@raidtest ~]#


There you have it, we made a 4 disk raid 5 array, dealt with a disk failure, converted to raid 6 and increased it's size by 2GB. All this was done without ever bringing the file system offline. You could have VMs or any other data running on there during any of these procedures. The procedures we went through could occur over the course of any time such as several years, and you never need to bring it offline

There are much more things to explore such as chunk sizes and other settings, but this article was only to really scratch the surface of mdadm. I have been using mdadm for over 5 years without any issues. My data has survived some pretty harsh crashes due to various hardware failure, hard power downs due to failing UPS, and other such crashes. At one point I even lost 2 drives out of a raid 5 forcing the array offline due to some unexplainable hardware issue. The drives were not failed, there was some other hardware issue. I was later on able to bring both drives back into the array, rebuild, run a fsck on the file system, and short of VMs and other active files, all the data was fine. I restored the backups of the VMs and was ready to rock and roll. It is truly a resilient system and I actually recommend it over hardware raid for mass data storage simply because you are not relying on a specific controller, and you can even spam arrays across multiple controllers.







Next Page
spacer
29906 Hits Pages: [1] [2] [3] [4] 0 Comments
spacer


Latest comments (newest first)
Be the first to post a comment!


Top Articles Latest Articles
- What are .bin files for? (669062 reads)
- Text searching in linux with grep (161180 reads)
- Big Brother and Ndisuio.sys (150471 reads)
- PSP User's Guide (139547 reads)
- SPFDisk (Special Fdisk) Partition Manager (117240 reads)
- How to Use MDADM Linux Raid (188 reads)
- What is Cloud Computing? (1225 reads)
- Dynamic Forum Signatures (version 2) (8769 reads)
- Successfully Hacking your iPhone or iTouch (18714 reads)
- Ultima Online Newbie Guide (35906 reads)
corner image

This site best viewed in a W3C standard browser at 800*600 or higher
Site design by Red Squirrel | Contact
© Copyright 2017 Ryan Auclair/IceTeks, All rights reserved