Upgrading a RAID-1 Ubuntu-server

It’s almost 4 years now since I installed my server and I decided it was time for some new (bigger) harddisks. I wrote about my server earlier in my blog, but here is a short summary about its current state before the upgrade:

The two 1TByte hdd were in a RAID-1 configuration, according to the following specs:

with both data and system in a single, flat volume. Previously I had always kept the system separate from my data, but lately I became lazy…

Now I wanted to swap the two harddisk for new ones with 2TByte capacity. I googled the internet and found an uncountable number of guides and discussions, indicating that this might not be compeletely simple.

Therefore I now add yet another description of a successful upgrade and swap of the drives.

1. say goodbye to your server and shut it down
   root@phoenix:/home/uwezi#sudo shutdown -h now
2. open the computer and unplug one of the harddisks from the RAID1 array, you could first check the drive status in case one of the drives is more reliable than the other: that one should stay in the system for now.
3. prepare a Linux live-CD. You might be able to use your server’s operating system for sme of the steps, but at leatest when it comes to resizing the filesystem you need to run another system. I used a Knoppix 7.2 from herewww.knoppix.org
4. take the first of the new drives out of its bag and connect it to your server.
5. boot the system off the live-CD. You will not need a graphical system, booting into a console might save you some time. For Knoppix give it the following command at boot:
   knoppix 2
6. Analyze the structure of the remaining disk, in order to identify its partitions:
   root@Knoppix:/# fdisk /dev/sda
   Command (m for help): p
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sda1   *        1306      121601   971513900   83  Linux
   /dev/sda2               1        1306    10485760   82  Linux swap / Solaris
7. for all but the main partition use fdisk now to create the same structre on the new drive. The main partition will receive all the remaining space.
   root@Knoppix:/# fdisk /dev/sdb
   Command (m for help): p
   
   Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1   *        1306      243202  1943027800   83  Linux
   /dev/sdb2               1        1306    10485760   82  Linux swap / Solaris
8. Knoppix does not by default look for RAID arrays, but now we have come to the point where we will include the new drive into the array. Lets assume that /dev/sda is your old drive and /dev/sdb your new, bigger one. The information about the old RAID array is still on /dev/sda:
   root@Knoppix:/# mdadm --examine --scan /dev/sda >> /etc/mdadm/mdadm.conf
   root@Knoppix:/# mdadm --assemble --scan

   Then you can check the status of your RAID array at this point in time:

   root@Knoppix:/# cat /proc/mdstat
   md0 : active raid1 sda1
   xxxxxxxxxx blocks [1/2] [U_]

   indicating that one drive from the array is down or gone.

9. now you add your newly prepared partition on the new drive to the array:
   root@Knoppix:/# mdadm --add /dev/sdb1
10. the system will now take all the data from the remaining drive and copy it onto the new mirror drive. This will take a while, but you can watch the progress using
    root@Knoppix:/# cat /proc/mdstatmd0 : active raid1 sda1
    xxxxxxxxxx blocks [2/2] [UU]
    [****>.....................] recovery = 15% (xxxxx/yyyyyy)

    On my system it took about 3 hours to copy everything.

11. now everything is copied and your new drive is a mirror of the contents of the RAID array. It’s time to swap the other drive. Shutdown the system again, disconnect the old harddisk and connect the new harddisk.
12. boot back into the live-CD console and repeat the above steps:
13. Analyze the structure of the remaining disk, in order to identify its partitions:
    root@Knoppix:/# fdisk /dev/sdb
    Command (m for help): p
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdb1   *        1306      243202  1943027800   83  Linux
    /dev/sdb2               1        1306    10485760   82  Linux swap / Solaris
14. use fdisk now to create the same structre on the new drive.
    root@Knoppix:/#  fdisk /dev/sda
    Command (m for help): p
    
    Device Boot      Start         End      Blocks   Id  System
    /dev/sda1   *        1306      243202  1943027800   83  Linux
    /dev/sda2               1        1306    10485760   82  Linux swap / Solaris
15. Now /dev/sdb is your “old drive” and /dev/sda the second new one. The information about the old RAID array is still on /dev/sdb:
    root@Knoppix:/# mdadm --examine --scan /dev/sdb >> /etc/mdadm/mdadm.conf
    root@Knoppix:/# mdadm --assemble --scan

    Then you can check the status of your RAID array at this point in time:

    root@Knoppix:/# cat /proc/mdstat
    md0 : active raid1 sdb1
    xxxxxxxxxx blocks [1/2] [_U]

    indicating that one drive from the array is down or gone.

16. now you add your newly prepared partition on the new drive to the array:
    root@Knoppix:/# mdadm --add /dev/sda1
17. the system will now take all the data from the remaining drive and copy it onto the new mirror drive. This will take a while, but you can watch the progress using
    root@Knoppix:/# cat /proc/mdstatmd0 : active raid1 sda1 sdb1
    xxxxxxxxxx blocks [2/2] [UU]
    [****>.....................] recovery = 15% (xxxxx/yyyyyy)

    On my system it again took about 3 hours to copy everything.

18. Now I had a 1TByte RAID 1 partition’s data on a 2TByte partition. The next step is to resize the RAID array:
    root@Knoppix:/# mdadm --grow /dev/md0 --size=max

    This again will take some time to finish…

19. Now you’ll have to check/correct the filesystem on the RAID volume, then resize it and check it again:
    root@Knoppix:/# e2fsck -f /dev/md0
    root@Knoppix:/# resize2fs /dev/md0
    root@Knoppix:/# e2fsck -f /dev/md0
20. Hopefully everything went fine so far, but in my case I still did not have a running system: the grub boot loader had not been copied in the process. And here the internet was somewhat reluctant to give the ultimate clues. I had to look up information from different sites and discussions to get it working without mysterious errors or warnings.
21. Still in the live-CD’s console create a mount-point and mount your RAID array:
    root@Knoppix:/# mkdir /media/md0
    mount /dev/md0 /media/md0
22. Now for the first time you can browse the contents of your drives – it is quite comforting to see all your file again, but in case anything had gone wrong you would still have both old drives with unchanged contents!
23. Finally you have to also mount the virtual file systems of the life-CD inside the mounted RAID-array:
    root@Knoppix:/# for i in /dev /dev/pts /proc /sys /run; do mount -B $i /media/md0$i; done

    There is no /run in a Knoppix live-system, so you will get an error message…

24. And finally you can switch context to the mounted RAID system and execute the installer for the bootloader:
    root@Knoppix:/# chroot /media/md0
    grub-install /dev/sda
    grub-install /dev/sdb
25. Shutdown the live-CD system, take out the CD and reboot your computer – and if you are as lucky as I am you have a running system again!

Total downtime including some sleep was about 24 hours.

Thanks especially to

• ypchen: DNS-321 RAID-1 upgrade…
• MeierFra: Boot Problems:Cannot Find A Device For boot/grub
• …and of course all contributors to Ubuntu, Knoppix and Linux, as well as WordPress

The next big project will be an update/upgrade of the Ubuntu-version my server is running… If only I dared…