I woke up this morning (Well actually afternoon but meh), and powered up my desktop. Only to see that my server cluster was down.

So I headed down, to see what happened, seems that it had crashed, seems to have been a kernel panic. However I also saw quite a few disk errors.

So after doing a `cat /proc/mdstat` I was greeted by the following:

[root@MasterChief xeross]# cat /proc/mdstat
Personalities : [raid1] [raid0]
md1 : active raid1 sda4[1]
19431356 blocks super 1.1 [2/1] [_U]
bitmap: 1/1 pages [4KB], 65536KB chunk

md2 : inactive sda2[1]
1949696 blocks super 1.1

md0 : active raid1 sda1[1]
102388 blocks super 1.0 [2/1] [_U]

md3 : active raid1 sda3[2]
955271935 blocks super 1.2 [2/1] [U_]

 

Seems one of my disks died, I’m currently running full back-ups of all VMs in case the 2nd one decides to give up too.

Luckily the disk still has warranty, so I’m going to swap it for a new one, and hopefully get everything mirrored again soon.

In the meantime I’ll just run part of the cluster so I can at least continue developing.

~Xeross