Yesterday Masterchief went down, and didn’t come back up, after hooking up a monitor and keyboard it is now displaying an error stating “No root device /dev/md1 found” and dropping me into the Dracut rescue shell.

I ran through some basic checks, fsck’ed the device, checked the RAID status and even successfully mounted it. So I’m now at quite the loss what’s causing this and so far #fedora hasn’t offered any help.

I’ll continue trying trying things, perhaps I can at least get it to boot and go from there.

Update 18-Jul-2012 (15:55): One of the errors seems related to the renaming of one of the LVM volume groups, will try changing it back to the old name, see if that works.
Update 18-Jul-2012 (19:41): Renaming the LVM volume group to its former name caused the server to boot again, now to figure out how to properly rename it and keep boot working.
Update 23-Jul-2012 (17:38): It’s odd that the root device is getting an error on a volume group rename, because the root volume is just a raw partition, guess we’ll poke some more.
Update 23-Jul-2012 (18:34): Not going to investigate further, renaming the LVM volume group to the original fixes it but I was going to install Arch Linux anyway.

As I posted yesterday it seemed that one of the disks in the server died, so I immediately decided to first make back-ups. Well that finally finished today.

So after shutting the server down I decided to first check all the cables, push them in further where possible, in case it might have just been a loose cable.

So after doing that, and booting into the setup screen I checked the list of disks it detected. And presto it showed up. So now the following is running on the server:

Personalities : [raid1] [raid0]
md1 : active raid1 sda4[0] sdb4[1]
      19431356 blocks super 1.1 [2/1] [_U]
        resync=DELAYED
      bitmap: 1/1 pages [4KB], 65536KB chunk

md2 : active raid0 sda2[0] sdb2[1]
      3899392 blocks super 1.1 512k chunks

md0 : active raid1 sda1[0] sdb1[1]
      102388 blocks super 1.0 [2/1] [_U]
        resync=DELAYED

md3 : active raid1 sda3[1] sdb3[2]
      955271935 blocks super 1.2 [2/1] [U_]
      [=============>.......] recovery = 69.5%
      (664634304/955271935) finish=43.2min speed=112045K/sec

unused devices: <none>

So it turns out my assessment was wrong, I immediately assumed the worst, while it was just a loose cable.

Well at least it made it so there’s full back-ups of all virtual machines again (Well recent ones).

So basically. Be sure to prepare for the worst, but don’t assume the worst when shit hits the fan.

~Xeross

I woke up this morning (Well actually afternoon but meh), and powered up my desktop. Only to see that my server cluster was down.

So I headed down, to see what happened, seems that it had crashed, seems to have been a kernel panic. However I also saw quite a few disk errors.

So after doing a `cat /proc/mdstat` I was greeted by the following:

[root@MasterChief xeross]# cat /proc/mdstat
Personalities : [raid1] [raid0]
md1 : active raid1 sda4[1]
19431356 blocks super 1.1 [2/1] [_U]
bitmap: 1/1 pages [4KB], 65536KB chunk

md2 : inactive sda2[1]
1949696 blocks super 1.1

md0 : active raid1 sda1[1]
102388 blocks super 1.0 [2/1] [_U]

md3 : active raid1 sda3[2]
955271935 blocks super 1.2 [2/1] [U_]

Seems one of my disks died, I’m currently running full back-ups of all VMs in case the 2nd one decides to give up too.

Luckily the disk still has warranty, so I’m going to swap it for a new one, and hopefully get everything mirrored again soon.

In the meantime I’ll just run part of the cluster so I can at least continue developing.

~Xeross

Well the first reasons of why our slogan fits have appeared, more will probably follow, consdering my luck.

Minecraft Back-ups

So yesterday evening the Minecraft server ran out of memory due to a misconfiguration of the admin (Not me btw), and this caused the world to go corrupt. Not a big problem on its own as we have hourly back-ups.

Well that’s what we are supposed to have, when I tried connecting to the Bacula director (The back-up management daemon) I simply couldn’t connect. Turned out it wasn’t running, since the 20th of June.

Luckily the Minecraft admin managed to resolve it using parts of the 20th of June back-up and parts of the copy I had for the map generation.

Back-ups are running again, but I really need to set-up Nagios so I actually get notified when shit doesn’t work.

Scripting Fuck-up

I also decided to update phpMyAdmin on the Silicon node, as it was outdated and some exploits were found in this version. I also decided to write a script to, in the future, automate this process somewhat to save time.

The first test run failed due to some little bugs in the script, but after working those out I accidentally ran the script without arguments (First argument should be the root of the phpMyAdmin folder), so then this happened.

Luckily the thing that did work was the manual back-up images I make periodically, so after a day of recovering files, restoring SELinux contexts and whatnot it lived \o/.

After this I patched up the script so it would check if the folder indeed looks like a phpMyAdmin folder, and also exit the script if it doesn’t get any arguments.

So yeah, just another day at The Clusterfuck HQ

~Xeross

As I was setting up my Icinga install today I needed to modify the notification command so it had a specific from address, and it was using mailx.

So off I go onto google, etc. most of the answers said you could do something like:

mailx [args] example@domain.tld -- <sendmail options>

Turns out that that syntax doesn’t exist in the versions nowadays, luckily it turns out to be quite easy:

mailx [args] -S from="Name <example@domain.tld>"

Of course you can also only specify an email for the from parameter.

Anyway, hope this was helpful.

~Xeross

So I’ve just tested running a Varnish caching server in front of the NGINX server, to see if this would speed up the website. The tests I did were done on 127.0.0.1 to exclude any network latency. Let’s just say page loads should be significantly faster when I completely set-up varnish:

Without Varnish

Concurrent Connections: 10, Num. Requests: 1000

Percentage of the requests served within a certain time (ms)
  50%   1870
  66%   1975
  75%   2083
  80%   2141
  90%   2237
  95%   2299
  98%   2730
  99%   3071
 100%   7214 (longest request)

Concurrent Connections: 2, Num. Requests: 200

Percentage of the requests served within a certain time (ms)
  50%    558
  66%    566
  75%    571
  80%    574
  90%    586
  95%    627
  98%    800
  99%    869
 100%    926 (longest request)

With Varnish

Concurrent Connections: 10, Num. Requests: 1000

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      2
  80%      2
  90%      2
  95%      3
  98%      4
  99%      5
 100%      9 (longest request)

Concurrent Connections: 2, Num. Requests: 200

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      1
  95%      3
  98%      3
  99%      4
 100%      4 (longest request)

All tests were done using ab (Apache Benchmark)

So I found this nice deal on a 2nd hand server, it had 2 Single-core Xeon processors both clocked at 3.8ghz and having HT, and it had 12TB of RAM and a 300GB HDD, and that for just 320EUR.

Unforunately the place I was getting it from had to pick them up at a business that was getting rid of them, but apparently it already sold before they got it.

So yeah no new server, sad panda nao :c

So yeah, I’ve ran across this StackOverflow entry yesterday and decided to adopt one of those naming schemes, well multiple.

So now all my local servers will use the chemical elements as naming conventions where the last number of the IP equals the atom number.

External servers will start on the other end of the chemical element list, and non-server computers will be named with the NATO Phonetic Alphabet.

Now to actually set up all these servers…

Xeross out.

So continuing with configuring the new Fedora server setup I was working on I decided I might as well modify the NGINX package a bit, it now uses www-data as its account, uses /var/www/ for the webroot and has some reorganization in the configs that I always did manually, simply awesome.

Next to that I discovered PHP will always install the Apache HTTPd, even when a different HTTPd is present, so I just decided to get rid of the HTTPd requirement completely.

I’ve already set up a new node to work as my repository server and it’s happily running the new NGINX package, now waiting for the PHP to finish compiling, and then I can upload it and generate the repo metadata.

After having my trusty server running for 4 years (+5 years before that as a desktop) it became clear that it would probably die soon (Well some part of it). the graphics card was making noises (it doesn’t have an on-board graphics card) which turned out to be the fan that was vibrating heavily, and I didn’t like the idea of running it without a fan.

I coincidentally had acquired enough spare parts to create a new computer with so I decided to use those to build a new server. After disassembling and assembling I reinstalled Debian Linux 5.0 and started restoring my back-ups, this time I’m trying to keep it as organized as possible.

So this is basically what I’ll be finishing the coming days besides playing EVE. Oh yes and E-ON magazine arrived today, read about 1/3rd of it so far (Great job on the article Mynxee), my new USB stick also arrived, a 16GB Sony Microvault Excellence, it’s 2 times faster then my current 2GB Titanium Cruzer (Will still use them both) and I’ll primarily run Linux and EVE from it (Not EVE inside Wine, just EVE when inside a windows environment).

Well that wraps it about up I think, thanks for reading my huge wall of text and see you at the next post. or leave a comment and we can discuss or talk about something related or unrelated.

~Xeross

Note to self: Don’t write posts when sleepy, they turn into huge walls of  text.