Category Archives: System Administration

Linux system administration.

DBI reading MySQL connection details from .my.cnf

Useful trick: I often have my MySQL account credentials stored in .my.cnf so the mysql command-line client can use them. I also often have Perl scripts which want to connect to the database, and want them to use that file, not have to put the params into the script or have the script read its own config file with the credentials duplicated there.

The answer:

my $dsn = "DBI:mysql:database_name;mysql_read_default_file=$ENV{HOME}/.my.cnf";
my $dbh = DBI->connect($dsn,undef,undef,{RaiseError => 1}) 
    or die "Failed to connect to DB!";

Easy!

Extract part of a Subversion repo into a Git repo

A few times I’ve wanted to extract part of a large monolithic Subversion repository out into a seperate Git repo, but maintain the commit history.

Here’s how I do it.

First, I set up a mapping of Subversion user => Username in a file, so that the committer can match up easily via GitHub etc – each committer should have an entry like the below, one per line:


davidp = David Precious

Now, I clone the entire Subversion repo via git svn into a new git repository:


# Clone the Subversion repo into a new Git repo:
# (~/subversion_authors.txt is file mentioned above)
git svn clone file:///shared/svn/scripts --no-metadata -A ~/subversion_authors.txt tmp/scripts-repo-tmp

Some tags get added during this process, I believe; I don’t need/want to preserve them, so I remove any and all tags:


# remove tags - we don't need them
git tag -l | xargs git tag -d

Now, the clever part; using git filter-branch to select the path within the repo I want to preserve, and remove everything else, promoting the desired path to the “root” of the repository:


# remove all except a given path:
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter path/to/desired/dir HEAD

In the above, path/to/desired/dir is the path within the repo that I want to move to the root of the repo; everything else will be discarded.

At this point, I can add a GitHub repository via git remove add origin $url, and push the new repository.

I *think*, because I pushed to GitHub, then deleted my temporary repo and cloned back down, that unrelated previous commits were automatically removed. In case that’s not true, though, the following ought to purge unrelated commits from the new Git repo:


git reset --hard
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --aggressive --prune=now

VLC getting proxy settings via gconf

I had a problem with VLC, using outdated proxy settings rather than connecting directly, even with no proxy configured in VLC’s settings (and even when trying to override it using command-line options).

(It would show that it was trying to use a proxy:

[0x134b4b0] main access error: connection failed: Connection refused
[0x134b4b0] access_http access error: cannot connect to supernova:3140

(supernova:3140 was the proxy setting it was picking up.)

After using strace to follow the execution of VLC when trying to play a network stream, I realised it was getting the settings via gconf; I needed to execute gconf-editor, navigate to system, http_proxy, then untick use_http_proxy – and also edit the value of the host setting to an empty string (without doing this, VLC ignored the use_http_proxy setting being false, and tried to use the proxy anyway!).

I could have done the same using gconftool-2 with:

gconftool-2 -s /system/http_proxy/use_http_proxy --type bool false
gconftool-2 -s /system/http_proxy/host --type string ''

Blogging for my own future reference, and for anyone else who’s using vlc on XFCE and wondering where it was getting the proxy settings. I’ve used Gnome 2 on this machine before, and I think that system proxy setting was set by Chromium (Google Chrome).

SMART monitoring drives on 3ware cards

I’ve been updating my Nagios monitoring to make sure I’m carefully monitoring my server hardware, including temperature, fan speeds and hard drives, and I wanted to use S.M.A.R.T. monitoring to monitor drives on a 3ware RAID controller for signs of imminent failure.

I already monitor the status of the RAID array itself using my nagios_3ware_raid_check Nagios plugin (which I previously blogged about), but I wanted to use SMART monitoring to look for signs of imminent drive trouble, rather than simply finding out when a drive has just failed.

After installing smartmontools, I was able to edit /etc/smartd.conf, disabling the default of scanning for devices, and listing devices explicitly, as follows:


# Monitor the drives on our RAID array; schedule self tests for Sundays.
/dev/twa0 -d 3ware,0 -a -s L/../../7/02
/dev/twa0 -d 3ware,1 -a -s L/../../7/04

The above monitors both drives of a RAID-1 mirrored pair on a 3ware controller card; the -s option schedules a long self-test every Sunday, starting between 2-3am and 4-5am respectively.

I’m still looking for a good way to monitor via Nagios, though; the (poorly-named) check_ide_smart plugin doesn’t support the ability to monitor drives on other interfaces as far as I can see. I found a couple of Perl scripts such as Check-SMART-status-modified, but they had issues.

Increasing loop devices on Xen host

Trying to start another Xen guest domain, and getting an error indicating that there are no loop devices left, like the following? :


Error: Device 51714 (vbd) could not be connected. Failed to find an unused loop device

Increase the number of loop devices that the loop kernel module will create, by editing/creating /etc/modprobe.d/local-loop containing something like:

Continue reading Increasing loop devices on Xen host

Ubuntu 11.04 – X freezing after screensaver active / laptop lid closed

I recently upgraded my laptop (a Dell 17R / N7010) to Ubuntu 11.04 (Natty), and since then, it has been randomly freezing when I come back to it after leaving it for a while.

I see my X session exactly as I left it, and can move the mouse around, but nothing seems to respond to mouse/keyboard events. The system is still alive, I can SSH to it from another box, or switch to a different virtual terminal (e.g. Ctrl + Alt + F1), and I can kill Xorg (sudo pkill X) at which point gdm immediately gets restarted, and I can log back in.

It seems to only strike when the laptop has been left unattended for some time. I can close the lid, wait ten seconds and re-open it, and everything is fine. I can also trigger the screensaver by locking the screen with Ctrl + Alt + L, then unlock, and all is also fine.

I’ve done some Googling, and I’m certainly not the only person experiencing this:

Bug 772925 – Screen freezes after laptop lid closed for more than 1 minute
Bug 781402 – Ubuntu 11.04 / Unity freezing when screen is ‘woken up’
a discussion of bug 781402 on fossplanet.com
Bug 740126 – Disabling an output can cause vblank events to be missed

I’m using ‘Ubuntu Classic’, i.e. using GNOME rather than Unity, but the reports of the problem I’ve seen are split between both.

I’ve found a suggestion of a possible fix – installing compizconfig-settings-manager then disabling Sync to vblankon ubuntuforums.org.

I’ve just tried that out; I’ll update this post when I know whether it has worked or not.

EDIT: I’ve checked logs for anything illuminating and found little, but I did just spot the following in /var/log/Xorg.0.log.old:


[437515.524] (EE) intel(0): [DRI2] DRI2SwapComplete: bad drawable

I’ve no idea if it’s related to the problem or not.

EDIT 2: So far, it would seem that disabling “Sync to vblank” as mentioned above has fixed the problem, for me. (It’s also possible that it was fixed in one of the updates since, but I haven’t spotted any package updates that looked relevant.)

If you’re having the same problem, feel free to comment here on whether the “Sync to vblank” change stopped the freezes for you.

Incremental backups with rdiff-backup

My current backup solution is using rdiff-backup to do incremental backups. I’d previously been using plain rsync, but I wanted snapshots too, so I could retrieve a file as it looked at some point in the past (as Sod’s law guarantees that if a file was mistakenly deleted/clobbered, rsync will run between that happening and you noticing, so the backup will be clobbered too).

My setup involves the backup box running rdiff-backup, connecting to the machine to be backed up via SSH, using a passwordless SSH key for authentication. The entry in ~/.ssh/authorized_keys on the machine to be backed up allows that key to be used only to run rdiff-backup, nothing else, and only from the backup host, to provide as much security as possible.

I thought I’d document my setup here, both for easy future reference for myself when adding other boxes to back up, and for anyone else who may find it useful.

So, the steps I use are as follows:
Continue reading Incremental backups with rdiff-backup

Nagios plugin to monitor 3ware/LSI RAID

I wrote a basic Nagios plugin named nagios_3ware_raid_check to monitor the status of RAID arrays on 3ware/LSI hardware RAID controllers, but it was pretty limited.

Today I got a chance to improve it as I needed it for some machines at work; it now automatically figures out which card(s) are present, and checks the status of each RAID array on it. This means you don’t need to configure it to know which card number and unit to monitor, it just Does The Right Thing.

It uses the tw_cli utility available from the LSI website, which supports a wide range of 3ware/LSI RAID cards.

Example:

[dave@rasputin:~]$ /usr/local/bin/nagios_3ware_raid_check -v
Card 6 is a 8006-2LP
Unit u0 on card6 is a RAID-1 array of 931 GB and is OK
RAIDCHECK OK - Card 6 unit u0 RAID-1 OK (931GB)

When actually using it from Nagios, don’t use the -v (verbose) option – just use something like the following in nrpe_local.cfg:

# Monitor RAID status
command[check_raid]=/usr/local/bin/nagios_3ware_raid_check