Category Archives: Geeky

Technical stuff about Perl, Linux and computing & technology in general.

Graphing time-based data in Perl

I recently wanted to produce some graphs from a web app powered by the Dancer Perl web framework, and reevaluated the various Perl graphing moduiles out there.

Modules I considered were:

  • Chart::Strip
  • Chart::Graph
  • Google::Chart
  • Chart::Clicker
  • Chart::Gnuplot
  • Unfortunately, I didn’t have time to do a full in-depth writeup trying every module like the excellent ones Neil Bowers has been doing, but I thought I’d write up a quick post on the choice I made, with example code, in case it helps other people looking to graph potentially irregularly-spaced time-based data samples in Perl easily.

    Chart::Clicker looked to be a nice choice (with a nice example of doing just what I want given as the topic answer to a question on StackOverflow), but had a huge chain of dependencies, finally failing when demanding Cairo and various X11 libraries (on my headless server).

    Chart::Strip seemed to do exactly what I wanted in a simple way, but I encountered a div-by-zero bug when dealing with a certain dataset with > 89 data points.

    I reported this to the author, Jeff Weisgberg in RT #72288, and he promptly released 1.08 with a fix (thanks Jeff!).

    Chart::Strip made it simple to do what I wanted:

    my @dataset;
    while (my $row = $sth->fetchrow_hashref) {
        push @dataset, { time => $row->{timestamp}, value => $row->{value} };
    }
    
    my $chart = Chart::Strip->new( title => "My chart" );
    $chart->add_data(\@dataset, { style => 'line' });
    
    # then get the chart as an image with $chart->png
    

    Nice and easy, just what I wanted – a way to say “here’s some timestamps and values (quite possibly irregularly spaced) – work out how to plot this sensibly for me”.

    The resulting graphs look good enough to me, e.g.:

    (Rendered intentionally a little smaller to fit the blog; naturally the graphs can be whatever size you want. Also, I had to use the transparent option to disable transparent backgrounds.)

Why IRC is a valuable tool to your development team

IRC (Internet Relay Chat) is a protocol for multi-server text chat between many participants in many channels, started back in 1988.

There are plenty of IRC networks out there for social chatter, including the likes of Freenode and irc.perl.org hosting many channels for Perl and open source channels in general, making it easy to get quick help from developers and users of your favourite project.

However, I find IRC to be a very valuable tool indeed to help development teams collaborate effectively; at work we make extensive use of it. It’s useful whether you’re a formal development team in a corporate environment, or an open source project whose developers / collaborators gather on IRC.

Why is it so useful? Well:

It enables quick discussion and collaboration without breaking your workflow

As a developer, you don’t want to lose your concentration – when you’re “in the zone”, you’re carrying information about the code you’re working on in your brain, and it doesn’t stay there for long if you’re distracted. Someone walking over to you and starting talking to you, or a phone call, demand more or less 100% attention; you will be distracted, and you will “fall out of the zone”, causing your productivity to fall until you get back to where you were.

IRC, on the other hand, means you don’t have to respond quite so immediately, and I find it easy to flick between coding and IRC (both terminals within Terminator for me) without losing focus on where I’m up to and what I’m doing.

Most IRC clients support alerting you when your nick is mentioned in a channel or you receive a direct message, so you can ignore general chatter in the channel until you’re ready to read it, but know if someone is trying to get your attention.

Of course, it’s even more valuable when your development team work from multiple locations, whether that’s having employees working from home, or multiple offices.

Logs of discussions can be valuable for future reference

If you keep logs of your discussions, it’s easy to refer back to later – sometimes you’ll remember “ah, yes, we talked about this – what was the outcome?” – quick log search, and your answer is there. “Why did we decide that this was the best way to implement this?” – log search – “ah, that was why”.

Open, widely-supported protocol

IRC is an open, widely supported protocol; there’s various clients available for pretty much every platform, so whatever system your devs work on, they’ll be able to find a client that suits them.

Easily extensible to integrate with other tools

It’s easy to write IRC “bots” which can help integrate with various other tools in various ways.

A good example is providing easy links to commits / bug reports or issues / pull requests etc.

If you’re using my Bot::BasicBot::Pluggable::Module::GitHub for instance, you can mention an issue and have the bot automatically provide a summary and an URL for anyone who wants to see what the issue in question is – e.g.:

<user1> Anyone had a moment to look at Issue 42 and see what's going on?
<bot> Issue 42 (It doesn't work) https://github.com/....
<user2> Oh yeah - I fixed that in 5fcbb01
<bot> Commit 5fcbb01 (Retarded logic fail.) - https://github.com/....

It’s easy to cobble together a simple bot or bot module to do this kind of stuff for whatever your in-house situation requires, if there’s nothing suitable already out there on CPAN (which, a lot of the time, there already will be).

GitHub provide post-receive hooks which can be configured to announce pushes to your IRC channel(s). Bot::BasicBot::Pluggable::GitHub::Announce can automatically announce new/updated issues, and, in future releases, also pull requests and commits/pushes.

System problems reported instantly

Use something like my Bot::BasicBot::Pluggable::Module::Nagios, and you can have system problems reported automatically to the appropriate IRC channels, for quick attention by whoever needs to deal with them. I use an applet in my GNOME system tray which alerts me to problems, but seeing them reported in detail on IRC is handy, and also strikes up conversation about it – a simple “I’m on it ^^” is enough to let others know you’re dealing with the issue and they don’t need to worry about it.

Announce tweets about your company/brand/project/interests

My Bot::BasicBot::Pluggable::Module::TwitterWatch module allows you to have the bot watch for and report new posts on Twitter about your company/project/brand/stuff of interest, and post them to your IRC channel – either for awareness, or to strike up discussion about them.

Wrap various other tools

Your IRC bot(s) can provide various other useful facilities – for instance, find the corelist command useful? Bot::BasicBot::Pluggable::Module::CoreList makes it easy for your bot to answer corelist lookups within the flow of a conversation.

<user1> Could use File::Spec - that's part of core, isn't it? 
<user2> bot: corelist File::Spec
<bot> File::Spec was first released with perl 5.00503 (released on 1999-03-28)
<user2> Yep :)

Bot::BasicBot::Pluggable::Module::Nagios released

I’ve just released Bot::BasicBot::Pluggable::Module::Nagios – a module for IRC bots powered by Bot::BasicBot::Pluggable which monitors one or more Nagios instances and reports problems to IRC channels.

I’ll be using this at work to have service problems reported to us on IRC for quick attention, but figured it’s something that’s likely to be of use to others elsewhere, too, so I’m releasing it.

There’s still some more features and improvements I want to make (a TODO list is included in the module POD), but it’s at a state where I consider it to be usable (it works for me).

Feedback/suggestions welcome.

It should be on a CPAN mirror near you soon, and the repo is on GitHub should you wish to submit pull requests or raise issues for bug reports/feature requests.

Bot::BasicBot::Pluggable::Module::GitHub released

I’ve released a new distribution for Bot::BasicBot::Pluggable powered IRC bots, providing some useful GitHub-related functionality, named Bot::BasicBot::Pluggable::Module::GitHub.

The following modules are included – see the documentation for each for more details on how to use them.

Bot::BasicBot::Pluggable::Module::GitHub::EasyLinks

Provides quick URLs to view issues/pull requests etc when someone mentions one – for example:


<user> Go have a look at Issue 42
<bot1> Issue 42 (It doesn't work) https://github.com/....
<user> I fixed that in 5fcbb01
<bot1> Commit 5fcbb01 (Retarded logic fail.) - https://github.com/....

You can set a default project per-channel, so the above examples will look at whatever project is set as default for the channel the message was in.

You can also explicitly tell it to look at any other GitHub project:


<user> 5fcbb01 @ bigpresh/Bot-BasicBot-Pluggable-Module-GitHub
<bot1> Commit 5fcbb01 (Retarded logic fail.) - https://github.com/...

Bot::BasicBot::Pluggable::Module::GitHub::PullRequests

Monitor pull requests for GitHub projects.

Example:


<@bigpresh> !pr
< sophie> Open pull requests for sukria/Dancer : 8 pull requests open (felixdo:3, perlpilot:1, jamhed:1, dams:1, ambs:1, JTimothyKing:1)

Bot::BasicBot::Pluggable::Module::GitHub::Announce

Announces issues raised/closed for each channel’s default project.

Periodically checks on issues for each project, and reports changes,
for example:


< sophie> Issues closed : 667 (YAML::XS for Session::YAML) by jamhed : https://github.com/sukria/Dancer/issues/667

The code is on CPAN, and is available on GitHub. Contributions / bug reports / suggestions encouraged.

Using p3rl.org for FF search keyword

The lazy side of me enjoys Quick Search bookmarks in Firefox, so that I can type e.g. cpan ModuleName to look up a module on CPAN etc, pd perlfoo to look up the imaginary perlfoo documentation on perldoc, etc.

I also like p3rl.org‘s intelligent handling of short URLs, e.g. http://p3rl.org/Dancer, http://p3rl.org/perlrun etc.

It suddenly occurred to me to create a keyword search with the keyword p3, and the URL as http://p3rl.org/%s – now I can jump straight to pages I want with e.g. p3 Dancer, p3 perlrun etc.

My laziness likes this very much indeed.

Happy birthday Dancer!

Today marks two years to the day since the first version of Dancer hit CPAN!

According to the BackPAN, Dancer-0.9003.tar.gz hit CPAN on 07-Aug-2009.

I think you’ll agree we’ve come a long way since then, thanks to the awesome community and user base built up around the project since then.

In these two years, we’ve had countless valuable contributions from a large list of contributing users (see the list on the about page), gathered over 300 watchers on GitHub, had 84 people fork the repository on GitHub, had 620 pull requests submitted… amazing stuff.

Continue reading Happy birthday Dancer!

Renaming MP3s based on ID3 tags with Perl

As mentioned in my previous post on retagging MP3s by filename, I have a fairly large music collection, and prefer to keep it well organised.

I like the filenames used to always follow the same pattern, so I wrote mp3-rename, a Perl script to rename them based on the filename.

It takes a directory name as an argument (if not provided, it will operate upon the current directory). It will find all the MP3 files within that directory, extract information from the ID3 tags, then display a table showing the renames it will make if you tell it to go ahead, then ask for confirmation.

An example in action:


[davidp@supernova:~]$ mp3-rename /[...]/Rob\ Costlow\ -\ Woods\ of\ Chaos/
The following renames will be performed:
01 - Meant to Be.mp3    -> 00 - Meant to Be - Rob Costlow.mp3                                                           
02 - Reflections.mp3    -> 00 - Reflections - Rob Costlow.mp3                                                           
03 - Semester Days.mp3  -> 00 - Semester Days - Rob Costlow.mp3                                                         
[...]

Review the above renames to check the info is correct.

Type 'yes' to go ahead: ?>

(Some lines omitted for brevity)

It uses CPAN modules to do most of the work, including File::Find::Rule to find matching files to work on, Music::Tag to handle the tag processing, and Term::ReadKey to find out the terminal size.

Retagging MP3s by filename with Perl

I have a fairly large music collection in MP3s, and I like my files to be sensibly named and tagged.

One of the tools I use to do that is a Perl script I wrote named retag-by-filename, which allows you to provide a regular expression with named captures (so requires Perl 5.10 or later), and uses that to retag a bunch of MP3 files.

An example of it in use:


[davidp@supernova:~]$ ls -1 /shared/music/Complete\ Albums/Oasis\ -\ The\ Masterplan/ | head -3
01 - Acquiesce - Oasis.mp3
02 - Underneath The Sky - Oasis.mp3
03 - Talk Tonight - Oasis.mp3

[davidp@supernova:~]$ ./retag-by-filename --dry-run \
  --pattern="(?<track> \d+) \s - \s (?<title> .+ ) \s - \s (?<artist> .+ ) \.mp3" \
  /shared/music/Complete\ Albums/Oasis\ -\ The\ Masterplan/*.mp3
[01] Acquiesce by Oasis ()
[02] Underneath The Sky by Oasis ()
[03] Talk Tonight by Oasis ()

The --dry-run option shows the details of each track, but doesn’t actually update the tags. The --pattern option supplies the regular expression to match against filenames; you’ll use named captures named track, title, artist and comment to capture the appropriate parts. (In the example above, there is no comment to match.)

Thanks to the awesome power afforded by using modules from CPAN (Music::Tag and Getopt::Lucid), the actual script is 37 lines of code.