Category Archives: Geeky

Technical stuff about Perl, Linux and computing & technology in general.

Catalyst auto HTML/XSS scrubbing

At work, we needed to implement some HTML scrubbing and XSS protection across a Perl Catalyst-powered API, so went looking for existing solutions. We found Catalyst::Plugin::HTML::Scrubber which did some of what we needed, but did not scrub within encoded PUT/POST bodies e.g. POSTed JSON.

I implemented some improvements to provide this, but sadly the original author could not be reached – it seems he hasn’t been active in the Perl community for quite some time. With a little help from the CPAN admins (thanks!) I obtained maintainership of it, and have since got a couple of releases out which add the features we needed:

  • scrubbing HTML/XSS attempts within both normal parameters (querystring / POSTed forms) and also recursively within PUTted/POSTed JSON etc
  • the ability to whitelist certain parameters by name or regex to exclude them from scrubbing – we have some admin-only areas where staff can enter “message of the day” content which is allowed to contain HTML
  • a “no encode HTML entities” option to undo HTML::Scrubber‘s automatic HTML entity encoding of e.g. angle brackets – whilst content destined for the browser wants to be HTML-encoded, inbound parameters don’t want that, we just want HTML /XSS attempts stripped, a parameter value like >= 5 should be left alone

The amended version can be found on CPAN – Catalyst::Plugin::HTML::Scrubber.


(Aside: yes, it has been, er, quite some time since I posted anything on this blog.)

DBI reading MySQL connection details from .my.cnf

Useful trick: I often have my MySQL account credentials stored in .my.cnf so the mysql command-line client can use them. I also often have Perl scripts which want to connect to the database, and want them to use that file, not have to put the params into the script or have the script read its own config file with the credentials duplicated there.

The answer:

my $dsn = "DBI:mysql:database_name;mysql_read_default_file=$ENV{HOME}/.my.cnf";
my $dbh = DBI->connect($dsn,undef,undef,{RaiseError => 1}) 
    or die "Failed to connect to DB!";

Easy!

Using SSL client certs with Perl’s LWP::UserAgent

I recently needed to authenticate to a remote API using an SSL client certificate, and had a bit of trouble getting LWP::UserAgent to work with it.

The examples I found which looked like they should work involved e.g.:

use LWP::UserAgent;

my $ua = LWP::UserAgent->new(
    ssl_opts => {
        SSL_use_cert => 1,
        SSL_cert_file   => "/path/to/clientcert.crt",
        SSL_key_file    => "/path/to/privatekey.key",
    },
);

However, that didn’t work; changing the paths to the cert/key to non-existent files didn’t cause any difference, so I suspected that those options were actually being ignored.

After a fair bit of digging, the option I found that actually worked was loading Net::SSL first, to make LWP use Net::SSLeay, and setting env vars to the client cert to use:

use Net::SSL;
use LWP::UserAgent;

$ENV{HTTPS_CERT_FILE} = "/path/to/clientcert.crt";
$ENV{HTTPS_KEY_FILE}  = "/path/to/privatekey.key";
my $ua = LWP::UserAgent->new();

This, to me, is pretty icky – I’d much rather pass config to affect just that single LWP object. However, it gets it working.

Testing Samsung auto-dial shortcode exploit on Galaxy Note

It’s been widely reported today that the Samsung Galaxy S3 and other Samsung Galaxy phones have a glaring vulnerability: a webpage can trigger the Samsung dialer to dial a code which wipes the phone.

Example reports:

Gaping Hole in TouchWiz UI is Wiping Samsung Androids Clean (dailytech.com)


Samsung Galaxy S3 can be wiped and hard-reset with a single line of HTML (reddit.com)

Security Bug Can Wipe Out Your Android Phone By Visiting a Web Page (gizmodo.com)

I wanted to see if this was really true, so I thought I’d knock up a proof of concept using a much safer short-code, *#*#4636#*#*, which, if entered in the dialler, will take you to a testing / control menu where you can change various device settings (it’s useful to know about that hidden trick – but don’t change things unless you know what you’re doing).

So, I created a testmenu.html containing a frame which attempts to load tel:*#*#4636#*#*; it does indeed call up the dialler, but does not appear to actually trigger the test menu. (This debug code doesn’t require you to push send to confirm it – when you type the last “#”, the menu pops up – this is one reason I chose it as a test, as I assume that the problem with the wipe code (which I’m not testing out on my phone!) is that it works the same way (that, or people push to dial it, not knowing what will happen.)

Next, I decided to try a code that does need you to push send – *#100#. which typically returns your phone number. I created ownnumber.html. Loading that page in my Galaxy Note’s stock browser does indeed launch the dialler again – this time, though, the code *#100# is displayed, ready to be “dialled” if the user desires. This is the behaviour I’d expect from anything that links to tel:$number – the user to be asked for confirmation before placing a “call”.

Is it just that the Galaxy Note doesn’t have the same problem, or is there something special about the “wipe device” code?

Is, perhaps, the test menu code “executing”, but invisibly?

I am somewhat disturbed that Samsung modified the stock browser to support tel: URLs; I could see them being mildly useful for actual links to click to bring up the dialler, but using them as the source for a frame / image would never make sense.

(For what it’s worth, this is a Samsung Galaxy Note (GT-N7000) running Android 4.0.4, on UK Vodafone.)

Extract part of a Subversion repo into a Git repo

A few times I’ve wanted to extract part of a large monolithic Subversion repository out into a seperate Git repo, but maintain the commit history.

Here’s how I do it.

First, I set up a mapping of Subversion user => Username in a file, so that the committer can match up easily via GitHub etc – each committer should have an entry like the below, one per line:


davidp = David Precious

Now, I clone the entire Subversion repo via git svn into a new git repository:


# Clone the Subversion repo into a new Git repo:
# (~/subversion_authors.txt is file mentioned above)
git svn clone file:///shared/svn/scripts --no-metadata -A ~/subversion_authors.txt tmp/scripts-repo-tmp

Some tags get added during this process, I believe; I don’t need/want to preserve them, so I remove any and all tags:


# remove tags - we don't need them
git tag -l | xargs git tag -d

Now, the clever part; using git filter-branch to select the path within the repo I want to preserve, and remove everything else, promoting the desired path to the “root” of the repository:


# remove all except a given path:
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter path/to/desired/dir HEAD

In the above, path/to/desired/dir is the path within the repo that I want to move to the root of the repo; everything else will be discarded.

At this point, I can add a GitHub repository via git remove add origin $url, and push the new repository.

I *think*, because I pushed to GitHub, then deleted my temporary repo and cloned back down, that unrelated previous commits were automatically removed. In case that’s not true, though, the following ought to purge unrelated commits from the new Git repo:


git reset --hard
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --aggressive --prune=now

Dancer talk at YAPC::NA 2012

Mark Allen will give a talk at YAPC::NA 2012 on the Dancer Perl web framework he describes as:

This talk presents the Dancer web framework beginning with “Hello World” and progressing through a couple of easy to digest introductory applications. All of the primary Dancer features are presented including URL routing, writing handlers, and output templating. A selection of useful and common Dancer plugins will also be covered. This talk is best suited for beginning and intermediate Perl programmers.

(via JT Smith, in turn via the YAPC::NA blog.)

I hope it’s recorded, as I’d like to see it, but won’t be able to afford to attend YAPC::NA.

VLC getting proxy settings via gconf

I had a problem with VLC, using outdated proxy settings rather than connecting directly, even with no proxy configured in VLC’s settings (and even when trying to override it using command-line options).

(It would show that it was trying to use a proxy:

[0x134b4b0] main access error: connection failed: Connection refused
[0x134b4b0] access_http access error: cannot connect to supernova:3140

(supernova:3140 was the proxy setting it was picking up.)

After using strace to follow the execution of VLC when trying to play a network stream, I realised it was getting the settings via gconf; I needed to execute gconf-editor, navigate to system, http_proxy, then untick use_http_proxy – and also edit the value of the host setting to an empty string (without doing this, VLC ignored the use_http_proxy setting being false, and tried to use the proxy anyway!).

I could have done the same using gconftool-2 with:

gconftool-2 -s /system/http_proxy/use_http_proxy --type bool false
gconftool-2 -s /system/http_proxy/host --type string ''

Blogging for my own future reference, and for anyone else who’s using vlc on XFCE and wondering where it was getting the proxy settings. I’ve used Gnome 2 on this machine before, and I think that system proxy setting was set by Chromium (Google Chrome).

Perl Advent Calendars for 2011

Well, December is upon us – time for advent calendars, and as usual, the Perl community doesn’t disappoint – here’s a list of the Perl-related advent calendars I’m aware of:

There are also several Japanese-language advent calendars:

If you know of any others, please feel free to let me know and I’ll add them to the list :)

LPW2011 : my thoughts overall

Yesterday, I attended the 2011 London Perl Workshop – my first ever Perl conference.

I had a good day, met a few members of the Perl community I knew from online interactions who I’d never met in person before, saw some good talks, and partook in some free food and beer (kindly paid for by the sponsors, including my employer, UK2).

Some brief mentions of talks I attended:

Matt S Trout (mst) – First, Tak wrote the world‎

I have IO::Pipeline. I have App::FatPacker. I have IPC::Command::Multiplex. And yet I still couldn’t whip up a five line example of bolting them all together that made a compelling argument for a perl-loving sysadmin to stop using fabric.

This problem, among others, will be solved by the conclusion of this talk.

Tak sounds like something which will be very useful to me – running code on multiple other hosts via SSH, but including Perl code – with all locally-installed modules available for use at the remote end!

As mst went through explaining how it all worked, my thoughts went from “hmm, useful”, to “hmm, useful but looks over-engineered, not sure it needs to be that complex” to “whoah, that’s genius”. Fatpacking and sending code to the remote side, which then adds a coderef to @INC which requests other modules from the local end, sent over and loaded remotely, is awesomely creative.

These kind of tricks remind me of why I love Perl.

Mike Whitaker (‎Penfold‎) – ‎Perl and Unicode, the 5.14 edition‎

A very good talk on handling Unicode safely in Perl, and the gotchas to avoid. Provided major impetus for me to upgrade to 5.14, too.

Zefram – ‎why time is difficult‎

Dates, times, time intervals, clocks, calendars, and related phenomena are major contributors to hassle in programming, and the source of innumerable bugs.

Zefram’s talk, whilst barely Perl related, was very interesting, and very well delivered. I hadn’t realised quite how complex time was :)

Zefram’s amusing lightning talk on doing away with source code by simply storing bytecode and editing it by deparsing the source, editing it, then “compiling” back to bytecode was also entertaining.

Claes Jakobsson (‎claes‎) – ‎Don’t debug now, debug later

Runops::Recorder is a alternate runloop for perl that writes down what your program does to disk for playback later

It also comes with a viewer and some helper classes for you to write your own playback tools such as diffs etc.

This looks like a very useful debugging tool, recording the path of execution through your code and writing it to a file which can then later be “replayed” using a viewer – much like single-stepping or tracing through the debugger, but after the fact. The ability to leave it running and have it dump out a configurable amount of trace data when a die is encountered looks excellently useful for catching intermittent / rare problems – you should be able to leave it in place, wait until the problem occurs, then replay what happened leading up to the die to see what was going on.

Future versions should also be able to track changes to variables, etc, which will be very useful indeed.

There were a couple of workshops I’d like to have attended, but which I didn’t; partly because they conflicted with talks I wanted to see, and partly because I didn’t have a laptop with me to “work along” and didn’t think I could take much of value away from them.

Andrew Solomon – ‎[[TRAINING SESSION]] Web development for beginners using Dancer‎

As a core developer for the Dancer perl web framework I’d love to have attended Andrew Solomon’s workshop, to see what was being taught, and offer any input desired. Unfortunately, I wasn’t there, but I’ll be looking with interest for any feedback from people who were, and what they learned and what they thought of Dancer if they hadn’t encountered it before. Making Perl accessible for new users is an important thing.

Gabor Szabo (‎szabgab‎) – ‎[[TRAINING SESSION]] Testing in Perl

I’d also like to have taken part in Gabor’s workshops, but they were in two parts and conflicted with several other talks I wanted to see.

I met a few members of the Perl community who I knew from online interactions but had never met in meatspace, so it was great to meet them. Unfortunately, there were a few others I’d meant to go introduce myself to, but never got a chance to do so – including Tatsuhiko Miyagawa and Gabor Szgabo.

Overall, it was a good day, and I imagine there’s a very good chance I’ll be back next year :)

LPW2011 : abigail’s “Business Aware Developer” talk

I caught abigail’s “Business Aware Developer” talk yesterday at the London Perl Workshop 2011.

Overall, I think it was a good talk, and raised some good points, even if the “you don’t always have to write tests, write them only if they provide value” is a little controversial with some of the audience, leading to a reasonable amount of debate and running late with the talk so having to skip some slides.

Personally, I agree to some degree – I think some people write tests simply to push up their test coverage figure, without really writing tests which are likely to catch bugs (exercising the code in both expected and unexpected ways, providing strange input and edge cases (does it blow up if given undef or a ref, say).

However, I do think a fair amount of the talk is summed up by advice given to me by a boss at work, Ditlev, with regards to getting stuff out – sometimes you have to “launch crap but launch” – sometimes code that works well enough to be put into use and making money for you can be more valuable than taking longer to produce better quality code – which may be nicer and better to work with in the future, but isn’t ready to launch now. In other words, examining the trade-off between quick results now, and better quality code which becomes more valuable later – but “what if later never comes?”.

The impression I took away from the talk, which might be a misconception, is that Booking.com don’t do code reviews or refactoring, which would seriously put me off applying for a position there – I think code review in particular (even if just casual – it needn’t be a formal procedure) is very valuable to push yourself to be a better coder. If you know other members of the team are going to be glancing over your commits when they have time and pointing at bits you could have done better, that’s a good motivation to write good code, and also often helps you realise other ways you could have done things.

I’d be interested in seeing the other slides which abigail had to skip over, but I haven’t been able to find them anywhere online.