Help Machine

8th January 2020

For far too long now I've been planning on writing a coding book for kids. I first had the idea when my daughter was eight, and I had some bizarre utopian dream about teaching her to code in a delightful father/daughter partnership. She's now fourteen, hasn't come out of her bedroom or managed a polysyllable since 2018, so I think that ship has sailed.

Still, the idea of a coding book still lingers, and various computers and notebooks are full of good ideas, which have never got past the good idea phase.

For the past year I've been teaching an 'enrichment' class on game coding at a local secondary school using Python and a very simple game engine which wraps PyGame, making 2D sprites and collision detection lovely and easy. The games and coding exercises I'm using are those I wrote to go in the book. ('Enrichment' classes are non-assessed courses lasting around 8 weeks for an hour a week. They are sort of after-school clubs, but compulsory.) The students are Key Stage 3 (ages 11 - 13) and they all have at least some experience with Scratch.

I'm finding the step up from Scratch to Python to be very challenging for them. In particular the precision of the syntax is preventing them from getting the confidence to be able to explore and experiment. I've put the students in pairs to help each other, and given them exercises where they correct or complete code samples, instead of just typing in chunks of given code. Even so, most of my time is spent galloping around the lab correcting syntax errors, and I think in most cases only surface learning is taking place.

By the fifth of eight weeks it was clear that I had lost half a dozen of the class of fifteen, and they were getting bored, fractious and disruptive. In these cases I decided to allow them to return to Scratch and do their own thing, which they seemed happy to do. Clearly this was not ideal but it at least headed off impending behaviour problems.

However there is clearly something about Scratch which facilitates this unthreatening play behaviour that Python doesn't. If I was still a proper academic I would immediately design a load of experiments to test this behaviour, then write lots of papers about it, become lauded in the field and become a professor. (Or, more honestly, think about designing some experiments, get excited for a few weeks, never get round to doing them because of crazy teaching load, and never get anywhere, other than getting as bored, fractious and disruptive as my secondary schoolkids.)

I suspect the difference is that Scratch effectively does not have syntax errors; blocks can be assembled in anyway that the coder see fit, and can test the semantic consequences. With Python my students are struggling to get over the step of correct syntax and then into the ability to be able to explore and play.

If I get chance to work one on one (or in tiny groups) with the students then I can get them past this, but obviously this is unfeasible in bigger classes, and impossible if the exercises are going to be in a book.

So I've been thinking about how to gameify the writing of Python code so that getting the syntax right becomes part of exploration behaviour, not just tedious rules to be learnt. I've also thought that if the IDE 'knew' about the code that the students were supposed to be typing in then it could generate much more helpful and targeted advice, rather than just normal syntax errors.

Thonny is a Python IDE aimed at beginners and has the scope for third-party plug-ins. It also appears to be shipped as the default Python editor on Raspbian. So my proposal is for a Thonny plug-in which can be given a set of exercises with solutions and advice which is given interactively to students as they type code in. Clearly this is not at all simple, and I'm very aware that I might finish up reimplementing the Microsoft Office paperclip.

However I believe that there is a lot of scope here, and the step from Scratch to Python (or other 'professional' coding languages) is a serious issue that needs addressing.

As well as designing an interface that genuinely supports students learning to code, I also want to allow teachers to be able to design their own exercises, not just have the system tethered to my book (that I may or may not ever get around to writing). This exercise authoring system should be nice and flexible, starting of with an exercises where the teacher simply gives a piece of 'target' code which students need to type in or complete, to an exercise where much more supportive advise and description about what the code actually does is included.

The system will be called 'Help Machine', which fits in nicely with the other naming and branding around the book. More details will follow...

The media form field in Joomla 2.5

13th October 2014

Guaranteed to generate a bit of online ire on StackExchange is the Joomla media form field. It doesn't do what it says on the tin; it only allows you to select images from the media manager, not pdfs, audio or video files. That said it does say very clearly on the Joomla docs that it only allows the user to select images, not arbitrary media files. Maybe it should be called images form field or similar. It is however a perfectly reasonable to expect that there will be situations where the user will want to select any media file.

Anyway, you would think that it would be easy to define a custom form field to allow the user to select any media file by extending existing code. Unfortunately, it doesn't seem to be easy at all. In fact on a couple of past projects (mentioning no names) I've committed the cardinal Joomla sin of hacking the core files to stop com_media filtering out files that aren't image files. The list of acceptable file extensions is hard coded into one of the com_media model files. Well, if the Joomla developers are going to do that, I'm going to hack their core.

However, lately I find myself working on a project where the client has set up a Joomla installation on a cloud hosted server where all the installation is done for you, and you can't get at the files on the server other than to upload extensions. So, no core hacks.

So what follows is my solution to getting round this problem with plug-ins only. Firstly caveats; this solution overrides the behaviour of the images view of com_media. It doesn't create a new custom form field leaving the core media form field to behave as described in the Joomla docs. This means that all admin-side uses of the media form field will allow all media files (e.g. in the Images and Links box of the Article editor). If you trust your users not to select videos when they should be selecting images, then this solution suffices. If you want to create a custom field to select all media files and leave the core media form field as is, then that adds another whole layer of complexity, and -- in the words of all the best text-books written by lazy authors -- is left as an exercise for the reader. Second caveat; I make no claims on general applicability. It works for the project I'm currently working on (which is Joomla 2.5). The code is on github if you want to work on it yourself, but I would treat this as a starting point for your project, not a polished solution. Thirdly, I'm first to admit that this solution is a mess, but if anyone can think of something cleaner...

What's the problem?

The core media form field opens a modal dialogue box and iframes the 'images' view of com_media into the dialogue box. You can create your own custom form field which puts different views into the iframe, but the other views defined for com_media don't allow you to select a file. There is no way of parameterising the images view to tell it which file extensions to include in the view; that is hardcoded into a com_media model file. See line 120 of...


So, possible solutions include creating your own version of com_media (e.g. com_mymedia) where you copy all of the core com_media files into a new component, change its model file and then point your custom form field at a view from that custom component. This is a horrendous amount of work. It is also just about possible to override the images view of com_media using a standard template override. However this does serious damage to the MVC concept because it requires putting a load of code in the layout file that really should be in view.html.php or even in the model file. If, like me, you simply use the default bluestork template for admin, then you'd have to create you own admin template (by copying all of bluestork into your custom template) and then add a template override. Again a solution bordering on insanity.

The solution is...

Lately there has been work showing how you can override core components, so this is the approach I will take. I will override the view.html.php file of the images view of com_media and I also need to override some of the layout files. The changes that actually need making to the view.html.php and layout files are actually quite minor.

All of this takes place in a system plug-in that uses the onAfterRoute event to load in a new version of com_media's images view class that overrides the core one, as follows...

class plgSystemComMediaOverride extends JPlugin {
      public function __construct(&$subject, $config = array()) {
         parent::__construct($subject, $config);

     public function onAfterRoute() {
         $app = JFactory::getApplication();
         if('com_media' == JRequest::getCMD('option') &&
$app->isAdmin()) { require_once(dirname(__FILE__) .
'/code/com_media/views/imageslist/view.html.php'); } } }

This checks to see if we're loading the com_media component and we're in the admin site. If we are, then we require our new view.html.php file.

Note that I have stuck with the new convention of putting these sort of overrides in a directory hierarchy which starts in a directory called code and matches the existing hierarchy of the components and their views. You don't have to, its purely conventional. You could put the view.html.php file anywhere as long as the the require_once command points to it.

Now in the tradition of creating template overrides, you copy the following file...



...and you can happily edit this copy without damaging the core code.

Getting all the files, not just images

It turns out that the model code used by the imageslist view already gets all the files for you, its just that the imageslist view doesn't use the. The model offers a method getList which returns an array of folders, images and docs, where folders and images are self explanatory, and docs is everything else. The model's getImages method calls getList and then only returns the images part as follows...

	function getImages()
		$list = $this->getList();

		return $list['images'];

(/administrator/components/com_media/models/list.php line 38.)

view.html.php then calls getImages on the model, assigns the images to $images and then calls the layout files which display all the images in $images. So all we have to do is add all the docs to $images and it will then display all the files in the media manager, not just the images. So we replace...

$images = $this->get('images');

$list = $this->get('list');
$images = array_merge($list['images'], $list['docs']);

...and job done, all files are displayed.

Job not done

Except of course its not quite that simple. The non-image files do not have thumbnail images for them, and the existing layout files attempt to create a thumbnail using the relative path of the file, which the layout files are expecting to be images and now aren't necessarily. So we finish up with output HTML looking like this...

<img src="path/mysong.mp3" alt="..." />

...and depending on your browser you get bad images with question marks, error messages and such like. So we need to assign a thumbnail image to the non-image files. Unfortunately the layout files use the property path_relative to create both the thumbnail and to determine the file that is selected. So if we set path_relative to be the address of some generic thumbnail file, then it would display the thumbnail correctly, but the user would then select the thumbnail image, not the file itself.

Instead of just merging the images and docs together we need to go through them and add a new thumb property which the layout files can use. Now fortunately the model file already assigns icon_32 and icon_16 properties to the docs files based on their extensions. Unfortunately though these properties don't actually point to the images in media/media/images. They nearly do, but not quite. I don't understand this at all, but if you chop off some of address and add some other bits you get the appropriate thumbnail url. Well I did promise this was a bit of a mess, didn't I? Messy code as follows...

    $list = $this->get('list');
    $images = array();
    foreach($list['docs'] as $d) {
        $d->thumb = JURI::root() .
                'media/media/images' .
                substr($d->icon_32, strpos($d->icon_32, '/'));
        $images[] = $d;
    foreach($list['images'] as $i) {
        $i->thumb = COM_MEDIA_BASEURL . '/' . $i->path_relative;
        $images[] = $i;

Finally we need to override the layout files so that they display the thumb property as the thumbnail instead of path_relative. Copy the file at...



Then change the line...

<?php echo JHtml::_('image', $this->baseURL.'/'.$this->_tmp_img->path_relative, JText::sprintf('COM_MEDIA_IMAGE_TITLE...

<?php echo JHtml::_('image', $this->_tmp_img->thumb, JText::sprintf('COM_MEDIA_IMAGE_TITLE...

Now we need to tell the view where to find this override. I'm not sure whether there's a more elegant way of doing this, but the _path['template'] property of the view contains an array of paths to possible layout files, usually containing the template override and the layout files in the component. Presumably the first existing file in the array gets used. So we unshift our override into this array before calling the parent's display method.

    array_unshift($this->_path['template'], dirname(__FILE__) . '/tmpl');

So that's all! Working that out is about five hours I won't be getting back again. I hope it helps someone.


1st July 2014

(November 2019 update.) This was a tribute -- of sorts -- to a dear friend and possibility one of the greatest influences on my life: Richard.

Since I published this blog I then went on to expand on it considerably and it finished up as my first novel King of the Woods.

One of the themes of King of the Woods is what is true and what is fantasy and what might be true and is misremembered are all jumbled together, and this blog post would have spoilt that mystery, so I've removed it.

I know that deleting history like this might be bad form, but I've kept an archive copy, and I'm pretty sure that all the people important in Richard's life have a print out of it.

So sorry, but you'll have to buy the book instead!

Dear Internet… Really?

29th March 2014

Dad used to listen to BBC Radio 3 in the car. This was, I'd guess, the late 70s, early 80s. To my fresh young ears it was awful. Turgid, dull and incomprehensible. I remember things came to a head on one journey when some endless piece of ear bending serialism was delivered into the car with no introduction or context that I can remember.

'Oh, for heaven's sake.' said Dad and switched the radio off. That was the last I heard of Radio 3 for a quarter of a century.

My interest in classical music grew through my 20s, mostly due late night listening to a programme on the BBC World Service where I discovered the Brandenberg concertos, and other delights. I used to put Classic FM on in the office when I was doing my PhD, but eventually gave up, having been thoroughly patronised and insulted by their adverts. I did not want a golf holiday then, do not now, and hopefully never will. All this time Radio 3 remained a vague, unapproachable threat on the radio dial. A place, I thought, where music is revered with crinkled, very high brows and belittling sneers to those of us who didn't have much of a musical upbringing. A place not for me, nor the likes of me.

In late 2001 driving home from work, I decided that I could not face listening to the Radio 4 news any more; the War on Terrorism was too depressing. But where on the dial to go? Far too old for Radio 1, not interested in Radio 2, Radio 4 too depressing, local radio too vapid.

'Oh, for heaven's sake' I said, and pressed the previously untouched Radio 3 button.

The Golden Age

Over the next few months I was enraptured, not just by the music but by the completely unstuffy, approachable presenters who oozed passion and joy in the music they introduced. I made a list of the words routinely used on Radio 3 to described the music; 'glittering', 'revolutionary', 'spiritual', 'rousing', 'devastating'. I even shoe-horned this into one of my lectures on programming, asking why this sort of language is the preserve of the arts, not of science or engineering.

What a revelation to me that this music and this radio station were the most accepting, least elitist on the dial. With other genres of music you're either the wrong age, have the wrong attitude, the wrong culture or the wrong clothes. On Radio 3 there was not the slightest whiff of exclusion; all welcomed, a passion and curiosity about music, purely for music's sake was the only entry criteria. The door that I'd started to open was flung wide and I was hugely grateful. Over a decade later I still feel that I've only scratched the surface. I'll listen to that ear bending serialism now, because someone who knows it and understands it has explained it to me with care and passion. I still don't like it much, but maybe give it time. At least I now don't believe that that music is somehow inherently not for me.

To me the thing that unites Radio 3's output is the belief that whatever genre I'm listening to, it is pretty much the best there is; that it is music made purely for the love of the music, free of commercially driven compromise that diminishes most other forms of expression. After John Peel died there was much soul searching in the media, asking where was his replacement, where was the voice of pure enthusiasm for music, a voice willing to experiment and go in new directions, to invite the listener along for the ride. Even if the listener didn't like the places they were ridden to, that wasn't remotely the point, the point was that it wasn't boring. It was challenging. To me the answer to the question of where John Peel's replacement was is simple. Its Radio 3. And it is not just a few hours a week, tucked into the darker corners of the schedule, its all day, every day.

Oh, so very wrong

Then after a few years I was sitting bored in my office and started Googling around to see what others thought of Radio 3 and imagine my shock at finding that -- apparently -- I'd got it oh, so very wrong.

Take to the internet and you find that Radio 3 is completely dumbed down, a pale shadow of its former self. Filled with inane chatter and fatuous, smug presenters. Almost as bad as, in fact as bad as… and here it comes, the ultimate insult,  it is worse than Classic FM. Anyone who suggests otherwise is an idiot or a BBC stooge.

What amazed me about all this online venom is that classical music is supposed to be the ultimate enrichment to the human condition; music that challenges and questions us about what it is to be us, and that can answer many of those questions. I remember first hearing Pärt's Spiegel im Spiegel (on Radio 3 of course) while watching my daughter asleep in her cot and feeling like the music had clothed some essential part of me in sound. You would think that people who have spent all their lives being enriched by this wonderful music would be the most enlightened, delightful, erudite people you could meet.

But do you get that impression from reading Radio 3's Facebook page? Or do you rather get the impression that Radio 3 listeners are a dismal bunch of pedantic misanthropes, engaged in a battle with each other to come out with the most self righteously outraged comment, utterly lacking in humour, grace or any form of panache? Do you rather get the impression that they hate modern life, and in particular all those horrid young people who hang about the place, having fun and being enthusiastic about things?

So you go to the effort of liking Radio 3 on Facebook and then spend all your time making snide comments about it? Really? How much time do you think you have in this world, that you think this is a useful way of spending it?

You take to Facebook in a flurry of indignant pedantry to make some big issue about some minor mistake made during 'Building a Library' but you misspell 'library'? Really? I was wiping the irony off my laptop for weeks after that one.

You take to the internet to decry the lack of standards, discipline and respect in modern society, and then follow it up with deeply personal, offensive and childish attacks on individual presenters? Particularly the ones who who have the twin failings of being both young and female? None too preux chevalier, really.

And as for the hapless chap who took the time to castigate Radio 3 for playing Classic FM-ish lollipops like 'Short Ride in a Fast Machine' but got the name of the composer wrong. I mean really. Even I know that.

But then this is a completely unfair picture of Radio 3 listeners, and, believe it or not, this is absolutely not an attack on Radio 3 listeners, because all the ones I've met are all lovely in every way. I'm just talking about a noisy minority who think they own the art form and will hose abuse all over anyone who doesn't share that opinion.

Misanthropic, Ugly, Dreary Comments by Anonymous Trolls

Its not just Radio 3 listeners that seem to think that the internet is the perfect receptacle for their dreary misanthropy. As well as classical music, my other musical love is for folk music. But, oh my goodness, there's a website which proclaims itself to be the centre of all things folk, centred mostly around a discussion group. This discussion group is just the most wince-makingly dreary place. I've lost count of the number of times that I've been Googling this, that or the other about a folk singer or group, only to finish up on the site in question and spend the next half an hour, mouth hanging open in amazement, scrolling through page after page of snotty, narrow-minded grumbling.

And whatever you do, don't waste your life reading up on what they think folk music is and/or isn't. In summary, take the sentence 'folk music is what I say it is, and anyone who disagrees is an idiot' then repeat minor variations on this sentence several hundred times. There. I've read pages of tedious drivel, so you don't have to.

I sometimes think it's a generational thing, because the demographic for folk and classic music has an age profile that, shall we say, tends towards the autumn years. And maybe we all have a fixed amount of anger that needs venting, and lacking major global conflicts and things that actually matter, we chose to vent about trivialities on the internet. (As, you'll have noticed, I'm currently doing. There's paradoxical for you.) There are clearly all sorts of cognitive biases going on, particularly the reinforcement bias where you surround yourself with a clique of like minded misanthropes and finish up trying to outdo each other with who make the most miserable comment, completely losing any sense of how dreadful you appear to anyone outside your clique. If I was twenty years younger I'd very probably write a thesis about it.

Maybe I shouldn't worry about it. I realised a long time ago that I only read the folk music discussion group in order to get annoyed by the rubbish posted there, and vowed never to read it again. A vow I have kept, to no noticeable ill effect. I should probably do the same with the Radio 3 Facebook page, and just revel in the glory of listening to the radio station and not worry about the assorted piffle that pedants care to post about it.

And the answer is...

But before I go, if your default setting is to switch your computer on, navigate to your favourite discussion page and then spill some bile over it, I have the following questions...

Are you missing the fundamental irony that in the world wide web, you have been handed a means of communication undreamt of in human history, that just about all the information there has ever been is delivered to your laptop, that you can use to communicate all over the world with hardly any barriers and minuscule cost, and you use it to do what exactly? You use it to moan about how rubbish modern life is?


Varnish and Joomla!

17th November 2013

In the summer the BBC did a radio documentary about The Full English project. The interest in the site was such that the server couldn't cope, and was gummed up by the high volume of traffic for half an hour or so during and after the program.

Clearly not ideal.

Two months later the BBC repeated the documentary, and so we had to find a way to get the server to cope with the demand. We opted for Varnish -- a very highly spoken of HTTP cache -- which holds copies of popular pages in memory and serves them very efficiently from memory instead of getting them from databases and hard drives, which is much more inefficient.

The sites are Joomla! based. The problem is that Varnish and Joomla! don't fit together very well. Joomla! always passes a session cookie to the browser and the default behaviour of Varnish is to always go to the server for pages with cookies and bypass the cache; so by default Varnish will have no effect because the cache will always be bypassed. You could tell Varnish just to ignore cookies, but this also causes problems because if the user is logged in then the site should bypass the cache, but if the user is a guest, then its safe to server content from the cache.

There's disappointingly little advice and discussion to be had from Mr Google, the main contenders seeming to be massivescale's commercial product where Joomla! is patched to stop it generating the session cookie. I wasn't too keen and shelling out for massivescale's version, and most Joomla! experts advise against patching the core. (That said, I haven't tried massivescale's product, so certainly have no other comment to make about it -- they offer documentation and some support -- so if you're running an important site and are not so much of a programmer, probably best to go with them.)

There is also some likely looking code on the snipt site which recommends adding a little code to the template or one of the plug-ins to generate a response header which signifies when a user is logged in or not, as follows...

// Set user state in headers
if (!$user->guest) {
	JResponse::setHeader('X-Logged-In', 'True', true);
} else {
	JResponse::setHeader('X-Logged-In', 'False', true);

Varnish can then read the header and serve content from the cache or server appropriately.

So we went with the configuration from the snipt site, and Ta Da! with a bit of minor tweaking the cache worked and the sites survived the repeat of the BBC documentary, with higher traffic than the first airing. So relief and happiness all round.

Then the problems started creeping in. Mostly to do with logging in and out. We haven't got many users logging in; currently only a handful of volunteers who are submitting crowd sourced content. When a user went to the log in page and logged in, then they would be shown a cached version of the log in page (ie showing them not logged in). A few refreshes and maybe jumping to the home page would -- usually -- get the response header working, the cache bypassed and everything working. This was clearly not ideal though, as we were having to tell our volunteers to bang refresh a few times and hope for the best.

The problem is with the response header, because the header comes from the server in the response, not from the client. So if Varnish decides to serve from the cache as opposed to from the server, you can set as many response headers as you like; Varnish won't see them because its serving a cached version, and if it has to go to the server every time to find out whether the user is logged in or not then that defeats the point of the cache.

The signal that user is logged in must come from the browser -- via a cookie -- not from the server via response headers.

What would be ideal if there was some way of examining Joomla's session cookie to see if it is a guest session or not. But as far as I can see there isn't, and there are probably security related reasons why. So we implemented another cookie to run in parallel with the Joomla session cookie which can be easily detected by Varnish's VCL and is simply a flag telling Varnish to bypass the cache or not. So we can tell Varnish to bypass the cache not only when the user is not a guest, but also when the user is doing something with log in pages (ie. when the Joomla component is 'com_users')

So, firstly there is a Joomla plug-in to set/unset the cookie, as follows...

class  plgSystemVarnish extends JPlugin
	function onAfterRoute( )
		$user = JFactory::getUser( );
		$option = JRequest::getVar( 'option' );
		if( !$user->guest || $option == 'com_users' )
			setcookie( 'varnishbypass', 'True', time( ) + 604800, '/' );
			setcookie( 'varnishbypass', '', time( ) - 604800, '/' );
	// setting the cookie in the afterRoute event is probably adequate,
	// but in an outburst of belt and braces I decided to set/unset the
	// cookie on the log in and out events too. Probably unnecessary,
	// but apparently harmless.
	function onUserLogin( $response )
		setcookie( 'varnishbypass', 'True', time( ) + 604800, '/' );
	function onUserLogout( $response )
		setcookie( 'varnishbypass', '', time( ) - 604800, '/' );

...then an adaptation to the VCL configuration from the snipt site as follows...

	if( req.url ~ "^/administrator" || req.request == "POST" ||
				req.http.cookie ~ "varnishbypass" )
		return( pass );

...and even more Ta Da! this works and respects the users logging in an out.

Two observations, though. Firstly this works for our site, but as I said, we only have a few people logging in and out, and they're volunteers who are fairly tolerant of the site doing odd things; they are not members of the public who would rightly give up in disgust and walk away if the site was doing odd things. I'm making no claims for the general applicability of this method, it is entirely possible that this solution will introduce strange behaviour if your site does log ins differently to our sites. But as far as I can see you have to signal that a user is logged in or not via a cookie, not by headers. Secondly it was a real mare getting this working on a live site; if at all possible you should design your site to work with Varnish from the start rather than adding it in later.

I did a PhD and did NOT go mad (2013 update)

2nd November 2013

!!In about 1997 I was asked to give a presentation in (what was then) the School of Computer Science at Middlesex University about my experience of doing a PhD. It was to be given as part of an induction day for to the PhD students just starting off in the school.

I'd just completed my PhD and felt somewhat jaundiced about the whole process, mixed with the startling feeling of exhilaration that lasts for several months after completion and decided that I should tell the new intake exactly what my experience had been -- warts, spots, disastrous relationships, and all.

This was probably somewhat unfair on the audience.

 It was also probably a bit unfair on the senior academics who had spent the morning telling the new intake how this was going to be the best time of their lives -- the thrill of pushing the boundaries of knowledge -- the freedom to pursue challenging research paths. My subsequent talk came as a contrast.

I'd done some cartoons to go on acetates for the overhead projector (remember those, academics over 40?) and instead of doing the usual thing of stuffing the cartoons in my cartoon drawer never to see the light of day again I decided to write up the text of talk, scan the cartoons and stick them on one of those new fangled web pages. This was long before the days that social media had been dreamt up, or even before the days when comment boxes were common.

In the subsequent sixteen years the web page travelled with me in the various incarnations of my different personal web sites as my career hopped about here and there, eventually finishing up forgotten about on some old server space that a friend had lent me when I first set up a website outside the groves of academe. I'd been aware that a few people were looking that page, and judging by occasional email I got saying from distraught students saying how I'd described their wretched lives perfectly, its contents seemed to be still relevant.

It came as quite a surprise when I finally started taking the strange world of Twitter seriously to find out that not only was I did a PhD and did NOT go mad still online but seemed to have gathered quite a loyal following. After a bit of Googling I was also rather gratified to find that my words had taken on quite a life of their own, and in the minds of several internet authors at least, I had been promoted to Professor Butterworth.

So as I'm setting up my new website and blog I thought it was obviously time to drag I did a PhD off its forgotten server, into the 21st century, and into the social media world.

It is, I'm pretty sure, the only thing I wrote as an academic that was read by more than ten people. A lot has changed since I wrote it, but the crucial importance of chocolate has not. And never will.

I did a PhD and did NOT go mad

3rd November 1997

Before reading these wise words advising you how to do a PhD (inspired by three years of the author carefully and diligently banging his head on a table) you are requested to read and digest the following irony...

The only way to find out how to do a PhD is to do one. Therefore all advice is useless.

To say that I enjoyed doing my PhD would be a lie, not just an ordinary lie mind you. More the sort of lie one would normally associate with Tory party conferences. A big wobbly lie with a dusting of sugar on top. At times I hated my PhD, so why do I have any authority to give advice on doing a PhD? Well, I don't claim to have any -- other than the fact that I completed and passed the thing, so I must have done somthing right.

This page is an attempt to...

There's only one way of doing a PhD and that's your own, that's your own, that's your own...

A PhD, by its very nature, is a very individualistic venture. There is no right way to do a PhD (there are however a multitude of wrong ways). This is the first big surprise for people who are starting their PhD having completed their undergraduate degrees -- there are well defined correct ways of getting a degree (usually 'turn up to lectures, do course work, revise for exams, use a modicum of common sense') but such prescriptive techniques don't work for research degrees. The award of a research degree effectively says `This person knows how to do research in his/her chosen area' and `research' is a nebulous, difficult to nail down thing which relies on insight, lateral thinking, inspiration and a lot of hard work. An undergraduate degree is a lot of hard work, but doesn't put so much emphasis on inspiration. Most (if not all) people cannot sit down and say 'right, today I'm going to have some inspiration'. The unpredicable nature of progress in a PhD means you spend a lot of time not sticking to the deadlines you set yourself. This is dispiriting.

In most departments there's always one smartarse who loudly proclaims that doing a PhD is easy and he (its usually a he) can't see what all the fuss is about, and he's just written another three chapters this morning, and he's submitted another five journal articles. People like this are lying, showing off, from Mars or over-compensating for inadequacies in other areas of their lives (if-you-know-what-I-mean). Ignore them.

The loneliness of the long distance researcher

The next big surprise for people who are starting PhDs after an undergraduate degree is just how excrutiatingly lonely a PhD is. When you submit your thesis you have to sign a piece of paper that says 'This is my work, my work alone, nobody else's, all mine, nobody but me did it. Honest.' A PhDs is so narrow and focused that the chances are that you and only you is going to understand anything about it. This is known as the Loneliness of the Long Distance Researcher and you have to get used to it. It means you're going to spend at least three years wandering around with a great chunk of bizarre, irrelevent nonsense in your head that only you can relate to. Get used to people's eyes glazing over and them shuffling in their seats when you try to explain exactly what it is you're doing. Try to avoid emotional entanglements with people who say `penny for your thoughts' during romantic moments. If, like me, you say `the problem of formally refining liveness properties stated in a temporal logic for reactive systems' then you're likely to find yourself rapidly emotionally unentangled. Or at least they ask for a full refund on their penny with a written apology. Anyway, quite frankly, after a couple years of doing my PhD the last people I wanted to spend any time with were the sort of people who would be remotely interested in my work. To quote Groucho Marx 'I wouldn't want to join a club that would have me as a member.'

Many departments have well integrated research programs with seminars and meetings and other such social gatherings. This is A Good Thing and will go a long way in relieving the loneliness of the long distance researcher. Many departments are, however, abysmal in this respect. If you're an active go-getting sort of person you may try sorting out social gatherings but be prepared for knockbacks. I did my best to get my colleagues down the pub, but the world record for number of research students in one pub at the same time was (from memory) nine out of forty -- and that was only when two Christmas parties accidentally went in the same pub at the same time. Remember, if you can't persuade other researchers to leave the sanctity of their computer screens and come down the pub its probably not your fault -- they've probably forgotten what a pub is, what you do in one and why. When you start your PhD you'll probably think that they're a sad bunch of herberts, but believe me, by the time you've spent three years doing a PhD you'll have a lot more sympathy for them.

Thank you for your support. I shall always wear it.

Although doing a PhD is an individual thing your department should give you as much support as possible. I fear that the attitude of some departments is rather Victorian to their research students -- that having a miserable time when doing your PhD is character forming and uplifting and you'll thank them for forcing you to spend three years in an unheated office with only a wasp nest for company in the long run. Such sentiments are usually followed by `anyway when I did a PhD we didn't have offices -- we did our research in the departmental lavatories and were hosed down once a month by the cleaning staff. You have it lucky'. Anyone of this attitude really deserves a kick in the pants of the most severe measure, but I bet you'll come across at least four of them. If you get treated badly by your department then don't be shy -- moan. Don't moan for the sake of it because you'll just get peoples backs up, but make sure that those in authority know about your grievances and don't have the excuse once people get fed up and leave their degrees halfway through (as they do) that it wasn't known how upset they were. It is likely that you have a good first degree and could therefore be In The Real World doing a job that pays about ten times as much as the pittance you get as a stipend for a research student. Really you're doing your department a favour by publishing results and upping their research credibility and hence they should show you at least a little respect and common decency.

Use your hanky

There are going to be times when it all gets to you and you can't cope any more. (The day I found a paper that I'd missed in my literature survey that covered all the `new' stuff I'd done in the previous nine months was my own personal nadir.) What do you do in such circumstances?


Your eyes out.

Since the tragic death of the Princess of Wales we all now know how to express our negative emotions fulsomely and publicly without caring a jot. So forget your stiff upper lip and let it tremble. But don't waste that outpouring of emotion -- make it count as much as possible. In a sentence, burst into tears in your supervisor's office. If you're really determined to gain dramatic effect then wait until he/she's entertaining some important guests -- Profs from other universities or co-authors on important papers. However if your supervisor is one of those 'My door's always open' types who spend all year at conferences in the Yemen or can't be seen without booking two months in advance with his secretary then you may have problems. An email isn't very effective in this case...

To: Prof Vacation []
From: Hopeless inadequate useless bag of rubbish []
Subject: Bleeeeeaaaah!

Dear Prof,

Boooooohoooooo, snivel, snivel, bleeeeaaaaaah...


Its not the same is it?

While we're on the subject of dealing with emotional catasphrophes I should mention the crucial role that chocolate played (and is still playing) in my academic career. There is no problem known to science that cannot be cured by the liberal application of chocolate. Leading doctors have testified to its mystical curative powers -- it is known to contain all sorts of wholesome, bracing chemicals that get straight to the happiness centres of your brain and get them working at full pitch, scattering love and joy and contentment all through your cortex. Leading dieticians and skin care specialists may make the odd carping comment, but take no notice. Just consider the following scientific breakthroughs directly attributed to chocolate...

Need I say more?

In case of emergency...

There are of course several methods of stress relief to be tried -- popular ones include Staring Out of Windows and Watching Old Black and White Films. Obviously with minimum effort you can combine these with Eating Chocolate for a previously unprecedented amount of stress relief. It is more difficult to combine Staring Out of Windows and Watching Old Black and White Films though, at least not unless you put your telly outside your window. (Not recommended during inclement weather or for those in high rise buildings.) I have been advised on medical and moral grounds not to include Getting Monsterously Bladdered and Uncomplicated Sex in my list of stress relievers, but you'd probably thought of those already.

In order to not become entirely divorced from reality it is a good idea to get a hobby and the less your hobby has to do with the subject of your PhD the better. If you are considering doing a PhD on a subject you really enjoy and find fascinating then beware -- the chances are that after three or so years of studying nothing but your chosen topic you'll hate the very mention of its name. Do you really want this? My PhD was in a Bizarre Corner of Computer Science and during my PhD some of the hobbies I'm willing to admit to are learning to play the mandolin, taking watercolour lessons, going on some very long walks and sewing. My chocolate consumption skills also improved dramatically as I think I may have mentioned elsewhere.

Its the end of the world as we know it, and I feel fine (sort of)

So, in summary...

Further reading...

If you're really determined to keep wasting time on the internet instead of knuckling down and doing that really dismally boring data analysis that you set yourself a deadline to complete five weeks ago, you can have a look at my notes about I did a PhD and did NOT go mad. Then after that there's Facebook and Twitter, and your own blog to update...