Richard Butterworth Digital Media Design

Varnish and Joomla!

 
 
 
 
 
 
 
 
 
 
 

In the summer the BBC did a radio documentary about The Full English project. The interest in the site was such that the server couldn't cope, and was gummed up by the high volume of traffic for half an hour or so during and after the program.

Clearly not ideal.

Two months later the BBC repeated the documentary, and so we had to find a way to get the server to cope with the demand. We opted for Varnish -- a very highly spoken of HTTP cache -- which holds copies of popular pages in memory and serves them very efficiently from memory instead of getting them from databases and hard drives, which is much more inefficient.

The sites are Joomla! based. The problem is that Varnish and Joomla! don't fit together very well. Joomla! always passes a session cookie to the browser and the default behaviour of Varnish is to always go to the server for pages with cookies and bypass the cache; so by default Varnish will have no effect because the cache will always be bypassed. You could tell Varnish just to ignore cookies, but this also causes problems because if the user is logged in then the site should bypass the cache, but if the user is a guest, then its safe to server content from the cache.

There's disappointingly little advice and discussion to be had from Mr Google, the main contenders seeming to be massivescale's commercial product where Joomla! is patched to stop it generating the session cookie. I wasn't too keen and shelling out for massivescale's version, and most Joomla! experts advise against patching the core. (That said, I haven't tried massivescale's product, so certainly have no other comment to make about it -- they offer documentation and some support -- so if you're running an important site and are not so much of a programmer, probably best to go with them.)

There is also some likely looking code on the snipt site which recommends adding a little code to the template or one of the plug-ins to generate a response header which signifies when a user is logged in or not, as follows...

// Set user state in headers
if (!$user->guest) {
  JResponse::setHeader('X-Logged-In', 'True', true);
} else {
  JResponse::setHeader('X-Logged-In', 'False', true);
}
 

Varnish can then read the header and serve content from the cache or server appropriately.

So we went with the configuration from the snipt site, and Ta Da! with a bit of minor tweaking the cache worked and the sites survived the repeat of the BBC documentary, with higher traffic than the first airing. So relief and happiness all round.

Then the problems started creeping in. Mostly to do with logging in and out. We haven't got many users logging in; currently only a handful of volunteers who are submitting crowd sourced content. When a user went to the log in page and logged in, then they would be shown a cached version of the log in page (ie showing them not logged in). A few refreshes and maybe jumping to the home page would -- usually -- get the response header working, the cache bypassed and everything working. This was clearly not ideal though, as we were having to tell our volunteers to bang refresh a few times and hope for the best.

The problem is with the response header, because the header comes from the server in the response, not from the client. So if Varnish decides to serve from the cache as opposed to from the server, you can set as many response headers as you like; Varnish won't see them because its serving a cached version, and if it has to go to the server every time to find out whether the user is logged in or not then that defeats the point of the cache.

The signal that user is logged in must come from the browser -- via a cookie -- not from the server via response headers.

What would be ideal if there was some way of examining Joomla's session cookie to see if it is a guest session or not. But as far as I can see there isn't, and there are probably security related reasons why. So we implemented another cookie to run in parallel with the Joomla session cookie which can be easily detected by Varnish's VCL and is simply a flag telling Varnish to bypass the cache or not. So we can tell Varnish to bypass the cache not only when the user is not a guest, but also when the user is doing something with log in pages (ie. when the Joomla component is 'com_users')

So, firstly there is a Joomla plug-in to set/unset the cookie, as follows...

class  plgSystemVarnish extends JPlugin
{
  function onAfterRoute( )
  {
    $user = JFactory::getUser( );
    $option = JRequest::getVar( 'option' );
    if( !$user->guest || $option == 'com_users' )
    {
      setcookie( 'varnishbypass', 'True', time( ) + 604800, '/' );
    }
    else
    {
      setcookie( 'varnishbypass', '', time( ) - 604800, '/' );
    }
  }
  // setting the cookie in the afterRoute event is probably adequate,
  // but in an outburst of belt and braces I decided to set/unset the
  // cookie on the log in and out events too. Probably unnecessary,
  // but apparently harmless.
  function onUserLogin( $response )
  {
    setcookie( 'varnishbypass', 'True', time( ) + 604800, '/' );
  }
  function onUserLogout( $response )
  {
    setcookie( 'varnishbypass', '', time( ) - 604800, '/' );
  }
}
 

...then an adaptation to the VCL configuration from the snipt site as follows...

  if( req.url ~ "^/administrator" || req.request == "POST" ||
        req.http.cookie ~ "varnishbypass" )
  {
    return( pass );
  }
 

...and even more Ta Da! this works and respects the users logging in an out.

Two observations, though. Firstly this works for our site, but as I said, we only have a few people logging in and out, and they're volunteers who are fairly tolerant of the site doing odd things; they are not members of the public who would rightly give up in disgust and walk away if the site was doing odd things. I'm making no claims for the general applicability of this method, it is entirely possible that this solution will introduce strange behaviour if your site does log ins differently to our sites. But as far as I can see you have to signal that a user is logged in or not via a cookie, not by headers. Secondly it was a real mare getting this working on a live site; if at all possible you should design your site to work with Varnish from the start rather than adding it in later. 

 

 

Comments

#3 Fotis Evangelou 2015-11-04 04:28
The "checking" needs a better combination of the HTTP header and the user cookie.

Here's the one configuration to rule them all (improved upon my initial post on Snipt.net), no hacks in Joomla, no elaborate setups, no need for a plugin. Just the .vcl file plus a short snippet to place at the top of your template's index.php file. The approach is similar for other CMSs too.

https://gist.github.com/fevangelou/84d2ce05896cab5f730a

Tested & working on some of the highest traffic Joomla websites worldwide :)
Quote
#2 Todd J 2014-03-24 23:22
Thank - this was a huge help.
Quote
#1 Keith 2014-03-13 18:37
Nice article. Glad for all the belts and braces you put into this! I am going to try it out.
Quote

Add comment


Security code
Refresh