Why do the Tories find it so hard to win elections? A loyalty explanation

Recently I've been looking at a lot of data from the BHPS, the long-running UK panel survey.  And amidst looking at unemployment, welfare-to-work and wellbeing, I've been side-tracked (procrastinating) by looking at people's choice of political party over time.

One of the main advantages of the BHPS is that it tracks the same individuals over time.  So unlike a traditional survey (where we only look at people once - a cross-section), we can examine how the same individuals' behaviour changes - or doesn't - over time.

With regards to political parties, this means we can use panel surveys to see how flexibly people move their support between different parties.  And this is where the BHPS can come in useful.  It ran between 1991 and 2008; precisely the time when the Tory hegemony over UK politics was coming to an end (and is yet to return).

One plausible explanation of why the Tories have found it so difficult to win a majority is that their support moves about more.  In other words, people might move closer to the Conservatives in year X but then shift to another party in year Y.  In addition, Labour might find it easier to win a majority because people are less likely to shift their allegiance from Labour to another party.  So, looking at the BHPS, to what extent is this the case?

Looking only at people who stated they were closest to one of the three main parties, over the course of the panel we see 60254 recorded instances of people choosing Labour (52%), 40743 choosing Conservative (35%) and 15556 choosing Liberal Democrat (13%).  This is not too surprising; during the course of the panel we had three major Labour general election victories.

No. of recorded preferences for the three main parties, BHPS 1991-2008

However, we're not really interested in the question 'how many people said they were closest to the main parties?' but, within their respective pool of supporters, how many people stuck with the same party over time?  Are Labour supporters more likely to stick with Labour than Tory ones are?

Percentage of respondents who stuck with the same party over time

And this what we see.  The Lib Dems - perhaps unsurprisingly as the third party of British politics - have the least loyal supporter base: only 64% of Lib Dem supporters only ever supported the Lib Dems and not the two main parties.  

However, the graph also shows the Tories falling short of Labour.  All in all, 88.5% of people who said they were closest to Labour out of the three main parties didn't switch their allegiance to the Tories or the Lib Dems.  The corresponding figure for the Conservatives is 83.8%.  This means that over time, whilst around 16% of Conservative supporters shifted their preference, only 11.5% of Labour supporters did.  This might sound a relatively small difference; yet as the Conservatives found out in 2010, they failed to recoup the vital 3-5% of the electorate they needed to get a majority.

There are of course many other reasons why the Tories haven't won since 1992: bad leadership, bad policies and, whilst it lasted, a booming economy.  But the above data suggests another important reason: Tory supporters are more likely than Labour ones to switch their political allegiance.  And with the rise of UKIP, for David Cameron things - surely -  can only get worse.