As far as the two main parties are concerned the Tories are probably more evil than Labour but possibly more competent (though I doubt it, they are cut from much the same cloth).
The real reason for voting out Labour is that they are wed to perpetuating failing policies because their credibility depends on it. Anyone new gets to make a fresh start. The trick is to rotate parties at elections, so that there is a constant refreshing of ideas, but no-one becomes entrenched long enough to do any irreversible harm. History would have remembered Blair much more kindly if he had only had the one term.