The difference between Ghosts and Brexit is that the people who like or dislike Ghosts are perfectly capable of deciding whether or not they enjoy watching it.
In stark contrast, 99% of the people who voted for or against Brexit know absolutely nothing about the subject and would not survive an argument (no matter how short) with any of the knowledgeable 1% who held an opposing view.