All the debate has come up against a stubborn fact: Brexit is damaging to the British national interest.