Of course neither he nor the Labour Party are anti-semitic. He supports the moderate Palestinian cause, and quite right too, and some Zionists love to confuse that and anti-semitism, and naturally the Hate-Mail, the Excess and the Scum will jump on that bad-wagon: any stick to hit Labour with, regardless of the truth.
[Edit] "Band-wagon" I meant above, although then again...