It's not him they hate. It's humanity. His brother isn't exactly eye-to-eye on his pacifistic life-loving thing; he's more on the 'everyone deserves to die' side. [He's trying to be vague, because really, it would be so much easier if Vash would just TELL PEOPLE THESE THINGS.] I'm guessing Vash hasn't told you much at all.