If a Jew wears clothing made from pig skin (e.g., suede), does s/he have to watch his/her hands before eating?

My question is based upon the fact that pigs are not kosher and thus it would seem a bit awkward to touch clothing from pig prior to eating without washing one's hands first. I do know there are opinions that a Jew may wear a pigskin hat, suede shoes, etc.