Yes, but on the other hand, if a man keeps coming into your house and yelling, burning your religious texts and comic books, cussing out your friends' mothers and making your guests/customers feel uncomfortable or angry, I don't think it counts as a violation of his speech rights to shut the door or call the police on him...
