Unless you are really tight with everyone, just move on. Its your time as well, **** him if he wants to be a bitch. Im married with 2 kids and I still get the **** out of the house to play games. If someone does come over to play, The kids are sequestered. I try very hard to put forth a good face for my children when interacting with people so that they learn what is acceptable. The fact that this guy does not says a lot more than anything else you've complained about. Of all the things, THAT should be your bright red flag.

Move on.