Adulterer (or adulteress) can be used for anyone who has a sexual relationship with a person who is married to another, but again does not always imply the specific social situational that mistress does.

What do you call a man that cheats with a married woman?

A man who cheats on his wife is an adulterer.

What do you call a woman who cheats with a married man?

A man who cheats on his wife is an adulterer. A woman who cheats on her husband is an adulteress. An adulterer commits adultery with his mistress, or lover, or paramour or girlfriend.