Adultery is a breach of contract.
Some guy going out and fucking women on the side you risk giving your wife an STD. Same if the wife cheats, and in many ways its fundamentally unfair for some one to work hard to support a spouse and children and then wake up your house taken away from you...