Society was, and is, interested in ensuring legitimacy of children so that distribution of property / wealth is within known framework, and no surprises spring up. Society is also interested in general health of people. Consequently, taboo on extra-marital affairs. So that no illegitimate children are born who can later become claimants on property; so that STDs do not spread and wipe out population.

Of course, at an individual, extra-marital affairs cause jealousy and insecurity. 

Contraceptives and Automobiles have solved these problems.Contraceptives prevent illegitimate children and STDs. Automobiles enable people to travel so that their partners do not get to know of their affairs.

PS: inspiration and elaboration at Marriage and Morals