If sex is important enough to you, try harder to give up other sensual pleasures, just an idea. The Buddha always encouraged Married men to be especially good to their wives and remain faithful. Just how giving up sex works in that equation doesn't make much sense to me, but some couples do it, just maybe not you, I wouldn't lose sleep over it, unless you're loseing sleep because you're having sex!!
18 years ago I made one of the most important decisions of my life and entered a local Cambodian Buddhist Temple as a temple boy and, for only 3 weeks, an actual Therevada Buddhist monk. I am not a scholar, great meditator, or authority on Buddhism, but Buddhism is something I love from the Bottom of my heart. It has taught me sobriety, morality, peace, and very importantly that my suffering is optional, and doesn't have to run my life. I hope to give back what little I can to the Buddhist community that has so generously given me so much, sincerely former monk John