How Fearing Adultery Hurts All of Us

I felt badly for Herman Cain's wife who, when news hit of her husband's 13-year affair, reportedly said, "Here we go again."
But it can't be ignored that it was news of his adultery that derailed his presidential nomination – NOT the reports of sexual harassment.
And this is, I think, part of the problem.
Sexual harassment is a crime. Adultery...is not. Adultery is hurtful absolutely. But it's not criminal.
Yet we live in a society that treats it as more serious...or at least a portion of society treats it as such. And by keeping it in the shadows, we don't talk about it, nor do we discuss how to deal with it if it happens.
Instead, we all pretend it only happens in bad marriages. Or is committed by people of poor character. While I'm not defending those who engage in adultery (though it might sound as if I am), I am saying that we can't tar all adulterers with the same brush (as tempting as it is to tar them with something...especially when you're the hurt party!!).
My overwhelming wish, having survived betrayal, is that society discuss this as something that can, and frequently does, happen in marriage. Even "good" marriages. But bringing it into the open, I'm convinced that, rather than make it more acceptable, we'll make it less attractive. By seeing the pain it causes in relationships, by having honest discussions about the impact and the long, rocky road to recovery, I would hope it would seem a whole lot less tempting.
What do you think? Does treating adultery like the scarlet letter make it more likely? Or less?
As always, I'd love to hear your thoughts.