Yaar deekhooo that same reason why we punish a murder with death; paradoxically under the logic of preserving the sanctity of life ! The same reason why we hang a rapist - Even though, logically speaking, rape is pretty much the same action as sex, but with the consensual part to it - One might even argue that you probably shouldn't punish anyone for that because it was the same series of movements, why is consent such a exalted barometer ?
But we don't - Why ? Because we attach value to such things ! Similarly we, attach, value to the sanctity of the social contract called marriage & recognize it as a building bloc of our society; therefore, we've determined, that we're going to keep a severe deterrent here known to all that we will not tolerate a violation of its sanctity !
Now who determines that - The Society ! For that is exactly what 'right' & 'wrong, purely from an academic point of view is, something we've agreed upon & we've spawned logical discourses to support that stance for logic is malleable to the point of supporting anything from the most heinous of crimes to decrying the most beautiful acts of kindness.