Well, it's not fully your fault. Now we do have a worried ruler who is freaking out about how things are going back in the Otherworld, and cries when they see mentions of the Otherworld because of how they left!
Also, your character could find or bring together the other's to help your spouse raise in your power (or that's how you gain their affections). Then we decide to stay together since your character is doing their studies, spouse cheats on you, cheater wants your character's throne, and sends us away.