I did not have children with my Psychopath but I did live in the same town and I have been in exactly the situation you have described, but, to my shame, I was stupid enough, and delighted enough to see him after missing him so badly, to have let him have sex too.
Unfortunately for me at the time, for the few hours he was there, it felt like real closeness and "coming home" with the two of us away from the world in a beautiful, wonderful bubble, until he left, but I knew nothing about "everyday" Psychopaths then, I only knew the term in relation to serial killers.
I found it was one thing for me to be strong enough to have no contact and keep myself away from him, but maintaining that was beyond me for a long time if he literally put himself in front of me when I was totally unprepared and had virtually no defences.
Of course he would disappear again having confirmed he could still affect me. He liked to get in touch at around 1-3 a.m, invariably after he had been drinking. I don't think it was Dutch courage, just boredom, and maybe feeling like sex.
From their point of view it is win/win. They are drunk and bored at a time when no one else is around for them to play with, so they come looking for us.
At the very least they have the satisfaction of disturbing us physically and waking us up, i.e. they still have power over us.
They force us to notice them i.e. they still have power over us, when they have power they can cause hurt, discord and confusion.
They force us to drop no contact, i.e. they still have power over us, when they have power they can cause hurt, discord and confusion.
They get to disturb us emotionally,i.e. they still have power over us,when they have power they can cause hurt, discord and confusion.
They get us to admit we still love them, i.e. they still have power over us, when they have power they can cause hurt, discord and confusion.
They get us to respond to them physically, i.e. they still have power over uswhen they have power they can cause hurt, discord and confusion,(and at best they get sex as well, and that intimacy, on our part, gives them more power).
It was no accident he prevented you seeing who was there before you opened the door and most people are going to be disorientated and not thinking quickly and clearly when disturbed under those circumstances with no chance to be prepared. (That's why the police do dawn raids.)
At least, unlike me, you didn't give him the satisfaction of being any more physical.
Put it behind you, but use the knowledge to be stronger in future.
We all of us have backward steps trying to get away from our Psychopaths.