IMHO, the only solution is to run. You can't meet him for dinner tomorrow, because it will definitely lead to more. You obviously seem drawn to him, so that is kind of a given?
I was in a sort-of similar situation as you, because I was in a long-term relationship when i met the psychopath. I tried to break up with my partner and told her I was going to start seeing the psychopath, but she still wanted to stay living with me for the time being. Eventually, we stopped living together, but remained close friends. Anyway, the point being is that, despite the fact I didn't want it to be an "affair" situation, it kind of wound up that way because the first partner wouldn't let go. Also, for part of the time the Psychopath lived far away, so we only could see each other periodically. Every time we did see each other, it was this fabulous, romantic, dreamy situation for the most part.
So, I think part of the allure of the situation is that it is like that... romantic, intense, temporary. But, if you were actually to LIVE with the Psychopath or be with him in a stable relationship, most likely it wouldn't be quite the romance you'd expect. It'd be a lot of giving and not much getting in terms of emotional support. That is when my Psychopath relationship went south... when I actually had emotional needs and troubles in my life, so some of the focus and attention of the relationship had to be toward me. Instead of giving it, she just went to someone else whom she could take from.
In other words, instead of looking at it as this one time meet-up, maybe it is good to purposefully think about whether you would actually want to have a life and future with the Psychopath, give up your marriage. If you think along those terms, maybe it is easier to just say 'no' when it comes to the dinner. Being with a Psychopath is an addiction. After the first kiss, you are a goner! Your hooked! don't let yourself be hooked and reeled in like a big fish.