Default OOTB behavior on a fresh 6.5 instance should actually be the way like you have described it. Never had any issues with it. Maybe you got some browser extensions for that? Have you also tried it with another user in the Incognito browser tab?
In general my advice would be to never customize functions like this since it might lead to a lot of issues also in conjunction with product updates. You should stay as close as possible to the OOTB behavior AEM-wise at least.
I am not facing anything what you explained in your query. For me Sites Console is opening in the current browser tab after clicking on cancel button in both the cases.. Please check if you are clicking cancel button with ctrl.