To cancel the event (the submit in this case) you woudl use xfa.event.cancelAction = true;
This was added in version 8 I believe so if you are using a Reader/Acrobat earlier than that this will not work.
You could have your real submit button hidden and expose a regular button to the user for submission. This woudl allow you to write code on the regular button to check your validations and if they work out then you woudl hit the submit button programmatically:
buttonName.execEvent("click")
Hope that helps
Paul