Check if you can do anything in script or field pattern.
Another alternative is :
Take the text field and add the below script in its change event
if(isNaN(xfa.event.change)) xfa.event.change = "";
if(isNaN(xfa.event.prevText)) xfa.event.change = "";