You can restrict user input by modifying xfa.event.change during the change event.
To remove the characters you mention, try this script:
xfa.event.change = xfa.event.change.replace(/[\"\\[:;<?!~]/g, "");
Note that most often xfa.event.change is a single character when the user is typing, but could be an entire string if the user pastes into the field.
This script handles both cases.
John