Business rules can help with that.
I experimented with inverse of this and it was success, meaning that I applied a rule "if there is an attachment show error message asking user to remove it" (there was additional condition involved).
And this worked, so I believe having rule to ensure document is uploaded would also work.
BTW. I also tested this with linked documents and they are evaluated in the same way, so if document is linked, it's like if it was attached.
here is modified sample of the rule I used:
IF({categoryID}="5cc7a96500f2a9c6907a4cf1b2a39135" && {hasDocuments}=true, ": This ...., attaching files is forbidden. Please include URL for the file","")