If you refer the OOB workflow 'DAM Update Asset' workflow, there you can configure which file types should be allowed. Similarly, you can write custom workflowprocess service to check the type and size of the asset and restrict the uploads accordingly.