I would follow the below steps for the above mentioned requirements -
1. I will create a File Upload component with the ability for the user or the author to browse the file.
2. Upon clicking on upload button I will make post request from File Upload component to custom servlet to trigger my workflow. In this servlet I will capture that uploaded file name and construct payload path as mentioned here -
https://aem.redquark.org/2019/12/workflows-in-aem-05-trigger-workflow.html
3. Finally my custom workflow model(process step) will upload the files to the DAM using below approach -

Please consider one more point if someone try to upload same file then do you want to show any message.