Working around with AEM page optimization, got a requirement to convert the DAM asset images to next-gen formats like webp. Is it possible to create a workflow to do the same. The workflow should access the DAM asset images and convert it to required format and rewrite the original file.
This is easily possible. Extend the "asset update workflow" with a new step, which creates the new image as a rendition to the original asset. Do not try to replace the original rendition, as this is normally the workitem of the workflow, and removing/re-adding while the workflow is running is likely to cause issues.
Also, I don't think that replacing the original asset is a good idea either. If you need to create a more efficient way to deliver it, I would either require the users to upload the assets already in that format, or create it as a rendition and deliver it instead of the original rendition.