In this use case you can leverage AEM DAM as an podcast asset management solution. You can use AEM DAM features like metadata, renditions, taxonomy, search and filter on podcast asset objects. I would say if there are many podcasts then it makes sense to use AEM DAM as using DAM the business users will be able to manage and find their podcasts easily. And think about creating a distribution workflow using which the business users can review and approve the podcasts before they are distributed to a channel. The RSS feed for distribution should be easy to build as well.
However, AEM does not provide any tools for podcast creation and editing. It can only be used for managing the podcast assets and distributing them to different channels.
With new AEM as a cloud service offering the asset processing(rendition generation etc.) happens asynchronously outside AEM in a separate microservice application. Therefore in cloud service there will be no performance impact on AEM as well if you have big size of podcasts.