Hi Harish,
If you want to set up the size limit for the entire assets folder then you can refer to OOTB configuration,
https://experienceleague.adobe.com/docs/experience-manager-64/assets/managing/managing-video-assets.html?lang=en#publishing-video-assets
But, in your case it is not, you need to set it up at the folder level, you can try the below approaches.
1. workflow process
Create a separate workflow process step and in the execute method get the file size either by reading the metadata of the file or reading the size of inputStream, etc, if it crosses a certain amount then you can stop uploading the Asset.
You need to add this in the "DAM Asset Upload" workflow.
2. If you are going to upload Asset via component, file upload field, then you need to write a custom servlet. you will have full control here.
3. We can handle this via Javascript, but it requires some investigation and also must have front end knowledge, check below link only to get some idea, it is not actual solution
http://experience-aem.blogspot.com/2020/04/aem-6540-show-file-size-in-units-in-asset-metadata-editor.html