metadataschema defines the layout for the page properties for the assets, I am doubtful it you can actually disallow a particular asset/file type to be allowed or restricted using this. But you can define the layout specific to an asset/file type if you need to capture any specific metadata info for it.
The only option , I can think is to create a folder metadata schema , to configure the supported file formats for the folder and customise the asset upload workflow to check if a specific file type can be uploaded or not.
Just curious to know, why would you want to restrict a specific file type to be uploaded to a folder, a well structured content folders and naming convention should be good enough to guide the content editor to place the assets in the correct folder.