You can check the following documentation https://docs.adobe.com/content/help/en/experience-manager-65/assets/managing/duplicate-detection.html for detecting duplicate assets.
This is done by checking SHA 1 checksum value.
//If the asset you upload already exists in AEM Assets, the Duplicates Detected dialog warns you that you are attempting to upload a duplicate asset. The dialog appears only if the SHA 1 checksum value of the binary of the existing asset matches that of the asset you upload. In this case, the names of assets is immaterial. In other words, the dialog can even appear for assets that have different names if the SHA 1 values for their binaries are the same. With each asset uploaded you will see SHA 1 JCR node created.
So in you use-case you need to write a custom service to do the needful.
Where to find SHA1 number in Image Assets -> jcr:content ->metadata -> dam:sha1 (/content/dam/geometrixx-outdoors/banners/best-season.jpg/jcr:content/metadata)