Several things to check.
1. When you upload the document initially, check to see if the metadata has the property @dam:sha1 with a value.
2. If item 1 is fine, then upload the document again. This time enable query debugging to see if the query below gets executed.
//element(*, dam:Asset)[(jcr:content/metadata/@dam:sha1 = 'SHA1-VALUE')]"
3. If the query is executed and you are still not getting duplicate document detection, I would validate the query in the CRXDE Lite Query Tool.
This should help to provide you with some hints on how to address this. If you still cannot resolve it, I would recommend opening a ticket with us to help you further.