When you say replace the AEM DAM - out of the box - you cannot upload files to OpenText DAM from the AEM web UI. Nor can you hook in the Content Finder into the Open Text DAM.
However - if you have assets in Open Text DAM that are able to be resolved over HTTP- you can most likely reference files located there from AEM components. That is - create a custom image component you can use in AEM that displays images in the Open Text DAM:
<img src =<URL TO OPEN TEXT DAM>