Go to the network tab and filter for calls being made by the assets.adobedtm.com and look at what scripts are taking so long to execute.
Get access to your DTM, allocate the tags (assuming it's due to a cookie look up code that is causing delays) and see if you can replace it with a image tag instead of JS. Normally you can reach out to the vendors/agencies of the tags and have them send a image tag (pixel based solution) that you can use.
Could also try moving the tags to the bottom of page, so they don't interfere with the performance on the page?
If none of the above is true and it's actually "assets.adobedtm.com" that is slow in response, you could host DTM on your own servers, but doubt, that it's the case.