It is quite possible and I've done it several times - Tag managers don't slow sites, obviously, there is a performance impact in adding any script to any site but these days negligible. The tags that the tag managers implement are usually the culprit for slow performing sites. Control and manage your tag managers by implementing rules and processes and you should be fine.
To add, it might be easier from a tag management perspective to have both GA and AA on the same tag management platform. I know Adobe Launch supports many analytics tools, which would allow your organization to simplify workflows.