I've discovered that whatever template you attach, will pull in all 'Queue Details' along with all tasks, documents.... So if you have queue topics attached to that template, they will be added to your project, regardless of whether you have the same queue topic name already in that project.
To avoid this issue, I've created templates to set up my base projects (with queue topics), and separate templates to be attached in depending on project needs. These additional templates have no queue topics, though their Queue Details match what I have in the base project template. Hope that makes sense