Unfortunately, not at the moment. Anytime you have a dashboard in the left nav, it will appear on all portfolios. As for access, it would need to be done via layout templates or each person individually doing it.
However, since it is just for one portfolio and everyone can see it. Have you thought about adding it to the Projects or Dashboards area all together? (Sadly, can't add it at Portfolios global area as that isn't customizable.)
Still have to do it on all the layout templates so everyone sees it. but they could all have access without the dashboard being in every portfolio, if it isn't needed.