We give permissions to portfolios via teams only. If you want access to a portfolio (and its contents), you MUST be on the corresponding team. This is one of many ways to handle access permissions.
We remove inherited permission through sharing. Then only the people who are assigned, system admins, or the project is shared with can access it. Just remember it may not show on reports for users who are not Admins. Once the project is no longer a secret or it is complete, we put back inherited permissions so it could be found later on.